Encryption at lucene index

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Encryption at lucene index

Kumaran Ramasubramanian
Hi All,


After looking at all below discussions, i have one doubt which may be silly
or novice but i want to throw this to lucene user list.

if we have encryption layer included in our analyzer's flow of filters like
EncryptionFilter to control field-level encryption. what are the
consequences ? am i missing anything basic?

Thanks in advance..


Related links:

https://issues.apache.org/jira/browse/LUCENE-2228 : AES Encrypted Directory
- in lucene 3.x

https://issues.apache.org/jira/browse/LUCENE-6966 :  Codec for index-level
encryption - at codec level, to have control on which column / field have
 personal identifiable information

https://security.stackexchange.com/questions/111153/is-a-lucene-search-index-effectively-a-backdoor-for-field-level-encryption


A decent encrypting algorithm will not produce, say, the same first portion
> for two tokens that start with the same letters. So wildcard searches won't
> work. Consider "runs", "running", "runner". A search on "run*" would be
> expected to match all three, but wouldn't unless the encryption were so
> trivial as to be useless. Similar issues arise with sorting. "More Like
> This" would be unreliable. There are many other features of a robust search
> engine that would be impacted, and an index with encrypted terms would be
> useful for only exact matches, which usually results in a poor search
> experience.


https://stackoverflow.com/questions/36604551/adding-encryption-to-solr-lucene-indexes






--
Kumaran R
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Encryption at lucene index

Erick Erickson
Encryption in Solr has a bunch of ramifications. Do you care about

- encryption at rest or in memory?
- encrypting the _searchable_ tokens?
- encrypting the searchable tokens per-user?
- encrypting the stored data (which a filter won't do BTW).

It's actually a fairly complex topic the discussion at LUCENE-6966
outlines much of it. Please ask specific questions as you research the
topic. One  per-user encryption package that I know of is by Hitachi
Solutions (commercial) and it explicitly does _not_ support, for
instance, wildcards (there are other limitations too). See:
http://www.hitachi-solutions.com/securesearch/

Most of the time when people ask for encryption they soon discover
it's much more difficult than they imagine and settle for just putting
the indexes on an encrypting file system. When they move beyond that
it gets complex and you'd be well advised to consult with Solr
security experts.

Best,
Erick

On Sun, Aug 6, 2017 at 11:30 PM, Kumaran Ramasubramanian
<[hidden email]> wrote:

> Hi All,
>
>
> After looking at all below discussions, i have one doubt which may be silly
> or novice but i want to throw this to lucene user list.
>
> if we have encryption layer included in our analyzer's flow of filters like
> EncryptionFilter to control field-level encryption. what are the
> consequences ? am i missing anything basic?
>
> Thanks in advance..
>
>
> Related links:
>
> https://issues.apache.org/jira/browse/LUCENE-2228 : AES Encrypted Directory
> - in lucene 3.x
>
> https://issues.apache.org/jira/browse/LUCENE-6966 :  Codec for index-level
> encryption - at codec level, to have control on which column / field have
>  personal identifiable information
>
> https://security.stackexchange.com/questions/111153/is-a-lucene-search-index-effectively-a-backdoor-for-field-level-encryption
>
>
> A decent encrypting algorithm will not produce, say, the same first portion
>> for two tokens that start with the same letters. So wildcard searches won't
>> work. Consider "runs", "running", "runner". A search on "run*" would be
>> expected to match all three, but wouldn't unless the encryption were so
>> trivial as to be useless. Similar issues arise with sorting. "More Like
>> This" would be unreliable. There are many other features of a robust search
>> engine that would be impacted, and an index with encrypted terms would be
>> useful for only exact matches, which usually results in a poor search
>> experience.
>
>
> https://stackoverflow.com/questions/36604551/adding-encryption-to-solr-lucene-indexes
>
>
>
>
>
>
> --
> Kumaran R

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Encryption at lucene index

Kumaran Ramasubramanian
Hi Erick,

    Thanks for the information. Any pointers about encryption options in
solr?


--
Kumaran R



On Mon, Aug 7, 2017 at 9:17 PM, Erick Erickson <[hidden email]>
wrote:

> Encryption in Solr has a bunch of ramifications. Do you care about
>
> - encryption at rest or in memory?
> - encrypting the _searchable_ tokens?
> - encrypting the searchable tokens per-user?
> - encrypting the stored data (which a filter won't do BTW).
>
> It's actually a fairly complex topic the discussion at LUCENE-6966
> outlines much of it. Please ask specific questions as you research the
> topic. One  per-user encryption package that I know of is by Hitachi
> Solutions (commercial) and it explicitly does _not_ support, for
> instance, wildcards (there are other limitations too). See:
> http://www.hitachi-solutions.com/securesearch/
>
> Most of the time when people ask for encryption they soon discover
> it's much more difficult than they imagine and settle for just putting
> the indexes on an encrypting file system. When they move beyond that
> it gets complex and you'd be well advised to consult with Solr
> security experts.
>
> Best,
> Erick
>
> On Sun, Aug 6, 2017 at 11:30 PM, Kumaran Ramasubramanian
> <[hidden email]> wrote:
> > Hi All,
> >
> >
> > After looking at all below discussions, i have one doubt which may be
> silly
> > or novice but i want to throw this to lucene user list.
> >
> > if we have encryption layer included in our analyzer's flow of filters
> like
> > EncryptionFilter to control field-level encryption. what are the
> > consequences ? am i missing anything basic?
> >
> > Thanks in advance..
> >
> >
> > Related links:
> >
> > https://issues.apache.org/jira/browse/LUCENE-2228 : AES Encrypted
> Directory
> > - in lucene 3.x
> >
> > https://issues.apache.org/jira/browse/LUCENE-6966 :  Codec for
> index-level
> > encryption - at codec level, to have control on which column / field have
> >  personal identifiable information
> >
> > https://security.stackexchange.com/questions/111153/is-a-lucene-search-
> index-effectively-a-backdoor-for-field-level-encryption
> >
> >
> > A decent encrypting algorithm will not produce, say, the same first
> portion
> >> for two tokens that start with the same letters. So wildcard searches
> won't
> >> work. Consider "runs", "running", "runner". A search on "run*" would be
> >> expected to match all three, but wouldn't unless the encryption were so
> >> trivial as to be useless. Similar issues arise with sorting. "More Like
> >> This" would be unreliable. There are many other features of a robust
> search
> >> engine that would be impacted, and an index with encrypted terms would
> be
> >> useful for only exact matches, which usually results in a poor search
> >> experience.
> >
> >
> > https://stackoverflow.com/questions/36604551/adding-
> encryption-to-solr-lucene-indexes
> >
> >
> >
> >
> >
> >
> > --
> > Kumaran R
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Encryption at lucene index

Erick Erickson
No, since you haven't defined what you want to encrypt, what your
requirements are, what you hope to get out of "encryption" etc.

Put the index on an encrypting filesystem and forget about it if you
possibly can, because anything else is a significant amount of work.
To encrypt the searchable tokens on a per-user basis in memory is a
_lot_ of work. It depends on your security needs.

Otherwise, as I said, please ask specific questions as the topic is
quite large, much too large to conduct a seminar through the user's
list.

Best,
Erick

On Mon, Aug 7, 2017 at 9:07 AM, Kumaran Ramasubramanian
<[hidden email]> wrote:

> Hi Erick,
>
>     Thanks for the information. Any pointers about encryption options in
> solr?
>
>
> --
> Kumaran R
>
>
>
> On Mon, Aug 7, 2017 at 9:17 PM, Erick Erickson <[hidden email]>
> wrote:
>
>> Encryption in Solr has a bunch of ramifications. Do you care about
>>
>> - encryption at rest or in memory?
>> - encrypting the _searchable_ tokens?
>> - encrypting the searchable tokens per-user?
>> - encrypting the stored data (which a filter won't do BTW).
>>
>> It's actually a fairly complex topic the discussion at LUCENE-6966
>> outlines much of it. Please ask specific questions as you research the
>> topic. One  per-user encryption package that I know of is by Hitachi
>> Solutions (commercial) and it explicitly does _not_ support, for
>> instance, wildcards (there are other limitations too). See:
>> http://www.hitachi-solutions.com/securesearch/
>>
>> Most of the time when people ask for encryption they soon discover
>> it's much more difficult than they imagine and settle for just putting
>> the indexes on an encrypting file system. When they move beyond that
>> it gets complex and you'd be well advised to consult with Solr
>> security experts.
>>
>> Best,
>> Erick
>>
>> On Sun, Aug 6, 2017 at 11:30 PM, Kumaran Ramasubramanian
>> <[hidden email]> wrote:
>> > Hi All,
>> >
>> >
>> > After looking at all below discussions, i have one doubt which may be
>> silly
>> > or novice but i want to throw this to lucene user list.
>> >
>> > if we have encryption layer included in our analyzer's flow of filters
>> like
>> > EncryptionFilter to control field-level encryption. what are the
>> > consequences ? am i missing anything basic?
>> >
>> > Thanks in advance..
>> >
>> >
>> > Related links:
>> >
>> > https://issues.apache.org/jira/browse/LUCENE-2228 : AES Encrypted
>> Directory
>> > - in lucene 3.x
>> >
>> > https://issues.apache.org/jira/browse/LUCENE-6966 :  Codec for
>> index-level
>> > encryption - at codec level, to have control on which column / field have
>> >  personal identifiable information
>> >
>> > https://security.stackexchange.com/questions/111153/is-a-lucene-search-
>> index-effectively-a-backdoor-for-field-level-encryption
>> >
>> >
>> > A decent encrypting algorithm will not produce, say, the same first
>> portion
>> >> for two tokens that start with the same letters. So wildcard searches
>> won't
>> >> work. Consider "runs", "running", "runner". A search on "run*" would be
>> >> expected to match all three, but wouldn't unless the encryption were so
>> >> trivial as to be useless. Similar issues arise with sorting. "More Like
>> >> This" would be unreliable. There are many other features of a robust
>> search
>> >> engine that would be impacted, and an index with encrypted terms would
>> be
>> >> useful for only exact matches, which usually results in a poor search
>> >> experience.
>> >
>> >
>> > https://stackoverflow.com/questions/36604551/adding-
>> encryption-to-solr-lucene-indexes
>> >
>> >
>> >
>> >
>> >
>> >
>> > --
>> > Kumaran R
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Encryption at lucene index

Kumaran Ramasubramanian
Hi Erick, i want to encrypt some fields of an document which has personal
identifiable information ( both indexed and stored data)... for eg: email,
mobilenumber etc.. i am able to find LUCENE-6966 alone while googling it..
any related pointers in solr or latest lucene version?


-
​-
Kumaran R​

On Mon, Aug 7, 2017 at 9:52 PM, Erick Erickson <[hidden email]>
wrote:

> No, since you haven't defined what you want to encrypt, what your
> requirements are, what you hope to get out of "encryption" etc.
>
> Put the index on an encrypting filesystem and forget about it if you
> possibly can, because anything else is a significant amount of work.
> To encrypt the searchable tokens on a per-user basis in memory is a
> _lot_ of work. It depends on your security needs.
>
> Otherwise, as I said, please ask specific questions as the topic is
> quite large, much too large to conduct a seminar through the user's
> list.
>
> Best,
> Erick
>
> On Mon, Aug 7, 2017 at 9:07 AM, Kumaran Ramasubramanian
> <[hidden email]> wrote:
> > Hi Erick,
> >
> >     Thanks for the information. Any pointers about encryption options in
> > solr?
> >
> >
> > --
> > Kumaran R
> >
> >
> >
> > On Mon, Aug 7, 2017 at 9:17 PM, Erick Erickson <[hidden email]>
> > wrote:
> >
> >> Encryption in Solr has a bunch of ramifications. Do you care about
> >>
> >> - encryption at rest or in memory?
> >> - encrypting the _searchable_ tokens?
> >> - encrypting the searchable tokens per-user?
> >> - encrypting the stored data (which a filter won't do BTW).
> >>
> >> It's actually a fairly complex topic the discussion at LUCENE-6966
> >> outlines much of it. Please ask specific questions as you research the
> >> topic. One  per-user encryption package that I know of is by Hitachi
> >> Solutions (commercial) and it explicitly does _not_ support, for
> >> instance, wildcards (there are other limitations too). See:
> >> http://www.hitachi-solutions.com/securesearch/
> >>
> >> Most of the time when people ask for encryption they soon discover
> >> it's much more difficult than they imagine and settle for just putting
> >> the indexes on an encrypting file system. When they move beyond that
> >> it gets complex and you'd be well advised to consult with Solr
> >> security experts.
> >>
> >> Best,
> >> Erick
> >>
> >> On Sun, Aug 6, 2017 at 11:30 PM, Kumaran Ramasubramanian
> >> <[hidden email]> wrote:
> >> > Hi All,
> >> >
> >> >
> >> > After looking at all below discussions, i have one doubt which may be
> >> silly
> >> > or novice but i want to throw this to lucene user list.
> >> >
> >> > if we have encryption layer included in our analyzer's flow of filters
> >> like
> >> > EncryptionFilter to control field-level encryption. what are the
> >> > consequences ? am i missing anything basic?
> >> >
> >> > Thanks in advance..
> >> >
> >> >
> >> > Related links:
> >> >
> >> > https://issues.apache.org/jira/browse/LUCENE-2228 : AES Encrypted
> >> Directory
> >> > - in lucene 3.x
> >> >
> >> > https://issues.apache.org/jira/browse/LUCENE-6966 :  Codec for
> >> index-level
> >> > encryption - at codec level, to have control on which column / field
> have
> >> >  personal identifiable information
> >> >
> >> > https://security.stackexchange.com/questions/
> 111153/is-a-lucene-search-
> >> index-effectively-a-backdoor-for-field-level-encryption
> >> >
> >> >
> >> > A decent encrypting algorithm will not produce, say, the same first
> >> portion
> >> >> for two tokens that start with the same letters. So wildcard searches
> >> won't
> >> >> work. Consider "runs", "running", "runner". A search on "run*" would
> be
> >> >> expected to match all three, but wouldn't unless the encryption were
> so
> >> >> trivial as to be useless. Similar issues arise with sorting. "More
> Like
> >> >> This" would be unreliable. There are many other features of a robust
> >> search
> >> >> engine that would be impacted, and an index with encrypted terms
> would
> >> be
> >> >> useful for only exact matches, which usually results in a poor search
> >> >> experience.
> >> >
> >> >
> >> > https://stackoverflow.com/questions/36604551/adding-
> >> encryption-to-solr-lucene-indexes
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Kumaran R
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Encryption at lucene index

Ishan Chattopadhyaya
Harry Ochiai (Hitachi) has some index encryption solution,
https://www.slideshare.net/maggon/securing-solr-search-data-in-the-cloud
I think it is proprietary, but I'm not sure. Maybe more googling might help
find the exact page where his solution is described.

On Mon, Aug 7, 2017 at 9:59 PM, Kumaran Ramasubramanian <[hidden email]>
wrote:

> Hi Erick, i want to encrypt some fields of an document which has personal
> identifiable information ( both indexed and stored data)... for eg: email,
> mobilenumber etc.. i am able to find LUCENE-6966 alone while googling it..
> any related pointers in solr or latest lucene version?
>
>
> -
> ​-
> Kumaran R​
>
> On Mon, Aug 7, 2017 at 9:52 PM, Erick Erickson <[hidden email]>
> wrote:
>
> > No, since you haven't defined what you want to encrypt, what your
> > requirements are, what you hope to get out of "encryption" etc.
> >
> > Put the index on an encrypting filesystem and forget about it if you
> > possibly can, because anything else is a significant amount of work.
> > To encrypt the searchable tokens on a per-user basis in memory is a
> > _lot_ of work. It depends on your security needs.
> >
> > Otherwise, as I said, please ask specific questions as the topic is
> > quite large, much too large to conduct a seminar through the user's
> > list.
> >
> > Best,
> > Erick
> >
> > On Mon, Aug 7, 2017 at 9:07 AM, Kumaran Ramasubramanian
> > <[hidden email]> wrote:
> > > Hi Erick,
> > >
> > >     Thanks for the information. Any pointers about encryption options
> in
> > > solr?
> > >
> > >
> > > --
> > > Kumaran R
> > >
> > >
> > >
> > > On Mon, Aug 7, 2017 at 9:17 PM, Erick Erickson <
> [hidden email]>
> > > wrote:
> > >
> > >> Encryption in Solr has a bunch of ramifications. Do you care about
> > >>
> > >> - encryption at rest or in memory?
> > >> - encrypting the _searchable_ tokens?
> > >> - encrypting the searchable tokens per-user?
> > >> - encrypting the stored data (which a filter won't do BTW).
> > >>
> > >> It's actually a fairly complex topic the discussion at LUCENE-6966
> > >> outlines much of it. Please ask specific questions as you research the
> > >> topic. One  per-user encryption package that I know of is by Hitachi
> > >> Solutions (commercial) and it explicitly does _not_ support, for
> > >> instance, wildcards (there are other limitations too). See:
> > >> http://www.hitachi-solutions.com/securesearch/
> > >>
> > >> Most of the time when people ask for encryption they soon discover
> > >> it's much more difficult than they imagine and settle for just putting
> > >> the indexes on an encrypting file system. When they move beyond that
> > >> it gets complex and you'd be well advised to consult with Solr
> > >> security experts.
> > >>
> > >> Best,
> > >> Erick
> > >>
> > >> On Sun, Aug 6, 2017 at 11:30 PM, Kumaran Ramasubramanian
> > >> <[hidden email]> wrote:
> > >> > Hi All,
> > >> >
> > >> >
> > >> > After looking at all below discussions, i have one doubt which may
> be
> > >> silly
> > >> > or novice but i want to throw this to lucene user list.
> > >> >
> > >> > if we have encryption layer included in our analyzer's flow of
> filters
> > >> like
> > >> > EncryptionFilter to control field-level encryption. what are the
> > >> > consequences ? am i missing anything basic?
> > >> >
> > >> > Thanks in advance..
> > >> >
> > >> >
> > >> > Related links:
> > >> >
> > >> > https://issues.apache.org/jira/browse/LUCENE-2228 : AES Encrypted
> > >> Directory
> > >> > - in lucene 3.x
> > >> >
> > >> > https://issues.apache.org/jira/browse/LUCENE-6966 :  Codec for
> > >> index-level
> > >> > encryption - at codec level, to have control on which column / field
> > have
> > >> >  personal identifiable information
> > >> >
> > >> > https://security.stackexchange.com/questions/
> > 111153/is-a-lucene-search-
> > >> index-effectively-a-backdoor-for-field-level-encryption
> > >> >
> > >> >
> > >> > A decent encrypting algorithm will not produce, say, the same first
> > >> portion
> > >> >> for two tokens that start with the same letters. So wildcard
> searches
> > >> won't
> > >> >> work. Consider "runs", "running", "runner". A search on "run*"
> would
> > be
> > >> >> expected to match all three, but wouldn't unless the encryption
> were
> > so
> > >> >> trivial as to be useless. Similar issues arise with sorting. "More
> > Like
> > >> >> This" would be unreliable. There are many other features of a
> robust
> > >> search
> > >> >> engine that would be impacted, and an index with encrypted terms
> > would
> > >> be
> > >> >> useful for only exact matches, which usually results in a poor
> search
> > >> >> experience.
> > >> >
> > >> >
> > >> > https://stackoverflow.com/questions/36604551/adding-
> > >> encryption-to-solr-lucene-indexes
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Kumaran R
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: [hidden email]
> > >> For additional commands, e-mail: [hidden email]
> > >>
> > >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Encryption at lucene index

Kumaran Ramasubramanian
Hi Ishan, thank you :-)

-
​-
Kumaran R



On Mon, Aug 7, 2017 at 10:53 PM, Ishan Chattopadhyaya <
[hidden email]> wrote:

> Harry Ochiai (Hitachi) has some index encryption solution,
> https://www.slideshare.net/maggon/securing-solr-search-data-in-the-cloud
> I think it is proprietary, but I'm not sure. Maybe more googling might help
> find the exact page where his solution is described.
>
> On Mon, Aug 7, 2017 at 9:59 PM, Kumaran Ramasubramanian <
> [hidden email]>
> wrote:
>
> > Hi Erick, i want to encrypt some fields of an document which has personal
> > identifiable information ( both indexed and stored data)... for eg:
> email,
> > mobilenumber etc.. i am able to find LUCENE-6966 alone while googling
> it..
> > any related pointers in solr or latest lucene version?
> >
> >
> > -
> > ​-
> > Kumaran R​
> >
> > On Mon, Aug 7, 2017 at 9:52 PM, Erick Erickson <[hidden email]>
> > wrote:
> >
> > > No, since you haven't defined what you want to encrypt, what your
> > > requirements are, what you hope to get out of "encryption" etc.
> > >
> > > Put the index on an encrypting filesystem and forget about it if you
> > > possibly can, because anything else is a significant amount of work.
> > > To encrypt the searchable tokens on a per-user basis in memory is a
> > > _lot_ of work. It depends on your security needs.
> > >
> > > Otherwise, as I said, please ask specific questions as the topic is
> > > quite large, much too large to conduct a seminar through the user's
> > > list.
> > >
> > > Best,
> > > Erick
> > >
> > > On Mon, Aug 7, 2017 at 9:07 AM, Kumaran Ramasubramanian
> > > <[hidden email]> wrote:
> > > > Hi Erick,
> > > >
> > > >     Thanks for the information. Any pointers about encryption options
> > in
> > > > solr?
> > > >
> > > >
> > > > --
> > > > Kumaran R
> > > >
> > > >
> > > >
> > > > On Mon, Aug 7, 2017 at 9:17 PM, Erick Erickson <
> > [hidden email]>
> > > > wrote:
> > > >
> > > >> Encryption in Solr has a bunch of ramifications. Do you care about
> > > >>
> > > >> - encryption at rest or in memory?
> > > >> - encrypting the _searchable_ tokens?
> > > >> - encrypting the searchable tokens per-user?
> > > >> - encrypting the stored data (which a filter won't do BTW).
> > > >>
> > > >> It's actually a fairly complex topic the discussion at LUCENE-6966
> > > >> outlines much of it. Please ask specific questions as you research
> the
> > > >> topic. One  per-user encryption package that I know of is by Hitachi
> > > >> Solutions (commercial) and it explicitly does _not_ support, for
> > > >> instance, wildcards (there are other limitations too). See:
> > > >> http://www.hitachi-solutions.com/securesearch/
> > > >>
> > > >> Most of the time when people ask for encryption they soon discover
> > > >> it's much more difficult than they imagine and settle for just
> putting
> > > >> the indexes on an encrypting file system. When they move beyond that
> > > >> it gets complex and you'd be well advised to consult with Solr
> > > >> security experts.
> > > >>
> > > >> Best,
> > > >> Erick
> > > >>
> > > >> On Sun, Aug 6, 2017 at 11:30 PM, Kumaran Ramasubramanian
> > > >> <[hidden email]> wrote:
> > > >> > Hi All,
> > > >> >
> > > >> >
> > > >> > After looking at all below discussions, i have one doubt which may
> > be
> > > >> silly
> > > >> > or novice but i want to throw this to lucene user list.
> > > >> >
> > > >> > if we have encryption layer included in our analyzer's flow of
> > filters
> > > >> like
> > > >> > EncryptionFilter to control field-level encryption. what are the
> > > >> > consequences ? am i missing anything basic?
> > > >> >
> > > >> > Thanks in advance..
> > > >> >
> > > >> >
> > > >> > Related links:
> > > >> >
> > > >> > https://issues.apache.org/jira/browse/LUCENE-2228 : AES Encrypted
> > > >> Directory
> > > >> > - in lucene 3.x
> > > >> >
> > > >> > https://issues.apache.org/jira/browse/LUCENE-6966 :  Codec for
> > > >> index-level
> > > >> > encryption - at codec level, to have control on which column /
> field
> > > have
> > > >> >  personal identifiable information
> > > >> >
> > > >> > https://security.stackexchange.com/questions/
> > > 111153/is-a-lucene-search-
> > > >> index-effectively-a-backdoor-for-field-level-encryption
> > > >> >
> > > >> >
> > > >> > A decent encrypting algorithm will not produce, say, the same
> first
> > > >> portion
> > > >> >> for two tokens that start with the same letters. So wildcard
> > searches
> > > >> won't
> > > >> >> work. Consider "runs", "running", "runner". A search on "run*"
> > would
> > > be
> > > >> >> expected to match all three, but wouldn't unless the encryption
> > were
> > > so
> > > >> >> trivial as to be useless. Similar issues arise with sorting.
> "More
> > > Like
> > > >> >> This" would be unreliable. There are many other features of a
> > robust
> > > >> search
> > > >> >> engine that would be impacted, and an index with encrypted terms
> > > would
> > > >> be
> > > >> >> useful for only exact matches, which usually results in a poor
> > search
> > > >> >> experience.
> > > >> >
> > > >> >
> > > >> > https://stackoverflow.com/questions/36604551/adding-
> > > >> encryption-to-solr-lucene-indexes
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Kumaran R
> > > >>
> > > >> ------------------------------------------------------------
> ---------
> > > >> To unsubscribe, e-mail: [hidden email]
> > > >> For additional commands, e-mail: [hidden email]
> > > >>
> > > >>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: [hidden email]
> > > For additional commands, e-mail: [hidden email]
> > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Encryption at lucene index

Erick Erickson
Encrypting the _tokens_ inevitably leads to reduced capabilities BTW.
Trivial example:
I have these tokens in my index
run
runner
running
runs

Any non-trivial encryption algorithm will not encrypt the first three
letters "run" identically in all three so searching for run* simply
won't work.

As you can see, there's quite a bit of back-and-forth with that JIRA
and it is pretty much been abandoned.

Best,
Erick

On Thu, Aug 10, 2017 at 11:17 PM, Kumaran Ramasubramanian
<[hidden email]> wrote:

> Hi Ishan, thank you :-)
>
> -
> -
> Kumaran R
>
>
>
> On Mon, Aug 7, 2017 at 10:53 PM, Ishan Chattopadhyaya <
> [hidden email]> wrote:
>
>> Harry Ochiai (Hitachi) has some index encryption solution,
>> https://www.slideshare.net/maggon/securing-solr-search-data-in-the-cloud
>> I think it is proprietary, but I'm not sure. Maybe more googling might help
>> find the exact page where his solution is described.
>>
>> On Mon, Aug 7, 2017 at 9:59 PM, Kumaran Ramasubramanian <
>> [hidden email]>
>> wrote:
>>
>> > Hi Erick, i want to encrypt some fields of an document which has personal
>> > identifiable information ( both indexed and stored data)... for eg:
>> email,
>> > mobilenumber etc.. i am able to find LUCENE-6966 alone while googling
>> it..
>> > any related pointers in solr or latest lucene version?
>> >
>> >
>> > -
>> > -
>> > Kumaran R
>> >
>> > On Mon, Aug 7, 2017 at 9:52 PM, Erick Erickson <[hidden email]>
>> > wrote:
>> >
>> > > No, since you haven't defined what you want to encrypt, what your
>> > > requirements are, what you hope to get out of "encryption" etc.
>> > >
>> > > Put the index on an encrypting filesystem and forget about it if you
>> > > possibly can, because anything else is a significant amount of work.
>> > > To encrypt the searchable tokens on a per-user basis in memory is a
>> > > _lot_ of work. It depends on your security needs.
>> > >
>> > > Otherwise, as I said, please ask specific questions as the topic is
>> > > quite large, much too large to conduct a seminar through the user's
>> > > list.
>> > >
>> > > Best,
>> > > Erick
>> > >
>> > > On Mon, Aug 7, 2017 at 9:07 AM, Kumaran Ramasubramanian
>> > > <[hidden email]> wrote:
>> > > > Hi Erick,
>> > > >
>> > > >     Thanks for the information. Any pointers about encryption options
>> > in
>> > > > solr?
>> > > >
>> > > >
>> > > > --
>> > > > Kumaran R
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Aug 7, 2017 at 9:17 PM, Erick Erickson <
>> > [hidden email]>
>> > > > wrote:
>> > > >
>> > > >> Encryption in Solr has a bunch of ramifications. Do you care about
>> > > >>
>> > > >> - encryption at rest or in memory?
>> > > >> - encrypting the _searchable_ tokens?
>> > > >> - encrypting the searchable tokens per-user?
>> > > >> - encrypting the stored data (which a filter won't do BTW).
>> > > >>
>> > > >> It's actually a fairly complex topic the discussion at LUCENE-6966
>> > > >> outlines much of it. Please ask specific questions as you research
>> the
>> > > >> topic. One  per-user encryption package that I know of is by Hitachi
>> > > >> Solutions (commercial) and it explicitly does _not_ support, for
>> > > >> instance, wildcards (there are other limitations too). See:
>> > > >> http://www.hitachi-solutions.com/securesearch/
>> > > >>
>> > > >> Most of the time when people ask for encryption they soon discover
>> > > >> it's much more difficult than they imagine and settle for just
>> putting
>> > > >> the indexes on an encrypting file system. When they move beyond that
>> > > >> it gets complex and you'd be well advised to consult with Solr
>> > > >> security experts.
>> > > >>
>> > > >> Best,
>> > > >> Erick
>> > > >>
>> > > >> On Sun, Aug 6, 2017 at 11:30 PM, Kumaran Ramasubramanian
>> > > >> <[hidden email]> wrote:
>> > > >> > Hi All,
>> > > >> >
>> > > >> >
>> > > >> > After looking at all below discussions, i have one doubt which may
>> > be
>> > > >> silly
>> > > >> > or novice but i want to throw this to lucene user list.
>> > > >> >
>> > > >> > if we have encryption layer included in our analyzer's flow of
>> > filters
>> > > >> like
>> > > >> > EncryptionFilter to control field-level encryption. what are the
>> > > >> > consequences ? am i missing anything basic?
>> > > >> >
>> > > >> > Thanks in advance..
>> > > >> >
>> > > >> >
>> > > >> > Related links:
>> > > >> >
>> > > >> > https://issues.apache.org/jira/browse/LUCENE-2228 : AES Encrypted
>> > > >> Directory
>> > > >> > - in lucene 3.x
>> > > >> >
>> > > >> > https://issues.apache.org/jira/browse/LUCENE-6966 :  Codec for
>> > > >> index-level
>> > > >> > encryption - at codec level, to have control on which column /
>> field
>> > > have
>> > > >> >  personal identifiable information
>> > > >> >
>> > > >> > https://security.stackexchange.com/questions/
>> > > 111153/is-a-lucene-search-
>> > > >> index-effectively-a-backdoor-for-field-level-encryption
>> > > >> >
>> > > >> >
>> > > >> > A decent encrypting algorithm will not produce, say, the same
>> first
>> > > >> portion
>> > > >> >> for two tokens that start with the same letters. So wildcard
>> > searches
>> > > >> won't
>> > > >> >> work. Consider "runs", "running", "runner". A search on "run*"
>> > would
>> > > be
>> > > >> >> expected to match all three, but wouldn't unless the encryption
>> > were
>> > > so
>> > > >> >> trivial as to be useless. Similar issues arise with sorting.
>> "More
>> > > Like
>> > > >> >> This" would be unreliable. There are many other features of a
>> > robust
>> > > >> search
>> > > >> >> engine that would be impacted, and an index with encrypted terms
>> > > would
>> > > >> be
>> > > >> >> useful for only exact matches, which usually results in a poor
>> > search
>> > > >> >> experience.
>> > > >> >
>> > > >> >
>> > > >> > https://stackoverflow.com/questions/36604551/adding-
>> > > >> encryption-to-solr-lucene-indexes
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> > --
>> > > >> > Kumaran R
>> > > >>
>> > > >> ------------------------------------------------------------
>> ---------
>> > > >> To unsubscribe, e-mail: [hidden email]
>> > > >> For additional commands, e-mail: [hidden email]
>> > > >>
>> > > >>
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: [hidden email]
>> > > For additional commands, e-mail: [hidden email]
>> > >
>> > >
>> >
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Encryption at lucene index

Kumaran Ramasubramanian
I got it Erick.. Thank you..

-
​Kumaran R


On Fri, Aug 11, 2017 at 10:35 PM, Erick Erickson <[hidden email]>
wrote:

> Encrypting the _tokens_ inevitably leads to reduced capabilities BTW.
> Trivial example:
> I have these tokens in my index
> run
> runner
> running
> runs
>
> Any non-trivial encryption algorithm will not encrypt the first three
> letters "run" identically in all three so searching for run* simply
> won't work.
>
> As you can see, there's quite a bit of back-and-forth with that JIRA
> and it is pretty much been abandoned.
>
> Best,
> Erick
>
> On Thu, Aug 10, 2017 at 11:17 PM, Kumaran Ramasubramanian
> <[hidden email]> wrote:
> > Hi Ishan, thank you :-)
> >
> > -
> > -
> > Kumaran R
> >
> >
> >
> > On Mon, Aug 7, 2017 at 10:53 PM, Ishan Chattopadhyaya <
> > [hidden email]> wrote:
> >
> >> Harry Ochiai (Hitachi) has some index encryption solution,
> >> https://www.slideshare.net/maggon/securing-solr-search-
> data-in-the-cloud
> >> I think it is proprietary, but I'm not sure. Maybe more googling might
> help
> >> find the exact page where his solution is described.
> >>
> >> On Mon, Aug 7, 2017 at 9:59 PM, Kumaran Ramasubramanian <
> >> [hidden email]>
> >> wrote:
> >>
> >> > Hi Erick, i want to encrypt some fields of an document which has
> personal
> >> > identifiable information ( both indexed and stored data)... for eg:
> >> email,
> >> > mobilenumber etc.. i am able to find LUCENE-6966 alone while googling
> >> it..
> >> > any related pointers in solr or latest lucene version?
> >> >
> >> >
> >> > -
> >> > -
> >> > Kumaran R
> >> >
> >> > On Mon, Aug 7, 2017 at 9:52 PM, Erick Erickson <
> [hidden email]>
> >> > wrote:
> >> >
> >> > > No, since you haven't defined what you want to encrypt, what your
> >> > > requirements are, what you hope to get out of "encryption" etc.
> >> > >
> >> > > Put the index on an encrypting filesystem and forget about it if you
> >> > > possibly can, because anything else is a significant amount of work.
> >> > > To encrypt the searchable tokens on a per-user basis in memory is a
> >> > > _lot_ of work. It depends on your security needs.
> >> > >
> >> > > Otherwise, as I said, please ask specific questions as the topic is
> >> > > quite large, much too large to conduct a seminar through the user's
> >> > > list.
> >> > >
> >> > > Best,
> >> > > Erick
> >> > >
> >> > > On Mon, Aug 7, 2017 at 9:07 AM, Kumaran Ramasubramanian
> >> > > <[hidden email]> wrote:
> >> > > > Hi Erick,
> >> > > >
> >> > > >     Thanks for the information. Any pointers about encryption
> options
> >> > in
> >> > > > solr?
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Kumaran R
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Mon, Aug 7, 2017 at 9:17 PM, Erick Erickson <
> >> > [hidden email]>
> >> > > > wrote:
> >> > > >
> >> > > >> Encryption in Solr has a bunch of ramifications. Do you care
> about
> >> > > >>
> >> > > >> - encryption at rest or in memory?
> >> > > >> - encrypting the _searchable_ tokens?
> >> > > >> - encrypting the searchable tokens per-user?
> >> > > >> - encrypting the stored data (which a filter won't do BTW).
> >> > > >>
> >> > > >> It's actually a fairly complex topic the discussion at
> LUCENE-6966
> >> > > >> outlines much of it. Please ask specific questions as you
> research
> >> the
> >> > > >> topic. One  per-user encryption package that I know of is by
> Hitachi
> >> > > >> Solutions (commercial) and it explicitly does _not_ support, for
> >> > > >> instance, wildcards (there are other limitations too). See:
> >> > > >> http://www.hitachi-solutions.com/securesearch/
> >> > > >>
> >> > > >> Most of the time when people ask for encryption they soon
> discover
> >> > > >> it's much more difficult than they imagine and settle for just
> >> putting
> >> > > >> the indexes on an encrypting file system. When they move beyond
> that
> >> > > >> it gets complex and you'd be well advised to consult with Solr
> >> > > >> security experts.
> >> > > >>
> >> > > >> Best,
> >> > > >> Erick
> >> > > >>
> >> > > >> On Sun, Aug 6, 2017 at 11:30 PM, Kumaran Ramasubramanian
> >> > > >> <[hidden email]> wrote:
> >> > > >> > Hi All,
> >> > > >> >
> >> > > >> >
> >> > > >> > After looking at all below discussions, i have one doubt which
> may
> >> > be
> >> > > >> silly
> >> > > >> > or novice but i want to throw this to lucene user list.
> >> > > >> >
> >> > > >> > if we have encryption layer included in our analyzer's flow of
> >> > filters
> >> > > >> like
> >> > > >> > EncryptionFilter to control field-level encryption. what are
> the
> >> > > >> > consequences ? am i missing anything basic?
> >> > > >> >
> >> > > >> > Thanks in advance..
> >> > > >> >
> >> > > >> >
> >> > > >> > Related links:
> >> > > >> >
> >> > > >> > https://issues.apache.org/jira/browse/LUCENE-2228 : AES
> Encrypted
> >> > > >> Directory
> >> > > >> > - in lucene 3.x
> >> > > >> >
> >> > > >> > https://issues.apache.org/jira/browse/LUCENE-6966 :  Codec for
> >> > > >> index-level
> >> > > >> > encryption - at codec level, to have control on which column /
> >> field
> >> > > have
> >> > > >> >  personal identifiable information
> >> > > >> >
> >> > > >> > https://security.stackexchange.com/questions/
> >> > > 111153/is-a-lucene-search-
> >> > > >> index-effectively-a-backdoor-for-field-level-encryption
> >> > > >> >
> >> > > >> >
> >> > > >> > A decent encrypting algorithm will not produce, say, the same
> >> first
> >> > > >> portion
> >> > > >> >> for two tokens that start with the same letters. So wildcard
> >> > searches
> >> > > >> won't
> >> > > >> >> work. Consider "runs", "running", "runner". A search on "run*"
> >> > would
> >> > > be
> >> > > >> >> expected to match all three, but wouldn't unless the
> encryption
> >> > were
> >> > > so
> >> > > >> >> trivial as to be useless. Similar issues arise with sorting.
> >> "More
> >> > > Like
> >> > > >> >> This" would be unreliable. There are many other features of a
> >> > robust
> >> > > >> search
> >> > > >> >> engine that would be impacted, and an index with encrypted
> terms
> >> > > would
> >> > > >> be
> >> > > >> >> useful for only exact matches, which usually results in a poor
> >> > search
> >> > > >> >> experience.
> >> > > >> >
> >> > > >> >
> >> > > >> > https://stackoverflow.com/questions/36604551/adding-
> >> > > >> encryption-to-solr-lucene-indexes
> >> > > >> >
> >> > > >> >
> >> > > >> >
> >> > > >> >
> >> > > >> >
> >> > > >> >
> >> > > >> > --
> >> > > >> > Kumaran R
> >> > > >>
> >> > > >> ------------------------------------------------------------
> >> ---------
> >> > > >> To unsubscribe, e-mail: [hidden email]
> >> > > >> For additional commands, e-mail: [hidden email].
> org
> >> > > >>
> >> > > >>
> >> > >
> >> > > ------------------------------------------------------------
> ---------
> >> > > To unsubscribe, e-mail: [hidden email]
> >> > > For additional commands, e-mail: [hidden email]
> >> > >
> >> > >
> >> >
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Loading...