Storing, indexing and searching XML documents in Solr

[Judioo-2]

Hi,
I'm new to solr so apologies if the solution is already documented.
I have installed and populated a solr index using the examples as a template
with a version of the data below.

I have XML in the form of

  <entity>
    <resource>
      <guid>123898-2092099098982</guid>
      <media_format>Blu-Ray</media_format>
      <updated>2011-05-05T11:25:35+0500</updated>
    </resource>
    <price currency="usd">3.99<price>
    <discounts>
      <discount type="percentage" rate="30"
start="2011-05-03T00:00:00" end="2011-05-10T00:00:00" />

      <discount type="decimal" amount="1.99" coupon="1" />
      .....
    </discounts>
    <aspect_ratio>16:9</aspect_ratio>
    <duration>1620</duration>
    <categories>
          <category id="drama" />
          <category id="horror" />
    </categories>
    <rating>
      <rate id="D1">contains some scenes which some viewers may find
upsetting</rate>
    </rating>
    .......
    <media_type>Video</media_type>
</entity>


Can I populate solr directly with this document (like I believe marklogic
does )?
If yes
Can I search on any attribute ( i.e. find all records where
/entity/resource/media_format equals "blu-ray" )

If no
What is the best practice to import the attributes above into solr ( i.e.
patterns for sub dividing / flattening document ).
Does solr support attached documents and if so is this advised ( how does it
affect performance ).

Any help is greatly appreciated. Pointers to documentation that address my
issues is even more helpful.

Thanks again


OJ

[Yury Kats]

On 5/18/2011 4:19 PM, Judioo wrote:

> Any help is greatly appreciated. Pointers to documentation that address my
> issues is even more helpful.

I think this would be a good start:
http://wiki.apache.org/solr/DataImportHandler#Usage_with_XML.2BAC8-HTTP_Datasource

[Judioo-2]

The data is being imported directly from mysql. The document is however
indeed a good starting place.
Thanks

2011/5/18 Yury Kats <[hidden email]>

> On 5/18/2011 4:19 PM, Judioo wrote:
>
> > Any help is greatly appreciated. Pointers to documentation that address
> my
> > issues is even more helpful.
>
> I think this would be a good start:
>
> http://wiki.apache.org/solr/DataImportHandler#Usage_with_XML.2BAC8-HTTP_Datasource
>

[Judioo-2]

Great document. I can see how to import the data direct from the database.
However it seems as though I need to write xpath's in the config to extract
the fields that I wish to transform into an solr document.

So it seems that there is no way of storing the document structure in solr
as is?


2011/5/18 Yury Kats <[hidden email]>

> On 5/18/2011 4:19 PM, Judioo wrote:
>
> > Any help is greatly appreciated. Pointers to documentation that address
> my
> > issues is even more helpful.
>
> I think this would be a good start:
>
> http://wiki.apache.org/solr/DataImportHandler#Usage_with_XML.2BAC8-HTTP_Datasource
>

[Mike Sokolov-2]

You might want to create a field that's analyzed using
HtmlStripCharFilter - this will index all the non-tag/non-attribute text
in the document, and if you store the value, will store the entire XML
document as well.

I've done some work on an XmlStripCharFilter, which does the same thing
(only for well-formed XML) using  the WSTX XML parser, which provides a
little bit of extra XML goodness (like entity resolution and xinclude
processing) that HtmlStripCharFilter doesn't.  I could share if there's
interest.

-Mike

On 05/18/2011 05:27 PM, Judioo wrote:

> Great document. I can see how to import the data direct from the database.
> However it seems as though I need to write xpath's in the config to extract
> the fields that I wish to transform into an solr document.
>
> So it seems that there is no way of storing the document structure in solr
> as is?
>
>
> 2011/5/18 Yury Kats<[hidden email]>
>
>    
>> On 5/18/2011 4:19 PM, Judioo wrote:
>>
>>      
>>> Any help is greatly appreciated. Pointers to documentation that address
>>>        
>> my
>>      
>>> issues is even more helpful.
>>>        
>> I think this would be a good start:
>>
>> http://wiki.apache.org/solr/DataImportHandler#Usage_with_XML.2BAC8-HTTP_Datasource
>>
>>      
>    

[Erick Erickson]

You're right, you can't store an XML document directly in Solr.
You have to pull it apart and index it such that you can get whatever
information back you need.

How you flatten data depends entirely upon your needs. The high-level
idea is that you want to create fields such that text searches work. The
moment you start thinking about "how can I express a relationship
in the query", back up and try to flatten the data so you can just *search*.

This is vague, I know. But so much depends on how you want to use
the data that specifics are hard to give.

You've gotta take off your DB hat and not worry about duplicating
data. De-normalize lots and lots and lots first...

Best
Erick

On Wed, May 18, 2011 at 5:27 PM, Judioo <[hidden email]> wrote:

> Great document. I can see how to import the data direct from the database.
> However it seems as though I need to write xpath's in the config to extract
> the fields that I wish to transform into an solr document.
>
> So it seems that there is no way of storing the document structure in solr
> as is?
>
>
> 2011/5/18 Yury Kats <[hidden email]>
>
>> On 5/18/2011 4:19 PM, Judioo wrote:
>>
>> > Any help is greatly appreciated. Pointers to documentation that address
>> my
>> > issues is even more helpful.
>>
>> I think this would be a good start:
>>
>> http://wiki.apache.org/solr/DataImportHandler#Usage_with_XML.2BAC8-HTTP_Datasource
>>
>