Parent Child Entity - DataImport

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Parent Child Entity - DataImport

bizzyim
I have a parent entity that grabs a list of records of a certain type from 1 table... and a sub-entity that queries another table to retrieve the actual data... for various reasons I cannot join the tables... the 2nd sql query converts the rows into an xml to be processed by a custom transformer (done due to the complex nature of the second table)

Full-import works fine but delta-import is not adding any new records...

Do I have to specify a deltaQuery for the sub-entity? What else might be goin on?

<document name="doc">
        <entity name="table1" pk="id"
    query=" SELECT ID,MY_GUID
    FROM activityLog
    WHERE type in (11, 15)"
                deltaQuery="
                        SELECT ID,MY_GUID
    FROM activityLog
    WHERE type in (11, 15) and created_date > '${dataimporter.last_index_time}'">
     <field column="MY_GUID" name="myGuid"/>
                <entity name="table2" pk="ID"
                                query="select dbms_xmlgen.getxml('
                                                        select Name, Title, Description
                                                                from metaDataTable
                                                        where MY_GUID = ${table1.MY_GUID_ID}
                                            ') mdrXmlClob
                                                from dual
                                          "
                                transformer="MD.Solr.Utils.transformers.MDTransformer">
                        <field column="Name" name="mdName"/>
                        <field column="Title" name="mdTitle"/>
                        <field column="Description" name="mdDescription"/>
           </entity>
        </entity>
</document>
Reply | Threaded
Open this post in threaded view
|

Re: Parent Child Entity - DataImport

Noble Paul നോബിള്‍  नोब्ळ्
I do not observe anything wrong.

you can also mention the 'deltaImportQuery' and try it

someting like

     <entity name="table1" pk="id"
               query=" SELECT ID,MY_GUID
                               FROM activityLog
                               WHERE type in (11, 15)"
              deltaImportQuery="SELECT ID,MY_GUID
                               FROM activityLog
                               WHERE type in (11, 15) AND
id=${dataimporter.delta.ID}"
               deltaQuery="SELECT ID,MY_GUID
                               FROM activityLog
                               WHERE type in (11, 15) and created_date >
'${dataimporter.last_index_time}'">



On Tue, Dec 16, 2008 at 5:54 AM, sbutalia <[hidden email]> wrote:

>
> I have a parent entity that grabs a list of records of a certain type from 1
> table... and a sub-entity that queries another table to retrieve the actual
> data... for various reasons I cannot join the tables... the 2nd sql query
> converts the rows into an xml to be processed by a custom transformer (done
> due to the complex nature of the second table)
>
> Full-import works fine but delta-import is not adding any new records...
>
> Do I have to specify a deltaQuery for the sub-entity? What else might be
> goin on?
>
> <document name="doc">
>        <entity name="table1" pk="id"
>                query=" SELECT ID,MY_GUID
>                                FROM activityLog
>                                WHERE type in (11, 15)"
>                deltaQuery="
>                                SELECT ID,MY_GUID
>                                FROM activityLog
>                                WHERE type in (11, 15) and created_date >
> '${dataimporter.last_index_time}'">
>            <field column="MY_GUID" name="myGuid"/>
>                <entity name="table2" pk="ID"
>                                query="select dbms_xmlgen.getxml('
>                                                        select Name, Title, Description
>                                                                from metaDataTable
>                                                        where MY_GUID = ${table1.MY_GUID_ID}
>                                            ') mdrXmlClob
>                                                from dual
>                                          "
>                                transformer="MD.Solr.Utils.transformers.MDTransformer">
>                                <field column="Name" name="mdName"/>
>                                <field column="Title" name="mdTitle"/>
>                                <field column="Description" name="mdDescription"/>
>           </entity>
>        </entity>
> </document>
> --
> View this message in context: http://www.nabble.com/Parent-Child-Entity---DataImport-tp21024979p21024979.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>



--
--Noble Paul
Reply | Threaded
Open this post in threaded view
|

Re: Parent Child Entity - DataImport

bizzyim
I'ev had a chance to play with this more and noticed the query does run fine but it only updates the records that are already indexed it doesn't add new ones.

The only option that i'ev found so far is to do a full-import with the "clean=false" attribute and created_date > last_indexed_date...

Is there a better way?

Thanks

Noble Paul നോബിള്‍ नोब्ळ् wrote
I do not observe anything wrong.

you can also mention the 'deltaImportQuery' and try it

someting like

     <entity name="table1" pk="id"
               query=" SELECT ID,MY_GUID
                               FROM activityLog
                               WHERE type in (11, 15)"
              deltaImportQuery="SELECT ID,MY_GUID
                               FROM activityLog
                               WHERE type in (11, 15) AND
id=${dataimporter.delta.ID}"
               deltaQuery="SELECT ID,MY_GUID
                               FROM activityLog
                               WHERE type in (11, 15) and created_date >
'${dataimporter.last_index_time}'">



On Tue, Dec 16, 2008 at 5:54 AM, sbutalia <sbutalia@gmail.com> wrote:
>
> I have a parent entity that grabs a list of records of a certain type from 1
> table... and a sub-entity that queries another table to retrieve the actual
> data... for various reasons I cannot join the tables... the 2nd sql query
> converts the rows into an xml to be processed by a custom transformer (done
> due to the complex nature of the second table)
>
> Full-import works fine but delta-import is not adding any new records...
>
> Do I have to specify a deltaQuery for the sub-entity? What else might be
> goin on?
>
> <document name="doc">
>        <entity name="table1" pk="id"
>                query=" SELECT ID,MY_GUID
>                                FROM activityLog
>                                WHERE type in (11, 15)"
>                deltaQuery="
>                                SELECT ID,MY_GUID
>                                FROM activityLog
>                                WHERE type in (11, 15) and created_date >
> '${dataimporter.last_index_time}'">
>            <field column="MY_GUID" name="myGuid"/>
>                <entity name="table2" pk="ID"
>                                query="select dbms_xmlgen.getxml('
>                                                        select Name, Title, Description
>                                                                from metaDataTable
>                                                        where MY_GUID = ${table1.MY_GUID_ID}
>                                            ') mdrXmlClob
>                                                from dual
>                                          "
>                                transformer="MD.Solr.Utils.transformers.MDTransformer">
>                                <field column="Name" name="mdName"/>
>                                <field column="Title" name="mdTitle"/>
>                                <field column="Description" name="mdDescription"/>
>           </entity>
>        </entity>
> </document>
> --
> View this message in context: http://www.nabble.com/Parent-Child-Entity---DataImport-tp21024979p21024979.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>



--
--Noble Paul
Reply | Threaded
Open this post in threaded view
|

Re: Parent Child Entity - DataImport

Noble Paul നോബിള്‍  नोब्ळ्
Are the queries being fired wrong/different when you tried full-import?



On Tue, Dec 16, 2008 at 9:57 AM, sbutalia <[hidden email]> wrote:

>
> I'ev had a chance to play with this more and noticed the query does run fine
> but it only updates the records that are already indexed it doesn't add new
> ones.
>
> The only option that i'ev found so far is to do a full-import with the
> "clean=false" attribute and created_date > last_indexed_date...
>
> Is there a better way?
>
> Thanks
>
>
> Noble Paul നോബിള്‍ नोब्ळ् wrote:
>>
>> I do not observe anything wrong.
>>
>> you can also mention the 'deltaImportQuery' and try it
>>
>> someting like
>>
>>      <entity name="table1" pk="id"
>>                query=" SELECT ID,MY_GUID
>>                                FROM activityLog
>>                                WHERE type in (11, 15)"
>>               deltaImportQuery="SELECT ID,MY_GUID
>>                                FROM activityLog
>>                                WHERE type in (11, 15) AND
>> id=${dataimporter.delta.ID}"
>>                deltaQuery="SELECT ID,MY_GUID
>>                                FROM activityLog
>>                                WHERE type in (11, 15) and created_date >
>> '${dataimporter.last_index_time}'">
>>
>>
>>
>> On Tue, Dec 16, 2008 at 5:54 AM, sbutalia <[hidden email]> wrote:
>>>
>>> I have a parent entity that grabs a list of records of a certain type
>>> from 1
>>> table... and a sub-entity that queries another table to retrieve the
>>> actual
>>> data... for various reasons I cannot join the tables... the 2nd sql query
>>> converts the rows into an xml to be processed by a custom transformer
>>> (done
>>> due to the complex nature of the second table)
>>>
>>> Full-import works fine but delta-import is not adding any new records...
>>>
>>> Do I have to specify a deltaQuery for the sub-entity? What else might be
>>> goin on?
>>>
>>> <document name="doc">
>>>        <entity name="table1" pk="id"
>>>                query=" SELECT ID,MY_GUID
>>>                                FROM activityLog
>>>                                WHERE type in (11, 15)"
>>>                deltaQuery="
>>>                                SELECT ID,MY_GUID
>>>                                FROM activityLog
>>>                                WHERE type in (11, 15) and created_date >
>>> '${dataimporter.last_index_time}'">
>>>            <field column="MY_GUID" name="myGuid"/>
>>>                <entity name="table2" pk="ID"
>>>                                query="select dbms_xmlgen.getxml('
>>>                                                        select Name,
>>> Title, Description
>>>                                                                from
>>> metaDataTable
>>>                                                        where MY_GUID =
>>> ${table1.MY_GUID_ID}
>>>                                            ') mdrXmlClob
>>>                                                from dual
>>>                                          "
>>>
>>> transformer="MD.Solr.Utils.transformers.MDTransformer">
>>>                                <field column="Name" name="mdName"/>
>>>                                <field column="Title" name="mdTitle"/>
>>>                                <field column="Description"
>>> name="mdDescription"/>
>>>           </entity>
>>>        </entity>
>>> </document>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Parent-Child-Entity---DataImport-tp21024979p21024979.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> --Noble Paul
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Parent-Child-Entity---DataImport-tp21024979p21027045.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>



--
--Noble Paul