HttpDataSource questions

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

HttpDataSource questions

Jon Baer
Hi,

On the wiki it says that url attribute can be templatized but Im not  
sure how that happens, do you I need to create something read from a  
database column in order to use that type of function?  ie Id like to  
run over some RSS feeds for multiple URLs (~ 30), do I need to copy 1  
per URL I want to read or is there an easier method?

Also does anything currently get compared when doing a delta-import  
for these types of data sources?  Does the dataimport.properties  
compare itself to anything?  (ie a pubdate on RSS, etc).

Thanks!

- Jon
Reply | Threaded
Open this post in threaded view
|

Re: HttpDataSource questions

Noble Paul നോബിള്‍  नोब्ळ्
On Fri, Jul 11, 2008 at 11:46 PM, Jon Baer <[hidden email]> wrote:
> Hi,
>
> On the wiki it says that url attribute can be templatized but Im not sure
> how that happens, do you I need to create something read from a database
> column in order to use that type of function?  ie Id like to run over some
> RSS feeds for multiple URLs (~ 30), do I need to copy 1 per URL I want to
> read or is there an easier method?
It is a simple passthrough of parameters you passed over to the http request.
eg:
If you want to read a feed with say an extra attribute (date) which
you will know in runtime
you can make the http request with command=full-import&date=some_date.
The date can be consumed in the url as
url="http://xyz.com?a=b&thedate=${dataimporter.request.date}


if you could give me a sample on how the diffrent urls look like I may
be able to suggest you something. Actually there are many ways to
achieve it.

>
> Also does anything currently get compared when doing a delta-import for
> these types of data sources?  Does the dataimport.properties compare itself
> to anything?  (ie a pubdate on RSS, etc).
>
> Thanks!
>
> - Jon
>



--
--Noble Paul
Reply | Threaded
Open this post in threaded view
|

Re: HttpDataSource questions

Jon Baer
Ahhh very cool, did not realize that one.  I was actually able to use  
a db entity over the http entity so I pulled a list of subdomains and  
include it that way.

One small *possible* feature request (or is it possible already) is to  
load entities by name?  For example if I wanted to cron up something  
like:

?command=delta-import&entity=newyork,chicago,etc

Im guessing this is something that would be possible using  
javax.script but Im on 5 and ScriptProcessor does not seem to work w/  
BSF.  It would be nice to have something like that *without* having to  
add anything like a conditional tag.

- Jon

On Jul 12, 2008, at 12:38 AM, Noble Paul നോബിള്‍  
नोब्ळ् wrote:

> On Fri, Jul 11, 2008 at 11:46 PM, Jon Baer <[hidden email]> wrote:
>> Hi,
>>
>> On the wiki it says that url attribute can be templatized but Im  
>> not sure
>> how that happens, do you I need to create something read from a  
>> database
>> column in order to use that type of function?  ie Id like to run  
>> over some
>> RSS feeds for multiple URLs (~ 30), do I need to copy 1 per URL I  
>> want to
>> read or is there an easier method?
> It is a simple passthrough of parameters you passed over to the http  
> request.
> eg:
> If you want to read a feed with say an extra attribute (date) which
> you will know in runtime
> you can make the http request with command=full-import&date=some_date.
> The date can be consumed in the url as
> url="http://xyz.com?a=b&thedate=${dataimporter.request.date}
>
>
> if you could give me a sample on how the diffrent urls look like I may
> be able to suggest you something. Actually there are many ways to
> achieve it.
>
>>
>> Also does anything currently get compared when doing a delta-import  
>> for
>> these types of data sources?  Does the dataimport.properties  
>> compare itself
>> to anything?  (ie a pubdate on RSS, etc).
>>
>> Thanks!
>>
>> - Jon
>>
>
>
>
> --
> --Noble Paul

Reply | Threaded
Open this post in threaded view
|

Re: HttpDataSource questions

Shalin Shekhar Mangar
Hi Jon,

Yes it is possible already :)

Just add a request parameter "entity". For example --
command=full-import&entity=entity1&entity=entity2 which will run full
import for entity1 and entity2. Note that this works for root entities
(top level entities) only.

On Sat, Jul 12, 2008 at 10:39 AM, Jon Baer <[hidden email]> wrote:

> Ahhh very cool, did not realize that one.  I was actually able to use a db
> entity over the http entity so I pulled a list of subdomains and include it
> that way.
>
> One small *possible* feature request (or is it possible already) is to load
> entities by name?  For example if I wanted to cron up something like:
>
> ?command=delta-import&entity=newyork,chicago,etc
>
> Im guessing this is something that would be possible using javax.script but
> Im on 5 and ScriptProcessor does not seem to work w/ BSF.  It would be nice
> to have something like that *without* having to add anything like a
> conditional tag.
>
> - Jon
>
> On Jul 12, 2008, at 12:38 AM, Noble Paul നോബിള്‍ नोब्ळ् wrote:
>
>> On Fri, Jul 11, 2008 at 11:46 PM, Jon Baer <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> On the wiki it says that url attribute can be templatized but Im not sure
>>> how that happens, do you I need to create something read from a database
>>> column in order to use that type of function?  ie Id like to run over
>>> some
>>> RSS feeds for multiple URLs (~ 30), do I need to copy 1 per URL I want to
>>> read or is there an easier method?
>>
>> It is a simple passthrough of parameters you passed over to the http
>> request.
>> eg:
>> If you want to read a feed with say an extra attribute (date) which
>> you will know in runtime
>> you can make the http request with command=full-import&date=some_date.
>> The date can be consumed in the url as
>> url="http://xyz.com?a=b&thedate=${dataimporter.request.date}
>>
>>
>> if you could give me a sample on how the diffrent urls look like I may
>> be able to suggest you something. Actually there are many ways to
>> achieve it.
>>
>>>
>>> Also does anything currently get compared when doing a delta-import for
>>> these types of data sources?  Does the dataimport.properties compare
>>> itself
>>> to anything?  (ie a pubdate on RSS, etc).
>>>
>>> Thanks!
>>>
>>> - Jon
>>>
>>
>>
>>
>> --
>> --Noble Paul
>
>



--
Regards,
Shalin Shekhar Mangar.