Reading video files from HDFS using OpenCV

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Reading video files from HDFS using OpenCV

Madhav Sharan
Hi hadoop users,

I need some help in reading video files from HDFS. I am processing videos using OpenCV and opencv is not able to read video file from hdfs. I tried prefixing path with "hdfs://" but it still does not works. Is there a way we can directly read from HDFS using opencv? 


Reply | Threaded
Open this post in threaded view
|

Re: Reading video files from HDFS using OpenCV

Ron Gonzalez-2
You can create a stream for an HDFS file and have OpenCV read from the stream.
Other option is to create a yarn job and you can resource localize the HDFS file so you can run OpenCV just like any other application.

Thanks,
Ron

On Jul 24, 2016, at 12:03 PM, Madhav Sharan <[hidden email]> wrote:

Hi hadoop users,

I need some help in reading video files from HDFS. I am processing videos using OpenCV and opencv is not able to read video file from hdfs. I tried prefixing path with "hdfs://" but it still does not works. Is there a way we can directly read from HDFS using opencv? 



Reply | Threaded
Open this post in threaded view
|

Re: Reading video files from HDFS using OpenCV

Madhav Sharan
Hi Ron, Thanks for replying.

Unfortunately I could not find a VideoCapture method accepting stream input.

I will look into second option. Will it be similar to copying file from hdfs to a tmp directory and then using tmp file?

--
Madhav Sharan


On Sun, Jul 24, 2016 at 12:08 PM, Ron's Yahoo! <[hidden email]> wrote:
You can create a stream for an HDFS file and have OpenCV read from the stream.
Other option is to create a yarn job and you can resource localize the HDFS file so you can run OpenCV just like any other application.

Thanks,
Ron

On Jul 24, 2016, at 12:03 PM, Madhav Sharan <[hidden email]> wrote:

Hi hadoop users,

I need some help in reading video files from HDFS. I am processing videos using OpenCV and opencv is not able to read video file from hdfs. I tried prefixing path with "hdfs://" but it still does not works. Is there a way we can directly read from HDFS using opencv? 




Reply | Threaded
Open this post in threaded view
|

Re: Reading video files from HDFS using OpenCV

Ron Gonzalez-2

In a manner of speaking. I would imagine that you would like to take advantage of resource management that comes with yarn. If you're planning to make this a product that your customers can submit jobs to do some image processing, then you're better off creating a yarn application master that would handle taking files from Hdfs, localize then process as a local process.

If not and more of a one-time effort, then it's as simple as:

hadoop fs -copyToLocal hdfspath localpath

You can then do whatever you want to do with this local file.

Thanks,
Ron


On Jul 24, 2016 12:28 PM, Madhav Sharan <[hidden email]> wrote:
Hi Ron, Thanks for replying.

Unfortunately I could not find a VideoCapture method accepting stream input.

I will look into second option. Will it be similar to copying file from hdfs to a tmp directory and then using tmp file?

--
Madhav Sharan


On Sun, Jul 24, 2016 at 12:08 PM, Ron's Yahoo! <[hidden email]> wrote:
You can create a stream for an HDFS file and have OpenCV read from the stream.
Other option is to create a yarn job and you can resource localize the HDFS file so you can run OpenCV just like any other application.

Thanks,
Ron

On Jul 24, 2016, at 12:03 PM, Madhav Sharan <[hidden email]> wrote:

Hi hadoop users,

I need some help in reading video files from HDFS. I am processing videos using OpenCV and opencv is not able to read video file from hdfs. I tried prefixing path with "hdfs://" but it still does not works. Is there a way we can directly read from HDFS using opencv? 


[0] Video capture code -  <a href="https://urldefense.proofpoint.com/v2/url?u&#61;https-3A__github.com_smadha_pooled-5Ftime-5Fseries_blob_master_src_main_java_gov_nasa_jpl_memex_pooledtimeseries_PoT.java-23L441&amp;d&#61;DQMFAg&amp;c&#61;clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&amp;r&#61;DhBa2eLkbd4gAFB01lkNgg&amp;m&#61;9cu9u3AmZ3FP8PejFB3CInJW4mTPFq1H-B3USMRuP8E&amp;s&#61;Mr2L2gdpcCZNngViWbS0XOGfcb8oP98lDKJLzJoyMXE&amp;e&#61;">https://github.com/smadha/pooled_time_series/blob/master/src/main/java/gov/nasa/jpl/memex/pooledtimeseries/PoT.java#L441

[1] TMP file solution - <a href="https://urldefense.proofpoint.com/v2/url?u&#61;https-3A__github.com_smadha_pooled-5Ftime-5Fseries_blob_master_src_main_java_gov_nasa_jpl_memex_pooledtimeseries_OpticalTimeSeries.java-23L66&amp;d&#61;DQMFAg&amp;c&#61;clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&amp;r&#61;DhBa2eLkbd4gAFB01lkNgg&amp;m&#61;9cu9u3AmZ3FP8PejFB3CInJW4mTPFq1H-B3USMRuP8E&amp;s&#61;Xas8nIDCouC0_Qx2RiQDDV9okicT_KWd2ZPV6gTEWzg&amp;e&#61;">https://github.com/smadha/pooled_time_series/blob/master/src/main/java/gov/nasa/jpl/memex/pooledtimeseries/OpticalTimeSeries.java#L66

--
Madhav Sharan




--------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Reading video files from HDFS using OpenCV

Madhav Sharan
Thanks a lot Ron. It helps

--
Madhav Sharan


On Sun, Jul 24, 2016 at 2:19 PM, Ron Gonzalez <[hidden email]> wrote:

In a manner of speaking. I would imagine that you would like to take advantage of resource management that comes with yarn. If you're planning to make this a product that your customers can submit jobs to do some image processing, then you're better off creating a yarn application master that would handle taking files from Hdfs, localize then process as a local process.

If not and more of a one-time effort, then it's as simple as:

hadoop fs -copyToLocal hdfspath localpath

You can then do whatever you want to do with this local file.

Thanks,
Ron


On Jul 24, 2016 12:28 PM, Madhav Sharan <[hidden email]> wrote:
Hi Ron, Thanks for replying.

Unfortunately I could not find a VideoCapture method accepting stream input.

I will look into second option. Will it be similar to copying file from hdfs to a tmp directory and then using tmp file?

--
Madhav Sharan


On Sun, Jul 24, 2016 at 12:08 PM, Ron's Yahoo! <[hidden email]> wrote:
You can create a stream for an HDFS file and have OpenCV read from the stream.
Other option is to create a yarn job and you can resource localize the HDFS file so you can run OpenCV just like any other application.

Thanks,
Ron

On Jul 24, 2016, at 12:03 PM, Madhav Sharan <[hidden email]> wrote:

Hi hadoop users,

I need some help in reading video files from HDFS. I am processing videos using OpenCV and opencv is not able to read video file from hdfs. I tried prefixing path with "hdfs://" but it still does not works. Is there a way we can directly read from HDFS using opencv?