[GitHub] [nutch] sebastian-nagel opened a new pull request #527: NUTCH-2496 Speed up link inversion step in crawling script

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[GitHub] [nutch] sebastian-nagel opened a new pull request #527: NUTCH-2496 Speed up link inversion step in crawling script

GitBox

sebastian-nagel opened a new pull request #527:
URL: https://github.com/apache/nutch/pull/527


   - disable URL filtering and normalizing when calling invertlinks in bin/crawl
   
   - add note that the steps invertlinks, dedup, index could also be done outside the loop over all segments created in the loop iterations
   
   - move webgraph construction (commented out anyway) outside the loop because it's done over all available segments


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[hidden email]