How to slow the crawler
|Mark Greenberg||11/30/2011 03:57 pm|
|I would like to know if there is any way to limit the growth of the file queue. First, it would help with the stability in suspend-to-file, for large websites. Also some sites monitor and block access to crawlers, and I would like to avoid this. I figure if I slow down the crawler, everybody is happy.|
|Mark Greenberg||11/30/2011 04:08 pm|
|Do you have any other ideas how to avoid being prevented as a crawler?|
|Oleg Chernavin||11/30/2011 04:16 pm|
|Yes. You may slow down it by specifying less connections in the Internet tab of the Ribbon. And also set a random delay between downloads. Just type in the corresponding setting:
This means random pause from 3 to 5 seconds.