How to slow the crawler

Author Message
Mark Greenberg 11/30/2011 03:57 pm
I would like to know if there is any way to limit the growth of the file queue. First, it would help with the stability in suspend-to-file, for large websites. Also some sites monitor and block access to crawlers, and I would like to avoid this. I figure if I slow down the crawler, everybody is happy.
Mark Greenberg 11/30/2011 04:08 pm
Do you have any other ideas how to avoid being prevented as a crawler?
Oleg Chernavin 11/30/2011 04:16 pm
Yes. You may slow down it by specifying less connections in the Internet tab of the Ribbon. And also set a random delay between downloads. Just type in the corresponding setting:


This means random pause from 3 to 5 seconds.

Best regards,
Oleg Chernavin
MP Staff