|Brad Konia||12/01/2004 03:21 pm|
|I am currently using OE Enterprise to spider a Yahoo category and download all the webites in that category and its subcategories. This project has been running for about twelve hours, but the longer it runs, the slower it seems to be getting.
So far, it has downloaded 75,105 files, it is parsing 28,085 and it has 306 in queue. OE is pretty much using 100% of the CPU and is currently at 219MB of memory usage.
It seems strange to me that 75,000 files is putting such a strain on an application that is supposed to be able to handle up to 100,000,000 files!
Any thoughts or suggestions?
|Oleg Chernavin||12/02/2004 06:17 am|
|Parsing files is a really time-consuming process. Offline Explorer supports a lot of methods to extract links from various file types - more than any other competing software. Also, its flexible Project settings make it more complex to decide whether to follow each of the extracted links or not.
To speed up parsing files, I would suggest you to uncheck "Evaluate script calculations" and "Explore HTML forms" settings in the Project Properties dialog | Advanced.
Besides, you need to check the "Prevent directories from overloading" box in the Options dialog | File Locations. Otherwise storing thousands of files in a single directory makes Windows very slow when accessing that folder.