QE crashes after number of file sebing parsed reaches 6000-8000

Author Message
chris 03/10/2011 12:58 pm
hi,

I amusing OE to download a larger number of files, my operating systems are Vista and Windows 7.

If I set OE to download at 0 seconds delay between downloads, and over 90 files simultaneous downloading, OE crashes within the next 2-3 minutes, as soon as the number of files waiting to be parsed increases to 6000-8000.

It shuts down or freezes if I try to download at those high speeds, and sometimes I get an error saying:
Thread creation error: Not enough storage is available to process this command.

I understand this to be somehow related to the virtual memory allocated to the multiple threading processes, but my system ran before at such high speeds without a problem downloading 100,000 files in one project, without any freezing.

I don't know what to do, to make it run well again. I have reinstalled my system completely, both Windows 7 and Vista, and installed very few applications so that nothing could affect OE efficiency, still OE pro freezes every time I run it.

OE is supposed to run well at 0 delay time and 150-200 files at a time for at least 30 minutes straight, without freezing. It doesn't.

OE runs well only if I have a 1 sec delay or more, but that is not acceptable, delay must be 0.

Please let me know what you can do, I need this software to work.
I updated to last version available, the software is registered, all is fine with regards to version used.

One last detail, I would love to remove the parsing process altogether, I don't need OE parsing the files, but I don't know how to, maybe that could solve the problem. On another system were I tested OE with exact same settings, OE doesn't crash, works fine at 180 files simultaneously and 0 seconds delay. When downloads are complete, the parsing continues for a while, but I stop it as I don't need parsing. All files are fully downloaded and available after manually stopping parsing.

Why on my second system with the crashing issues, if I stop the parsing, the files are not fully saved on HDD? Files not fully parsed are missing altogether if I stop the parsing? Why stopping parsing of files renders files inaccessible on one system but not on the other?

please reply
thanks for any input that may solve this crashing issue.

Chris
Oleg Chernavin 03/11/2011 10:00 am
Chris, parsing is important - Offline Explorer has to find all links in HTML files to follow them. Yes, parsing speed can be sometimes lower than download speed. In fact, Enterprise edition would use all CPU cores to speed up parsing. Otherwise, please lower connections or set 1 second delay to balance the download and parsing speed.

Best regards,
Oleg Chernavin
MP Staff