Downloading same file over and over . . .

Harry
03/29/2006 07:00 am
I`ve OE Pro to download a very large site and every time it seems to have completed it`s run (the queue reaches zero), I run the project again to double check and sure enough, it finds another 100,000 or more files to add to the queue.

The problem is that most of these files already exist when I search the hard drive for them. The only difference is that they are in the sub directories that are created when a directory has more than 1,000 files in it.

I have `do not download files if they already exist` enabled and I also have `do not overload directories` selected so that when a directory reaches 1,000 files, it generates a sub directory.

Is this a limit with the number of links in an entire project OE Pro can handle? I know OE Enterprise can handle 100,000,000 but I would imagine that OE Pro should be able to handle a fraction of a percentage of that (500,000 to 1,000,000 ?).

This makes it difficult to know which files I still need to get and which really already exist (and I`m not going to check half a million files by hand). It will certainly make updating the mirror incredibly difficult.
Oleg Chernavin
03/29/2006 07:00 am
Yes, I am aware of this problem with the new option (Preventing directories from overloading). I am working on that and I hope to publish an updated version shortly.

Best regards,
Oleg Chernavin
MetaProducts corp.
Oleg Chernavin
03/29/2006 07:00 am
It looks like I just fixed the bug. Please download the updated version here:
http://www.metaproducts.com/download/betas/oep988.zip

Please unzip the file and replace the old oe.exe file with the new one.

Please let me know how it works.

Oleg.