File Copies option

Author Message
Steve Sieloff 11/20/2003 12:25 pm
Oleg --

I have noticed again that whenever I use the "make copies of old files instead of overwriting them" and use the Keep old files: 9999, OE Pro really slows down in its downloading speeds around 400-500 files. Even if I use the Prevent directory overloading option which limits no more than 999 files in a folder, OE keeps slowing down until it reset to a new folder (one of the overflow default folders) -- then the speed picks back up.

It appears that the keep old files option forces the most recent file to be the lowest numbered file (search.asp(1) rather than search.asp(x) where x is the next sequential number of files downloaded). Could you please provide an option to alter the renumbering/renaming of all files in the directory and invert the numbering so that the first file downloaded is asp(1) and never renumbered/renamed -- currently 1 becomes 2, 2 becomes 3, 3 becomes 4, etc for each newly downloaded file which is where I think E Pro is spending so much time as the directory file count grows -- the downloads are the same as is the parsing requirements so I cannot see any other reason for the consistant slowdowns.

Thanks in advance for your time and consideration!

Steve
Oleg Chernavin 11/20/2003 03:45 pm
Yes, this is so - the slowdown is being caused by many renames.

There is an option to overcome it - if you select "Use date and time", there will be no mass-renaming and filenames will be still in the order.

I hope this helps.

Best regards,
Oleg Chernavin
MP Staff
Steve Sieloff 11/20/2003 03:51 pm
Oleg --

I tried this option but with several threads (even just 1), I get multiple files that download with same time stamp (it happens more often than I would have imagined -- a testament to OE speed and a function of the receiving server processing speed!) and end up overlaying existing files already downloaded ... resulting in less than the total expect files in the download folder when project completes .. is there any other option that would speed up the download by eliminating the renames but would also not result in possible naming collisions?

Thanks,

Steve


> Yes, this is so - the slowdown is being caused by many renames.
>
> There is an option to overcome it - if you select "Use date and time", there will be no mass-renaming and filenames will be still in the order.
>
> I hope this helps.
>
> Best regards,
> Oleg Chernavin
> MP Staff
Oleg Chernavin 11/21/2003 04:06 am
Steve,

We have two ways here:

1. I could make an option to make reverse-rename, so OE doesn`t rename all files, but finds the maximum available number for the loaded file and renames to it. However it will still be slow (although faster than now), because it will take time to lookup all occupied filenames. Also, when the number reaches maximum, it is still required to rename all other files to make 1 extra available number. Or to remove file(0).htm, file(1).htm numbers, etc.

2. You can set 1 second delay between downloads. This will slow down the downloading, but you would have purely unique names with the timestamp.

Oleg.
Steve Sieloff 11/21/2003 10:14 am
Oleg --

Why would you need to "pre determine" the maximum number of files? If you just allowed OE to keep track of a sequential number tied to the acutal number of files downloaded to a directory, you could just use it and allow for an undetermined number of files. If the "prevent directory overflow" option were used you could test the number for 999, create a new directory, reset the number to 1, and begin downloading the files asp(1), asp(2) ... to the new directory. If the overflow option were not used, you would start the increment number at 1 and bump by 1 until all the queued files were processed -- independent of what the total would be.

Of course, you might have to put a hard physical limit to the number of files in a directory as Windows might choke (say 32,766 for example) but then you could just "force" a new overflow direcotry to acommodate the additiaonl files.

What am I missing here?

Thanks again for the dialog and willingness to modify the routines!

Steve


> Steve,
>
> We have two ways here:
>
> 1. I could make an option to make reverse-rename, so OE doesn`t rename all files, but finds the maximum available number for the loaded file and renames to it. However it will still be slow (although faster than now), because it will take time to lookup all occupied filenames. Also, when the number reaches maximum, it is still required to rename all other files to make 1 extra available number. Or to remove file(0).htm, file(1).htm numbers, etc.
>
> 2. You can set 1 second delay between downloads. This will slow down the downloading, but you would have purely unique names with the timestamp.
>
> Oleg.
Oleg Chernavin 11/24/2003 07:56 am
> What am I missing here?

The problem is that I still have to find the maximum number, which takes time. Also, if the user puts maximum, say, 1000 copies, I cannot keep more than 1000 of old copies - they need to be automatically removed.

But can you please tell me how do you use File Copies? Maybe there is another better way to setup the Project? You can write us to support@metaproducts.com if this is too private to explain in the forum.

Oleg.