keep connection alive for FTP

Author Message
K.lopps 09/15/2003 06:46 am
how can i make OEP to keep the connection to an FTP?s alive and not disconnect after every file transfered?

when downloading 20000 files OEP logs on 20000 times which is on some ftp?s seen as hammering and leads to bans.
Oleg Chernavin 09/15/2003 07:35 am
We made that feature in past, but we faced certain problems on some FTP sites. If you want, I can make a version for you to test.

Best regards,
Oleg Chernavin
MetaProducts corp.
K.lopps 09/15/2003 07:48 am
this would be great !
i also thought about using another tool for downloading the ftp-server but OEP would be my first choice because of the filter possabilities.

thanks!
Oleg Chernavin 09/15/2003 12:45 pm
OK. I just uploaded the updated version here:

http://www.metaproducts.com/download/betas/oep1304.zip

You will need to place Additional=ReuseFTP line after the starting Project URL:

ftp://ftp.server.com/
Additional=ReuseFTP

I hope this helps.

Oleg.
K.lopps 09/16/2003 04:24 am
great! this helps a lot! thank you very much
(no more bannings for me)

i have noticed that it looks to work perfect as long as the files are really on the server. but if a file is missing OEP seems to loose connection to the ftp and logs in again. perhaps this is an bug?

but it?s also ok this way as missing files on an ftp aren?t very often occuring.

here?s an except from a log of the ftp:
FTP0: Download complete.
FTP0: Changing directory...
FTP0: Download complete. Status: 0 Unknown.
FTP0: Changing directory...
FTP0: Setting binary connection...
FTP0: Requesting passive mode...
FTP0: Starting transfer...
FTP0: Getting length from listing.
FTP0: Length=19236.
FTP0: Requesting passive mode...
FTP0: Starting transfer...
FTP0: Download complete.
FTP0: Changing directory...
FTP0: Download complete. Status: 0 Unknown.
FTP0: Changing directory...
FTP0: Setting binary connection...
FTP0: Requesting passive mode...
FTP0: Starting transfer...
FTP0: Getting length from listing.
FTP0: Length=15993.
FTP0: Requesting passive mode...
FTP0: Starting transfer...
FTP0: Download complete.
FTP0: Changing directory...
FTP0: Download complete. Status: 0 Unknown.
FTP0: Changing directory...
FTP0: Setting ASCII mode...
FTP0: Requesting passive mode...
FTP0: Starting transfer...
FTP0: Download complete. Status: 200 OK.
FTP0: Setting binary connection...
FTP0: Setting binary connection...
FTP0: Requesting passive mode...
FTP0: Starting transfer...
FTP0: Getting length from listing.
FTP0: Getting file length...
FTP0: Requesting passive mode...
FTP0: Starting transfer...
FTP0: Read Transaction complete. Status: 404 Not Found.
FTP0: Connecting to host ftp.asstr.org...
FTP0: Host ftp.%%%.org connected. Waiting for ftp://ftp.%%%.org/deck/writer.txt.
FTP0: Logging on...
FTP0: Sending password...
FTP0: Changing directory...
FTP0: Setting binary connection...
FTP0: Requesting passive mode...
FTP0: Starting transfer...
FTP0: Getting length from listing.
FTP0: Length=10997.
FTP0: Requesting passive mode...
FTP0: Starting transfer...
FTP0: Download complete.
K.lopps 09/16/2003 05:43 am
i think i found another bug:

ftp connections seem to ignore the timeout settings of OEP.
i had it several times now that transfers were active with 0byte/s for some minutes without OEP aborting and retrying them. i had to kill these manually and then OEP continued in the queue
Oleg Chernavin 09/16/2003 11:05 am
I just uploaded updated file:

http://www.metaproducts.com/download/betas/oep1305.zip

404 issue should be fixed. Please let me know if timeouts are ignored while downloading or connecting to a server. Also, does this happen only with the FTP reusing mode or with normal FTP downloads as well?

Oleg.

K.lopps 09/16/2003 01:08 pm
> I just uploaded updated file:
>
> http://www.metaproducts.com/download/betas/oep1305.zip
>
> 404 issue should be fixed. Please let me know if timeouts are ignored while downloading or connecting to a server. Also, does this happen only with the FTP reusing mode or with normal FTP downloads as well?
>
> Oleg.
>
>
great! i?ll test the update and report back about how it works.

the not-timeouts were occuring while downloading only (at least i didn?t notice them while connecting). i don?t know if this also happens without the ftp-reuse mode because without it i get banned after a very short time.
Oleg Chernavin 09/17/2003 12:39 am
OK. Please test the new update and let me know its 404 and timeout situation. By the way, by default the timeout is 5 minutes (300 seconds).

Oleg.
K.lopps 09/17/2003 02:54 am
i just had some new 404 replies and OEP was hanging this time completely (even worse than the not-timeouts i described before).

however i looked a little more into this and noticed some things:
1. the files which OEP reported 404 aren?t really missing on the FTP (with another FTP program i could open them without problems)

2. the files were hanging (no timeout) for more than 10 minutes(before i aborted them).

3. after aborting (right click->abort) the files the message in the status switched from "...404..." to "Download complete ."

4. after aborting no new file was started in the thread, the "Download complete ." message was standing there endless.

5 and finally: an new except from an logfile:

FTP0: Download complete. Status: 200 OK.
FTP0: Setting binary connection...
FTP0: Setting binary connection...
FTP0: Requesting passive mode...
FTP0: Starting transfer...
FTP0: Getting length from listing.
FTP0: Getting file length...
FTP0: Requesting passive mode...
FTP0: Starting transfer...
FTP0: Read Transaction complete. Status: 404 Not Found.
(this log was the same for every 404), looks like something is going wrong sometimes (Setting binary connection... 2x).

here you have also an log for an 200 reply (so you can compare):
FTP0: Download complete. Status: 200 OK.
FTP0: Changing directory...
FTP0: Setting ASCII mode...
FTP0: Requesting passive mode...
FTP0: Starting transfer...
FTP0: Download complete. Status: 200 OK.
Oleg Chernavin 09/17/2003 03:44 am
Can you please write us directly to support@metaproducts.com to discuss the problem? I need to reproduce the situation on the real site to improve the behavior.

Thank you.

Oleg.
K.lopps 09/17/2003 03:44 am
as addition:
in the 404 log OEP tries to set an binary connection but the file which was to be transfered was an *.txt which should have transfered as an ASCII connection.
Oleg Chernavin 09/17/2003 03:45 am
Well, binary mode is more compatible, so it is safe to use it with any file, except directory listing.

Oleg.
K.lopps 09/17/2003 04:58 am
> Well, binary mode is more compatible, so it is safe to use it with any file, except directory listing.
>
> Oleg.
ok i just thought this might be the reason as for other txt files OEP was using ASCII.

i just sent you an mail with one project with the described problem. (i noticed that this isn?t something special for one ftp server but happens on others too)
Oleg Chernavin 09/17/2003 05:57 am
Thank you. I will work on it. But please excuse me if you don`t hear from me for a week - I am going to be out of town until the next week.

Oleg.
K.lopps 09/17/2003 06:41 am
> Thank you. I will work on it. But please excuse me if you don`t hear from me for a week - I am going to be out of town until the next week.
>
> Oleg.
that?s ok. i?ll use an ftp-client in the meantime then.
thanks in advance anyway!
Oleg Chernavin 09/19/2003 02:33 pm
You are welcome. By the way, I already got your E-mail. Thanks!

Oleg.