proxy dieing in the middle of download

Author Message
Stefan 08/22/2005 09:59 am
When working with a proxy list, what will happen when one or more of the proxies die in the middle of dowloading a file? Will the download be readded to the queue using a different proxy?

Thx

Stefan
Oleg Chernavin 08/22/2005 10:18 am
Yes, it will be added to the end of the queue to be redownloaded with another proxy later.

Best regards,
Oleg Chernavin
MP Staff
Stefan 08/22/2005 10:51 am
Thx for the info. one more question regarding the use of a proxy list:

With the number of retries set to unlimited will OE readd an unsucessfully retrieved url with another proxy to the queue after each retry ?

Thx

Stefan
Oleg Chernavin 08/23/2005 03:11 am
Yes, this is how it works.

Oleg.
Stefan 08/29/2005 08:05 am
After further using the proxy list feature, I ran into an something that might be worthwile to fix/change. Whenever a proxy server returns an error (like error 400 for a bad request) the item that was scheduled to be downloaded is being skipped by OE instead of being readded to the queue using another proxy server. I do not know of this is intended to be that way, but it would be nice if OE could handle that situation more appropiate and requeue the download request.

Thx

Stefan
Oleg Chernavin 08/30/2005 08:29 am
OK. I will change this now.

Thank you!

Oleg.
Stefan 08/30/2005 12:37 pm
Thank you. Please let me know when you have the changed version ready, I`d be happy to test it immediately

Stefan
Stefan 08/30/2005 12:44 pm
Thank you. Please let me know when you have the changed version ready, I`d be happy to test it immediately

Stefan
Oleg Chernavin 08/31/2005 09:45 am
I am sorry. I am travelling now and I am unable to upload new versions for a while. I should be back in our office after September 12th. I hope it is not a big trouble. As a workaround, setup any proxy in the standard Options dialog. This should make Offline Explorer to treat 400/5xx errors as proxy server errors and retry.

Oleg.
Stefan 08/31/2005 12:42 pm
ok, I will try that in the meantime until you can update OE.

Stefan
Stefan 09/01/2005 06:37 am
While you are at it, I would like to suggest to have some kind of control that makes it possible to have these rescheduled items added at the top of the queue instead of at the bottom. That would make things much more easier for my download tasks.

Thx

Stefan
Stefan 09/01/2005 06:49 am
One more issue. Is it possible to force OE not to open any http connection for files if "do not download exisiting files" is checked? I am working with a download queue of about 3.6 Million files and I noticed that whenever I restart the project, OE seems to open a http connection to the server for every file regardless if it`s already been downloaded or not. Would be nice if OE could be forced to just skip these http connections when a file in the download queue is already found on disk.

Thx

Stefan
Oleg Chernavin 09/02/2005 02:02 pm
Regarding adding to top - try, maybe Add=DepthFirst parameter will help.

The other issue - do you use URL Substitutes in this Project? If yes, this may give unexpected results when updating the site.

Oleg.
Stefan 09/02/2005 02:04 pm
Yes, now that I have finally worked out the correct URL substitutes I am using them everywhere in my projects.

Stefan
Stefan 09/02/2005 02:07 pm
But still it would be desirable to supress doing http connections. Just imagine what it means when I have 300.000 files in my download directory and I need to restart the project for any reason and instead of skipping the files on disk, for each one already downloaded a connection is opened.

Stefan
Oleg Chernavin 09/02/2005 02:17 pm
I usually do it this way - start the update and then press F9 to pause. You can also pause that individual Project to allow other downloads.

Oleg.
Stefan 09/02/2005 02:20 pm
I am not sure if we talk about the same thing here. I wanted to to know if OE can be made to only check the queue entries against files on the disk without prior opening of a http connection to the webserver.

Stefan
Oleg Chernavin 09/02/2005 02:42 pm
This works exactly this way - if you press Ctrl+F5 on a Project, OE first checks the file on the disk and only if it is missing, it starts HTTP connection.

Oleg.
Stefan 09/26/2005 11:28 am
Actually I did a test case and this problem has not been solved, if a proxy returns an error like 405 then OE will not readd the item to the queue instead it seems to get skipped. My test project was with 20 files to be downloaded, and every time I run it, some files are not downloaded. Can you please look into this?

Thank you

Stefan

> After further using the proxy list feature, I ran into an something that might be worthwile to fix/change. Whenever a proxy server returns an error (like error 400 for a bad request) the item that was scheduled to be downloaded is being skipped by OE instead of being readded to the queue using another proxy server. I do not know of this is intended to be that way, but it would be nice if OE could handle that situation more appropiate and requeue the download request.
>
> Thx
>
> Stefan
Oleg Chernavin 09/27/2005 04:10 am
Can you try the latest version?

http://www.metaproducts.com/download/betas/oep2118.zip

Oleg.
Stefan 09/27/2005 11:03 am
Ok, I tested it and I still don`t have the correct number of files downloaded in each run ( I am running batches of 2000 urls, of which are all available). Do you need me to log one these download sessions, so you can see what`s going ?

Thx

Stef

> Can you try the latest version?
>
> http://www.metaproducts.com/download/betas/oep2118.zip
>
> Oleg.
Oleg Chernavin 09/28/2005 02:02 am
I looked at the code. Errors from 405 to 499 are not considered as proxy errors. Can you please tell me what exact 4xx codes are used by proxies?

Oleg.
Stefan 09/28/2005 04:18 am
That explains it. I will get you the information on what erroes are returned from the proxies, so may be you can fine tune the behaviour.

Thx

Stefan

> I looked at the code. Errors from 405 to 499 are not considered as proxy errors. Can you please tell me what exact 4xx codes are used by proxies?
>
> Oleg.
Oleg Chernavin 09/28/2005 04:33 am
OK.

Oleg.