When working with a proxy list, what will happen when one or more of the proxies die in the middle of dowloading a file? Will the download be readded to the queue using a different proxy?
Thx
Stefan
Thx
Stefan
Yes, it will be added to the end of the queue to be redownloaded with another proxy later.
Best regards,
Oleg Chernavin
MP Staff
Best regards,
Oleg Chernavin
MP Staff
Thx for the info. one more question regarding the use of a proxy list:
With the number of retries set to unlimited will OE readd an unsucessfully retrieved url with another proxy to the queue after each retry ?
Thx
Stefan
With the number of retries set to unlimited will OE readd an unsucessfully retrieved url with another proxy to the queue after each retry ?
Thx
Stefan
Yes, this is how it works.
Oleg.
Oleg.
After further using the proxy list feature, I ran into an something that might be worthwile to fix/change. Whenever a proxy server returns an error (like error 400 for a bad request) the item that was scheduled to be downloaded is being skipped by OE instead of being readded to the queue using another proxy server. I do not know of this is intended to be that way, but it would be nice if OE could handle that situation more appropiate and requeue the download request.
Thx
Stefan
Thx
Stefan
OK. I will change this now.
Thank you!
Oleg.
Thank you!
Oleg.
Thank you. Please let me know when you have the changed version ready, I`d be happy to test it immediately
Stefan
Stefan
Thank you. Please let me know when you have the changed version ready, I`d be happy to test it immediately
Stefan
Stefan
I am sorry. I am travelling now and I am unable to upload new versions for a while. I should be back in our office after September 12th. I hope it is not a big trouble. As a workaround, setup any proxy in the standard Options dialog. This should make Offline Explorer to treat 400/5xx errors as proxy server errors and retry.
Oleg.
Oleg.
ok, I will try that in the meantime until you can update OE.
Stefan
Stefan
While you are at it, I would like to suggest to have some kind of control that makes it possible to have these rescheduled items added at the top of the queue instead of at the bottom. That would make things much more easier for my download tasks.
Thx
Stefan
Thx
Stefan
One more issue. Is it possible to force OE not to open any http connection for files if "do not download exisiting files" is checked? I am working with a download queue of about 3.6 Million files and I noticed that whenever I restart the project, OE seems to open a http connection to the server for every file regardless if it`s already been downloaded or not. Would be nice if OE could be forced to just skip these http connections when a file in the download queue is already found on disk.
Thx
Stefan
Thx
Stefan
Regarding adding to top - try, maybe Add=DepthFirst parameter will help.
The other issue - do you use URL Substitutes in this Project? If yes, this may give unexpected results when updating the site.
Oleg.
The other issue - do you use URL Substitutes in this Project? If yes, this may give unexpected results when updating the site.
Oleg.
Yes, now that I have finally worked out the correct URL substitutes I am using them everywhere in my projects.
Stefan
Stefan
But still it would be desirable to supress doing http connections. Just imagine what it means when I have 300.000 files in my download directory and I need to restart the project for any reason and instead of skipping the files on disk, for each one already downloaded a connection is opened.
Stefan
Stefan
I usually do it this way - start the update and then press F9 to pause. You can also pause that individual Project to allow other downloads.
Oleg.
Oleg.
I am not sure if we talk about the same thing here. I wanted to to know if OE can be made to only check the queue entries against files on the disk without prior opening of a http connection to the webserver.
Stefan
Stefan
This works exactly this way - if you press Ctrl+F5 on a Project, OE first checks the file on the disk and only if it is missing, it starts HTTP connection.
Oleg.
Oleg.
Actually I did a test case and this problem has not been solved, if a proxy returns an error like 405 then OE will not readd the item to the queue instead it seems to get skipped. My test project was with 20 files to be downloaded, and every time I run it, some files are not downloaded. Can you please look into this?
Thank you
Stefan
> After further using the proxy list feature, I ran into an something that might be worthwile to fix/change. Whenever a proxy server returns an error (like error 400 for a bad request) the item that was scheduled to be downloaded is being skipped by OE instead of being readded to the queue using another proxy server. I do not know of this is intended to be that way, but it would be nice if OE could handle that situation more appropiate and requeue the download request.
>
> Thx
>
> Stefan
Thank you
Stefan
> After further using the proxy list feature, I ran into an something that might be worthwile to fix/change. Whenever a proxy server returns an error (like error 400 for a bad request) the item that was scheduled to be downloaded is being skipped by OE instead of being readded to the queue using another proxy server. I do not know of this is intended to be that way, but it would be nice if OE could handle that situation more appropiate and requeue the download request.
>
> Thx
>
> Stefan
Can you try the latest version?
http://www.metaproducts.com/download/betas/oep2118.zip
Oleg.
http://www.metaproducts.com/download/betas/oep2118.zip
Oleg.
Ok, I tested it and I still don`t have the correct number of files downloaded in each run ( I am running batches of 2000 urls, of which are all available). Do you need me to log one these download sessions, so you can see what`s going ?
Thx
Stef
> Can you try the latest version?
>
> http://www.metaproducts.com/download/betas/oep2118.zip
>
> Oleg.
Thx
Stef
> Can you try the latest version?
>
> http://www.metaproducts.com/download/betas/oep2118.zip
>
> Oleg.
I looked at the code. Errors from 405 to 499 are not considered as proxy errors. Can you please tell me what exact 4xx codes are used by proxies?
Oleg.
Oleg.
That explains it. I will get you the information on what erroes are returned from the proxies, so may be you can fine tune the behaviour.
Thx
Stefan
> I looked at the code. Errors from 405 to 499 are not considered as proxy errors. Can you please tell me what exact 4xx codes are used by proxies?
>
> Oleg.
Thx
Stefan
> I looked at the code. Errors from 405 to 499 are not considered as proxy errors. Can you please tell me what exact 4xx codes are used by proxies?
>
> Oleg.
OK.
Oleg.
Oleg.