queue just takes too long

Author Message
chris su 05/01/2005 07:20 am
downloading a site with 100,000 files in the queue. Then oe reports a thread error, stops, hangs. I go back to the desktop. But manage to restart, however it starts to queue the files again 100,000 of them and checks but does not download (good but...). Tried Properties>URL filters `skip following urls while downloading` but OE still checks EVERY file. Anyway to just skip it to the directory where it was last been before it hanged?
coriolis 05/01/2005 06:49 pm
> downloading a site with 100,000 files in the queue. Then oe reports a thread error, stops, hangs. I go back to the desktop. But manage to restart, however it starts to queue the files again 100,000 of them and checks but does not download (good but...). Tried Properties>URL filters `skip following urls while downloading` but OE still checks EVERY file. Anyway to just skip it to the directory where it was last been before it hanged?

Not really an answer to your question but perhaps you could use the "additional=depthfirst" option to keep your queue more manageable. Rather than queuing up all the pages and sub-pages etc until the final content, it will probe one path down to the final content and then work its way back up to the next so you won`t have all the mid-pages in the queue waiting to be processed but instead only the ones currently being processed -- i.e. if you have 10 links per page and your ultimate content is five levels deep, you`ll have at most 10+10+10+10+10 URLs in the queue instead of 10*10*10*10*10. It your queue explodes into the tens of thousands anyway be sure to specify a reasonable maximum depth in your project properties as well.
chris su 05/02/2005 07:07 am
> Not really an answer to your question but perhaps you could use the "additional=depthfirst" option to keep your queue more manageable. Rather than queuing up all the pages and sub-pages etc until the final content, it will probe one path down to the final content and then work its way back up to the next so you won`t have all the mid-pages in the queue waiting to be processed but instead only the ones currently being processed -- i.e. if you have 10 links per page and your ultimate content is five levels deep, you`ll have at most 10+10+10+10+10 URLs in the queue instead of 10*10*10*10*10. It your queue explodes into the tens of thousands anyway be sure to specify a reasonable maximum depth in your project properties as well.

what i meant was, while i`m satisfied the queue reaches AND finishes around 120,000 files, it HAS to go thru ALL of them (checking, not downloading this is SLOW) even though some of the files were already downloaded! anyway to skip this process...isn`t that what those .wd3 and .wqdh files are supposed to do?
So as not to cause any crash, i decided to `suspend to file`, then `backup project`, this pauses OE, then after a few minutes i press f5 (hope that was right)...then OE restarts queueing again from 120,000!!! MY GOD, isn`t this program supposed to be smart? or do i have to be it`s developer to be able to admire its power?
chris su 05/02/2005 07:29 am
...finally i think i got it....BUT it is REALLY crazy, so after ctrl-alt-del, deleting the project, I restored the backup (really SLOW because it`s compressed, right? i think the project is >100MB), then `resume from file`, it now starts from around 80,000 files. GOOD, but is this good design? why not have both `suspend and backup`, isn`t OE smart enough to know that there are still files to download...since there is a queue then backup should include resume from a certain point without goin thru all that pain? Those guys managing the servers we traffic a lot are goin to figure it out and forbid OE activity, and then what do we get for our $70???

Another thing...the wizard is just plain DUMB, for example it asks for a `project name` you give it one, then it lobs it into the default download folder...why not give it it`s own `project name` folder? i find this ANNOYING especially when i forget to specify another project into another folder...even after the wizard ends with `specify advanced project settings` this feature is not there! one has to close the dialog, click the `options` icon then go thru all the hogwash! It`s just NOT GOOD DESIGN fellas! ---------- $70 WOW!!! please improve. This feature alone is worth $70!
Oleg Chernavin 05/02/2005 08:07 am
Chris,

I am sorry for the frustration you have. I would suggest you to press Ctrl+F5 to start unfinished project download - this will not force Offline Explorer to check every file online whether it has changed since the last download or not.

Regarding the place where the downloaded files are stored - by default, all sites go to the same folder. If you want several download locations, please create several Project Folders and set their own download directories using the Folder Properties.

Best regards,
Oleg Chernavin
MP Staff
chris su 05/08/2005 03:37 am
> Regarding the place where the downloaded files are stored - by default, all sites go to the same folder. If you want several download locations, please create several Project Folders and set their own download directories using the Folder Properties.
>
> Best regards,
> Oleg Chernavin
> MP Staff


...that feature should be included with the wizard...it`s supposed to be a wizard - smart. But no, perhaps in a future version it will be smart, right now while one can specify a project name it does not create a separate project folder with that name...just not smart nor wizardly.
Oleg Chernavin 05/09/2005 04:58 am
Well, we made Offline Explorer Pro in an opposite way - it doesn`t create subfolders for different Projects with a purpose. Of course, there are pros and cons of each approach - with subfolders per Project and without them.

The beggest benefit of our approach is that all downloaded Web sites go to the same place and when you download different sites with cross-links to each other, you will be able to browse them offline altogether.

For example, if a link in a file from the Project A points to a file loaded by Project B, you can click that link and you will browse site B flawlessly. Besides that, if several Projects load sites with the same files, our approach improves download performance, because these common files should not be loaded several times again and again - they will be simply checked for changes and if they are not modified since the last download, they will be not redownloaded.

We thought a lot about what approach to choose and our analisys showed us that most of offline browsers users prefer our approach. The necessity to create different folders per each Project is quite rare and Offline Explorer Pro/Enterprise editions offer this ability by having the Download Directory setting in the Folder Properties dialog.

I would also appreciate if you share your thoughts on why separate download folders per Project are necessary. This will help us to make our software better.

Thank you!

Oleg.
chris su 05/12/2005 08:12 am
How about putting a checkbox just underneath the first page of the wizard..."Create Folder with this Name" kind of thing, that way i don`t have to go all the way to Options just to change things.

> The beggest benefit of our approach is that all downloaded Web sites go to the same place and when you download different sites with cross-links to each other, you will be able to browse them offline altogether.

That`s also the bigest problem because it mish-mashes everything together.
Besides what i downloaded arent related to each other anyway...astronomy and `simple plan` you get that?

>
> For example, if a link in a file from the Project A points to a file loaded by Project B, you can click that link and you will browse site B flawlessly. Besides that, if several Projects load sites with the same files, our approach improves download performance, because these common files should not be loaded several times again and again - they will be simply checked for changes and if they are not modified since the last download, they will be not redownloaded.
>
> We thought a lot about what approach to choose and our analisys showed us that most of offline browsers users prefer our approach.

Yes, assuming a lot of them want to download the ENTIRE internet.
But there should be CHOICE!


>The necessity to create different folders per each Project is quite rare and Offline Explorer >Pro/Enterprise editions offer this ability by having the Download Directory setting in the Folder Properties dialog.
>

Rare? assuming everyone thinks like a developer, what raises my temperature is when i forget to do this before i press `download`, of course you can always blame the dumb user, but that`s because the software isn`t so smart either.

> I would also appreciate if you share your thoughts on why separate download folders per Project are necessary. This will help us to make our software better.
>
> Thank you!
>
> Oleg.


Add a simple ability, dont just fix bugs.
Oleg Chernavin 05/12/2005 08:20 am
Yes, I agree that there are cases when you need this kind of feature. The problem is that the current internal structure of Offline Explorer is very hard to change to adopt to the new model - when a Project may have its own optional directory to keep downloaded files.

We plan to add this feature, but it will really take a lot of time to implement.

Oleg.