Big problem with OE PRO ENT 3.1

Author Message
Kenneth 04/27/2004 05:13 am
Hey, I have a problem with offline explorer enterprise 3.1, her we go
I have started downloading a BIG site.
When I try to load the queue file (suspended) it takes like 100% cpu for half a day before the queue shows up.
..And then when the download starts, it downloads a couple of files then it waits like 20-30secs before it downloads a couple more, this way the project will take forever=P
I have set the the time between downloads to 1sec so its not that.
The sites is big 512.000 files downloaded so far. the queue file contains 43.000 files.
The site is built like this /members/(contain all html) /good/ (contain all images) 600.000+
Running a AMD Athlon 64 3400+
2048 DDram 400mhz
500GB raided Raptor drives
So, I don`t get way it takes so damn long to load the queue file containing 43.000 enteries.
Ad why the download "idles" 20-30 sec between files.
help please..
Oleg Chernavin 05/03/2004 03:59 pm
I am sorry for the late reply. I just wrote you an E-mail using the address you supplied in the forum. If you don`t get a message from me, please contact me yourself using support@metaproducts.com .

Thank you!

Best regards,
Oleg Chernavin
MP Staff
im 05/06/2004 09:50 am
I have this problem Too.

> Hey, I have a problem with offline explorer enterprise 3.1, her we go
> I have started downloading a BIG site.
> When I try to load the queue file (suspended) it takes like 100% cpu for half a day before the queue shows up.
> ..And then when the download starts, it downloads a couple of files then it waits like 20-30secs before it downloads a couple more, this way the project will take forever=P
> I have set the the time between downloads to 1sec so its not that.
> The sites is big 512.000 files downloaded so far. the queue file contains 43.000 files.
> The site is built like this /members/(contain all html) /good/ (contain all images) 600.000+
> Running a AMD Athlon 64 3400+
> 2048 DDram 400mhz
> 500GB raided Raptor drives
> So, I don`t get way it takes so damn long to load the queue file containing 43.000 enteries.
> Ad why the download "idles" 20-30 sec between files.
> help please..
>
Oleg Chernavin 05/06/2004 04:17 pm
Yes, we are working on this problem now. Once I have results, I will post the news here.

Oleg.
Keith Mason 05/22/2004 12:14 pm
> Yes, we are working on this problem now. Once I have results, I will post the news here.
>
> Oleg.

I`ve noticed the problem, and it seems to have something to do with having the Queue tab selected. Whenever the queue is huge and it is being displayed, OE spends most of its time updating the GUI.

Get around this by clicking on the projects tab before un-suspending.
Oleg Chernavin 05/22/2004 12:17 pm
We also found out that such delays could happen when using File Copies feature (in the Project Properties). If time/date is not used to rename old copies, then renaming takes long time when there are tens or hundreds of copies.

Oleg.
Keith Mason 05/22/2004 12:36 pm
Interesting... I`ve never used Old Copies, so I wouldn`t know about that. But the queue text object does get painful with > 100,000 URLs. I`ve got over 500,000 in mine now, and if I even touch the Queue tab, it`s about half an hour before I can get back to stable. Windows just can`t keep up.

> We also found out that such delays could happen when using File Copies feature (in the Project Properties). If time/date is not used to rename old copies, then renaming takes long time when there are tens or hundreds of copies.
>
> Oleg.
Oleg Chernavin 05/22/2004 12:46 pm
Yes, if you download speed is very high, Queue consumes quit a lot CPU when you select it. But if you pause (Suspend) the download before switching to the Queue - does it take much time to show when you have that many files?

Oleg.
Keith Mason 05/22/2004 12:55 pm
No, when suspended, displaying the queue works reasonably efficiently. However, if I un-suspend but forget to first select away from the queue, I lose application responsiveness for quite a long time.

> Yes, if you download speed is very high, Queue consumes quit a lot CPU when you select it. But if you pause (Suspend) the download before switching to the Queue - does it take much time to show when you have that many files?
>
> Oleg.
Oleg Chernavin 05/22/2004 12:58 pm
Yes, it is so. Frankly, I have no idea how to make it better. I optimized the Queue manager a lot. I will think about it more, but just cannot guarantee that I will find a good solution.

Oleg.
Keith Mason 05/22/2004 01:01 pm
Perhaps something simple, such as hide the Queue tab when URL count is above 50,000?

I`m not sure this needs fixing... it`s really a Windows bug.

-- Keith

(PS: Sorry to keep you so busy on the weekend!)
Oleg Chernavin 05/22/2004 01:04 pm
Or maybe to suspend the download when you go to the Queue with too many files. I will think about it. I just want it to be really transparent for users.

Oleg.