Is there any way to prevent oe to parse the file when file size is large

Author Message
aa 09/17/2003 03:46 pm
Can I set some options to prevent the oe to parse the file when file is large than 10M?
thank you
Oleg Chernavin 09/19/2003 02:39 pm
Hello,

There is no such feature yet, but why do you need it? Does Offline Explorer takes much time to parse it? If yes, let me know the link to such file and I will try to optimize parsing, so it takes less time.

Thank you.

Best regards,
Oleg Chernavin
MetaProducts corp.
Scotty 09/29/2003 05:13 am
> There is no such feature yet, but why do you need it? Does Offline Explorer takes much time to parse it? If yes, let me know the link to such file and I will try to optimize parsing, so it takes less time.

In my case, OEP practically comes to a halt with many sites, due to all the parsing it must do. I`ve mentioned this here before, but no one seems to take it too seriously, normally attributing it to a slow computer. The problem is real, however. I`m currently downloading a site with (from what I`ve seen so far) 1.2 million pages; at this point I`ve downloaded 150,000 of them. OEP is taking about 5 seconds to parse each file, and I have a backlog of 40,000 files to parse! My machine has 2 Pentium III 500 MHz CPUs and 512 MB of RAM. I seriously doubt I`ll be able to finish downloading this site.


Check out <www.almacenes-paris.cl>. It`s a huge site (a virtual catalog for a store), and it`s a real problem for me.

And please, even if this site or others seem to download fine for you, do add a way to pause parsing, and to pause it without pausing downloading. And let me add that the parsing slow-downs normally start after having downloaded thousands of files from a site -- they don`t start immediately.
K.lopps 09/29/2003 07:38 am
have you enabled the option: prevent download directories from overloading ?
if not this could be your problem. especially windows slows down enourmously if directories have more than 10000 files in it.
Oleg Chernavin 09/29/2003 07:55 am
Yes, this kind of slowdown may take place. I also work to find ways to make parsing much faster. I hope to succeed here.

Oleg.