Several questions together...

Author Message
Juan Jose Gonzalez 12/12/2005 03:58 pm
Hi. I`m starting to use Offline Explorer Enterprise since about a week, and have colleted the following questions:

1) Sometimes the download process finishes (or seems to be finished), with no download activity, but the icon indicator does not change to finished; also, the Report says it is still downloading. And stays there for hours.
2) What is the difference between "Download Missing Files" and "Update Project" ?
3) I don`t want to download pages that have not changed; then I have checked "Download only Modified and new files", and also marked "Check file size". However, after I finish the update of a site, I start a Update Download again to verify that nothing is to be downloaded, but ... it starts to download files again. And I don`t think they have changed so many files in such a short time!
4) Trying to find the optimum (faster) configuration:
- What is best: to set a "small" number of connections (10), or a "large" one (like 30 or 40) ?
- What is best: to download a single website at a time (sequence), or to fire many sites at once ?

Thank you,
JJ

Oleg Chernavin 12/13/2005 08:22 am
1. Perhaps Offline Explorer processes downloaded files - please see in the Status Bar if there is a Parsing (XX) message.

2. Update Project checks all files whether they have changed on the server since the last download. Download missing files will not check all downloaded files - it will simply detect all links that were not yet downloaded and gets them.

3. Many servers have dynamic pages nowdays and they report them as just newly updated. You can easily recognize them - .asp, .aspx, .php, etc. This is a problem, but such servers may even not supply the file size.

4. From my experience, it doesn`t matter much on fast connections - whether you download 10 or 30 files at a time. Downloading several Web sites at a time is always better, because you balance load on several servers.

Best regards,
Oleg Chernavin
MP Staff
Juan Jose Gonzalez 12/14/2005 07:27 am
Dear Oleg,
Thank you very much for your answers.
Now I will focus only on point 1). These are the symptoms I have:

- Yes, I have a Parsing (1049) message on the bottom left status bar.
- The download seems to be finishing, but never finishes: it has 30 threads,
27 with "Download Complete", 2 with "Connection Timed out", and 1 with "File
is probably incomplete, retrying".
- The "Queue" folder is empty.
- Seems that there is no internet activity, but the hard disk is working a lot.
- If I try to suspend the project, the status bar says "Suspending", and stays there.
- The site is ASP (www.lanacion.com.ar), and I`m downloading it with a level=3 deep.

Reards,
JJ

> 1. Perhaps Offline Explorer processes downloaded files - please see in the Status Bar if there is a Parsing (XX) message.
> > 2. Update Project checks all files whether they have changed on the server since the last download. Download missing files will not check all downloaded files - it will simply detect all links that were not yet downloaded and gets them.
> > 3. Many servers have dynamic pages nowdays and they report them as just newly updated. You can easily recognize them - .asp, .aspx, .php, etc. This is a problem, but such servers may even not supply the file size.
> > 4. From my experience, it doesn`t matter much on fast connections - whether you download 10 or 30 files at a time. Downloading several Web sites at a time is always better, because you balance load on several servers.
> > Best regards,
> Oleg Chernavin
> MP Staff
Oleg Chernavin 12/14/2005 07:31 am
This means that Offline Explorer still has 1049 files to process (extract links and change them for offline browsing). Perhaps this process is slow because there are some directories overpopulated (have too many files). Windows filesystems slows down significantly in this case. To overcome it, please allow Directory Overload protection in the Options dialog - File Locations section. It is better to erase such overloaded directories after that and redownload files.

Oleg.
Juan Jose Gonzalez 12/14/2005 08:36 am
Hello.
I have tryed again with the Directory Overload protection (with previous erase of all)
Same result: the download process stops without finishing.
I have a "Parsing (925)" indication on the bottom left.
There is no internet nor hard disk activity.
I have 29 threads with "Download Complete", and one with "File is probably incomplete, retrying".

I hope you can have a hint on this...

Thank you very much,
JJ
Oleg Chernavin 12/14/2005 08:52 am
What version of Offline Explorer do you have now?

Oleg.
Juan Jose Gonzalez 12/14/2005 11:58 am
It is Offline Explorer Enterprise 4.0.2272


> What version of Offline Explorer do you have now?
> > Oleg.
Oleg Chernavin 12/15/2005 11:47 am
Can I ask you to try the latest version of oe.exe file?

http://www.metaproducts.com/download/betas/oep2282.zip

Please replace the old oe.exe file with the new one from the ZIP archive. Then let me know if the new version hangs on parsing or not.

Oleg.
Juan Jose Gonzalez 12/26/2005 03:35 pm
Hello again. Sorry, but the problem is still there. I have tryed oep2282 and oep2283.
Download stops and then all the sequence of downloads is stopped.
Regards, JJ

> Can I ask you to try the latest version of oe.exe file?
> > http://www.metaproducts.com/download/betas/oep2282.zip
> > Please replace the old oe.exe file with the new one from the ZIP archive. Then let me know if the new version hangs on parsing or not.
> > Oleg.
Oleg Chernavin 12/27/2005 09:39 am
Can you please turn on logging (Ctrl+W) and let me know which file starts to be parsed and copy/paste its URL.

Thank you!

Oleg.
Juan Jose Gonzalez 12/27/2005 12:57 pm
Hello. I have turned on logging, and started the download of the site with the problem (the site that does not finish). The first message logged is:
QUEUE - 27/12/2005 02:34:20 p.m. - Analizando (0) http://www.lanacion.com.ar/

Is this what you need ?

Also, for your information, I have copied and pasted the project:

[Object]
OEVersion=Pro 4.0.0.2283
Type=0
IID=7012
Caption=La Nacion (nivel 3)
URL=http://www.lanacion.com.ar/
Lev=3
Weekday=257
LimTSize=10000
LimNumber=5000
LimTime=100
FMGroup=2
CheckSize=True
FTText.Exts=htmlhtmaspaspxjspstmstmlidcshtmlhtxtxttextxspxmlrxmlcfmwmlphpphp3
FTImages.Exts=gifjpgjpegtiftiffxbmfifbmppngipxjp2j2cj2kwbmplwf
FTVideo.Exts=mpgavianimpegmovfliflcvivrmramrvasfasxwmvm1vm2vvob
FTAudio.Exts=wavriffmp3midmp2m3uravocwmaape
FTArchive.Exts=ziparcgzzarjlhalayleirarcabtarpakacejarpdf
FTUDef.Exts=jscssssivbsdtdxslswfclass
FTText.B=ooxooo
FTImages.B=ooxoxo
FTVideo.B=xoxooo
FTAudio.B=xoxooo
FTArchive.B=xoxooo
FTUDef.B=ooxooo
FTOther.B=xoxooo
FTSizes=0,0,20,0,0,0,0,0,0,0,0,0,0,0,1,1,3,3,0,1,0
RProt=127
LastStart=110:182:7:110:51:231:226:64:
LastEnd=175:155:240:110:51:231:226:64:
S304=137
SPar=1
SLast=304
LFiles=137
Stopped=True
Copies=10000
CopiesDate=YYYYMMDDHHNN
ImgDim=0,0,0,0
PrevURL=http://www.lanacion.com.ar/
SkipURLs=https://www.lanacion.com.ar
ConvertRSS=True

Thank you.


> Can you please turn on logging (Ctrl+W) and let me know which file starts to be parsed and copy/paste its URL.
> > Thank you!
> > Oleg.
Oleg Chernavin 12/28/2005 02:04 am
Well, I need last lines like that. If the parsing hangs, the Log should show the last file that started parsing, but didn`t end.

Oleg.
Juan Jose Gonzalez 12/29/2005 12:18 pm
Hello. Tyed again. The download finished with "Download Complete" in all the threads. However the project is still reported as "Loading", and interrupts the download sequence.

The last logged line was:

HTTP4 - 29/12/2005 02:02:48 p.m. - 100% of 35579 bytes of http://www.lanacion.com.ar/Archivo/nota.asp?nota_id=764698.

I have also copied the state of all the 20 finished threads. Click here to see the image:
http://www.lablar.com/Captura.jpg

Regards.
Oleg Chernavin 12/29/2005 12:23 pm
Can you please try the following updated oe.exe file:

http://www.metaproducts.com/download/betas/oep2294.zip

Replace the older oe.exe file with the new one. Please let me know if the bug reproduces or not.

Thank you!

Oleg.
Juan Jose Gonzalez 12/29/2005 02:13 pm
Downloaded and installed oep2924, but ... same result: site downloaded, but does not finish the project.

Here you have some behaviour that perhaps may help:
After finishing the (bad) website, I pressed STOP to force it to finish the project. On the report, it now shows "Stopped". In this state, if I start any other download (sites that have not showed the problem), it does not download! It shows "Download complete", but didn`t download anything.
The only way to start a new download is to close OE and start again, and now the download goes normal.
It looks like a problem on the internal state machine ...??

Regards,
JJ


> Can you please try the following updated oe.exe file:
> > http://www.metaproducts.com/download/betas/oep2294.zip
> > Replace the older oe.exe file with the new one. Please let me know if the bug reproduces or not.
> > Thank you!
> > Oleg.
Oleg Chernavin 12/30/2005 03:45 am
Can you please find the last QUEUE lines in the log and send me few of them?

Oleg.
Juan Jose Gonzalez 01/10/2006 08:38 am
Hello Oleg, I`m here again.

The QUEUE tab finishes EMPTY after download.
Also, the download progress finishes with a Download Complete.

Everything seems to be OK, BUT...
- The STOP button continues active, meaning that the software thinks there is a download in course.
- If there was a SEQUENCE, it is stopped.
- If you mannualy start a new download for a different site, it does not work.

There is a problem here. I think you may easily duplicate it by downloading
www.lanacion.com.ar with a Level Limit of 3.

Thank you,
JJ


> Can you please find the last QUEUE lines in the log and send me few of them?
> > Oleg.
Oleg Chernavin 01/10/2006 09:51 am
I mean that I need the last lines of the log with the QUEUE messages.

Oleg.
Juan Jose Gonzalez 01/10/2006 10:38 am
Sorry but I don`t understand what you want.
I have the full log file of the site with the problem; I can email it to you (tell me your email address)
Thanks.


> I mean that I need the last lines of the log with the QUEUE messages.
> > Oleg.
Oleg Chernavin 01/11/2006 06:25 am
Please zip the log and send it to support@metaproducts.com. Thank you!

Oleg.
Juan Jose Gonzalez 01/12/2006 12:05 pm
Did you receive the LOG file ?
JJ

> Please zip the log and send it to support@metaproducts.com. Thank you!
> > Oleg.
Oleg Chernavin 01/12/2006 12:20 pm
Yes and I will work on it soon. Thank you!

Oleg.
Juan Jose Gonzalez 01/19/2006 09:29 am
Today I sent you a new e-mail with more on this issue.
JJ

> Yes and I will work on it soon. Thank you!
> > Oleg.
Oleg Chernavin 01/19/2006 09:33 am
Did you send it directly to my E-mail address?

Oleg.
fabbr 04/29/2008 02:27 pm
Same problem here...

Program show parsing 678 files on status bar....... and stays like this indefinitely...

When i stop the project and choose download missing files........it says "parsing (678)" and doesn''t change status.

The icon of the project is green (downloading).

Do not overload directories activated....
Oleg Chernavin 04/30/2008 04:10 am
What version do you use? And please let me know other details - the site you are downloading, number of files downloaded by the time it stalls, number of files in the Queue, downloaded files size.

Oleg.
fabbr 05/01/2008 05:49 pm
> What version do you use? And please let me know other details - the site you are downloading, number of files downloaded by the time it stalls, number of files in the Queue, downloaded files size.
>
> Oleg.

Ok....

The version is 4.9.2694 SR2

The site is: http://www.forumnow.com.br/vip/foruns.asp?forum=94106 (IE cookies activated, IgnoreLogoutLinks activated...)

Actually, as it''s a huge forum site, i stopped downloading it sometimes. I think i''ve lost the map and choose CTRL + F5 to rebuild it.

Now it always downloads a small number of files, the queue stays with 678 files and nothing happens. The icon is green (downloading). The project has already downloaded aproximately 100.000 files, but i know it hasn''t downloaded everything in the site, as i want (the 678 files in queue confirms it).

fabbr 05/01/2008 06:36 pm
P.S.:

Sorry....

I gave wrong information.

The correct information is:

The version is 4.9.2694 SR2

The site is: http://www.forumnow.com.br/vip/foruns.asp?forum=94106 (IE cookies activated, IgnoreLogoutLinks activated...)

Actually, as it''''s a huge forum site, i stopped downloading it sometimes. I think i''''ve lost the map and choose CTRL + F5 to rebuild it.

Now it says in says in status bar: DOWNLOADED=0 ----- PARSING (678) ----- QUEUE=0 SIZE=0. BUT NOTHING HAPPENS. The icon is green (downloading). The project has already downloaded aproximately 100.000 files, but i know it hasn''''t downloaded everything in the site, as i want.





> > What version do you use? And please let me know other details - the site you are downloading, number of files downloaded by the time it stalls, number of files in the Queue, downloaded files size.
> >
> > Oleg.
>
> Ok....
>
> The version is 4.9.2694 SR2
>
> The site is: http://www.forumnow.com.br/vip/foruns.asp?forum=94106 (IE cookies activated, IgnoreLogoutLinks activated...)
>
> Actually, as it''''s a huge forum site, i stopped downloading it sometimes. I think i''''ve lost the map and choose CTRL + F5 to rebuild it.
>
> Now it always downloads a small number of files, the queue stays with 678 files and nothing happens. The icon is green (downloading). The project has already downloaded aproximately 100.000 files, but i know it hasn''''t downloaded everything in the site, as i want (the 678 files in queue confirms it).
>
>
Oleg Chernavin 05/02/2008 05:55 am
You will have to wait. Parsing means getting links from the downloaded page. It has to get through all pages on your hard disk to get the links to download further.

Oleg.