new detailed logging propose

Author Message
QuantumShadow 02/22/2004 05:05 pm
Hi Oleg & all,
having a very long personal experience with OE,
and from reading the forums, I found that a lot of problems could be solved much faster,
and by user own means, if the logging information was more detailed.
For example one of the most asked question and problem is why certain files or links not being downloaded by OE, or on the opposite why some file being download in spite of that the user thinks the should not pass the filters.
Current logging can be obtained by running OE.exe /log from the command line,
but this log doesn`t have all the necessary information.
I want to propose here all the necessary log printouts as I see it, which could help in determining the downloading decision about every file or link. This is not yet the full list I know I missed some of the possibilities, but I hope that with ur help it will be completed.
I hope the OE team will support this initiative, and add all the necessary logs.

This is the log:

1. Get directory listing:
SUCCESS!
PRINT(list of files that OE got)
OR
FAILURE!
Reason*

Links inserted to queue:
PRINT(list of inserted links)
Links rejected:
PRINT(list of rejected links)
Reason* for rejection for each link

2. Get file: filename1.html
SUCCESS!
PRINT( File Size)
OR
FAILURE!
Reason*

3. Parse file: filename1.html, parsing results:
Links inserted to queue:
PRINT(list of inserted links)
Links rejected:
PRINT(list of rejected links)
Reason* for rejection for each link

*Possible Reasons List
Authorization failed. (Username & Password)
Access denied
File not found
Can’t connect
File corrupted
Disk Full
Level Limit
File already exists
Failed Extension Filter
Failed Size Filter
Failed Protocol Filter
Failed Server Filter
Failed Directory Filter
Failed FIlename Filter
Failed Content Filter
One of the Advanced Limits achieved
Do not know how to parse (unsupported script)


The log should be enabled from the GUI (like now), and it should be possible to save it to a file from the GUI (unlike now).
There are two types of logs:
1. The Information log that i described above. (almost always useful)
2. The debug log, this is something that can have the OE developer team to understand and fix the bug, that they for some reason can`t recreate in their labs. (Useful only when some bug occurs)

It should be possible to enabled the Info Log without enabling the debug log, and vise versa.

Waiting for ur comments and additions
(hope that this was not too much ;)
Thanks,
QuantumShadow.
Oleg Chernavin 02/23/2004 06:42 am
Hello,

In fact, all such logging capabilities exist in OE for a very long time. You can press Ctrl-W to access the Log Window (or use View | Messages | Log Window in the main menu). Right-click the window to access its settings in the Filters menu. You can pause/unpause the log, save it to a file, etc.

I think that the logging level is detailed enough for most purposes.

Best regards,
Oleg Chernavin
MP Staff
QuantumShadow 02/24/2004 03:53 pm
Oleg, I know about the two types of logs OE has:
Oe.exe /log from cmd line
and
Ctrl-W from GUI
basically they give the same printouts.
both have several disadvantages:
(please use same numbers when answering, it would be easier)

1. GUI can`t save to file, the cmd line log can save to file, but u can`t control it with GUI interface
(i.e. can`t set the log filters, etc..) it would be nice to add to GUI log the ability to save to file.

2. One of the main ideas in my log proposition is the ability to print the list of links/files OE got when parsed a file or script, so u could actually see if OE found that link or maybe couldn`t even parse it or whatever. cause sometimes I don`t have some file downloaded but I also don`t have it`s name mentioned in the log, so I wonder where did it go.

3. I think that existing log indeed has most of the variants of error or rejections, but some are missing, for example i didn`t see that a file was skipped because it already on the disk. I think there are some more edge possibilities that r missing, but i can`t check without the source of course.
I just know that sometimes i can`t find in the log what happen to some file or link that i miss.

Thanks,
QuantumShadow.


> Hello,
>
> In fact, all such logging capabilities exist in OE for a very long time. You can press Ctrl-W to access the Log Window (or use View | Messages | Log Window in the main menu). Right-click the window to access its settings in the Filters menu. You can pause/unpause the log, save it to a file, etc.
>
> I think that the logging level is detailed enough for most purposes.
>
> Best regards,
> Oleg Chernavin
> MP Staff
Oleg Chernavin 02/25/2004 01:46 pm
1. You can use the Save to File command in the Log Window context menu.
2&3 I can add output of all found files with the reason of the reject - either it was already added to the queue or it exists on the disk. But this would be too much info, which will onloy slowdown the program. Each Web page contains hundreds of links usually and they all repeat several times in each page.

Are you sure you want that amount of information?

Oleg.
QuantumShadow 02/28/2004 08:40 am
Hi,
sorry about number 1, i didn`t notice u have already added this in the new version.
about numbers 2&3 - this is really a lot of info which is usually not needed, so i was thinking about some log filter (like u have already), something like "reject details", and it would be off by default.
So only in critical cases it will be used.
Iif u could add it, i would really appreciate.
Thanks.
QuantumShadow.


> 1. You can use the Save to File command in the Log Window context menu.
> 2&3 I can add output of all found files with the reason of the reject - either it was already added to the queue or it exists on the disk. But this would be too much info, which will onloy slowdown the program. Each Web page contains hundreds of links usually and they all repeat several times in each page.
>
> Are you sure you want that amount of information?
>
> Oleg.
QuantumShadow 02/28/2004 11:18 am
Hi Oleg, sorry again for the number 1, but as i looked now in the newest version,
u have the option to save to file only what u have already got in the log window,
i was interested in constantly sending to file all the log output, so right now if OE crashes for example, u can`t save to file because there is no more console.
(the /log in the cmd line is not the solution, as i exlained before, it doesn`t have filters support, etc..)

> Hi,
> sorry about number 1, i didn`t notice u have already added this in the new version.
> about numbers 2&3 - this is really a lot of info which is usually not needed, so i was thinking about some log filter (like u have already), something like "reject details", and it would be off by default.
> So only in critical cases it will be used.
> Iif u could add it, i would really appreciate.
> Thanks.
> QuantumShadow.
>
>
> > 1. You can use the Save to File command in the Log Window context menu.
> > 2&3 I can add output of all found files with the reason of the reject - either it was already added to the queue or it exists on the disk. But this would be too much info, which will onloy slowdown the program. Each Web page contains hundreds of links usually and they all repeat several times in each page.
> >
> > Are you sure you want that amount of information?
> >
> > Oleg.
Oleg Chernavin 03/12/2004 09:34 am
I just added Log filtering:

To filter the amount of the logging information, use /LogFilter=1111101
parameter. Each number allows (1) or (0) disallows to output the following
information: connection, download progress, detailed information (headers,
etc.), disconnection, queue messages, accesses to the Internal Web server
and rejected URLs.

Here is the updated oe.exe file:

http://www.metaproducts.com/download/betas/oep1528.zip

Oleg.
QunatumShadow. 03/13/2004 09:56 am
Thanks, why don`t u add the "continious save to file to the GUI too"?
the problem with the cmd line is that it must be started with log BEFORE i run OE
i.e. BEFORE i even know i that i will have a problem with certain site, so when i do have a problem i must stop, and rerun, unfortunatly the rerun phase could take days.
But if i could enable and change logging DURING the downloading it would make my life much easier.
QunatumShadow.

> I just added Log filtering:
>
> To filter the amount of the logging information, use /LogFilter=1111101
> parameter. Each number allows (1) or (0) disallows to output the following
> information: connection, download progress, detailed information (headers,
> etc.), disconnection, queue messages, accesses to the Internal Web server
> and rejected URLs.
>
> Here is the updated oe.exe file:
>
> http://www.metaproducts.com/download/betas/oep1528.zip
>
> Oleg.
Oleg Chernavin 03/13/2004 05:00 pm
This is easy - just keep OE running and start it again from the command-line with the changed logging parameter. The running copy of OE will receive them and change its settings on-the-fly.

Oleg.