A problem, and a question

Author Message
Pablo 05/23/2006 11:26 am
Hello, I've been using OEP (V4.2.2377) for months, configured to download about 100 sites daily, and have a problem to report, and a question:

1) The problem: In the "File Modification Check" settings, I use the check "Skip files on levels higher than...1"; after using OEP for some minutes, downloading that or other sites, I find the setting (radio button) in "Download only modified and new files". That is: the configuration was changed ! I may repeat the setting to "Skip files on levels higher than...1", and after a some runs, randomly it is set again in "Download only modified and new files".
Is it possible that if the configuration file (webdown.dat) is being used by a running download project, and if I change the configuration of another project, it may produce the above error ??

2) The question: If I set a file filter to exclude for example "redirect.html", when OEP finds a redirect.html file will it "execute but not save it" ? That is: the filters mean that that files will not be SAVED, but it WILL be CRAWLED ?

Thank you very much,
Pablo.
Oleg Chernavin 05/23/2006 02:04 pm
1. This is possible if you have one or more Projects selected and use Ctrl+F5, Alt+F5 or Shift+F5 hotkeys or the submenu of the Download button, which change the selected Projects settings and start the download.

2. All filters except Content Filters do not allow even downloading a file (crawling) when you disable certain URLs using them.

Best regards,
Oleg Chernavin
MP Staff
Pablo 05/23/2006 05:19 pm
Oleg, Thank you very much for your prompt answers.
I have posted more information about my questiones:

> 1. This is possible if you have one or more Projects selected and use Ctrl+F5, Alt+F5 or Shift+F5 hotkeys
> or the submenu of the Download button, which change the selected Projects settings and start the download.

The situation was like this: I have had 1 (one) project downloading. At the same time, I was reviewing the settings of some other projects (not running). In about 40% of these reviewed projects, I have found that the File Modification Check settings were changed (and I didn't change them).
I'm very worried about this, because now I don't feel confident that my project's settings are kept along the time, and I don't want to spend time everyday checking all the project'ts settings...



> 2. All filters except Content Filters do not allow even downloading a file (crawling) when you disable
> certain URLs using them.
Then I don't fully understand how OEP works. I have a project that downloads a news site in which most of the links go to a "redirect.html=variable number" file. After downloading it for several weeks, I realized that I have had about 175,000 redirect.html files stored in the project's folder (and also the related content files) !!!.
So I put "redirect.html" in the excluded filenames.
Now I don't have more redirect.html files stored, but I'm wondering if I am downloading all the site's content...

Thank you very much for your always excellent service !
Regards,
Pablo.

Pablo 05/30/2006 11:35 am
Please answer my last post.
Thank you very much.
Oleg Chernavin 05/31/2006 04:25 am
1. Do these 40% of projects have the same setting or they are different, but changed from the original?

2. In this case you have to use DeleteAfterParsing= command in the URLs field of the Project.

Oleg.
Pablo 06/05/2006 07:24 am
Oleg, my answer to your first question:

> 1. Do these 40% of projects have the same setting or they are different, but changed from the original?
All the projects are different because they crawl a different URL. However, many of the settings of one project are derived from another project. Is this a problem ?
Please note that when I report that I found a project's setting changed was from a previous "known" state; they were not derived from a "wrong" master project.


> 2. In this case you have to use DeleteAfterParsing= command in the URLs field of the Project.
>
Thank you.
Oleg Chernavin 06/05/2006 07:46 am
Frankly, I have no idea why this may happen except that you selected several Projects and used Ctrl+F5 or Alt+F5 or Shift+F5 keys to start them - these keys change Project File Modification Check setting.

Oleg.
Pablo 06/05/2006 01:55 pm
OK Oleg, thank you. I will follow watching if it happens again.


> Frankly, I have no idea why this may happen except that you selected several Projects and used Ctrl+F5 or Alt+F5 or Shift+F5 keys to start them - these keys change Project File Modification Check setting.
>
> Oleg.
Oleg Chernavin 06/05/2006 04:15 pm
Yes, please keep me informed.

Oleg.