Hello, I've been using OEP (V4.2.2377) for months, configured to download about 100 sites daily, and have a problem to report, and a question:
1) The problem: In the "File Modification Check" settings, I use the check "Skip files on levels higher than...1"; after using OEP for some minutes, downloading that or other sites, I find the setting (radio button) in "Download only modified and new files". That is: the configuration was changed ! I may repeat the setting to "Skip files on levels higher than...1", and after a some runs, randomly it is set again in "Download only modified and new files".
Is it possible that if the configuration file (webdown.dat) is being used by a running download project, and if I change the configuration of another project, it may produce the above error ??
2) The question: If I set a file filter to exclude for example "redirect.html", when OEP finds a redirect.html file will it "execute but not save it" ? That is: the filters mean that that files will not be SAVED, but it WILL be CRAWLED ?
Thank you very much,
Pablo.
1) The problem: In the "File Modification Check" settings, I use the check "Skip files on levels higher than...1"; after using OEP for some minutes, downloading that or other sites, I find the setting (radio button) in "Download only modified and new files". That is: the configuration was changed ! I may repeat the setting to "Skip files on levels higher than...1", and after a some runs, randomly it is set again in "Download only modified and new files".
Is it possible that if the configuration file (webdown.dat) is being used by a running download project, and if I change the configuration of another project, it may produce the above error ??
2) The question: If I set a file filter to exclude for example "redirect.html", when OEP finds a redirect.html file will it "execute but not save it" ? That is: the filters mean that that files will not be SAVED, but it WILL be CRAWLED ?
Thank you very much,
Pablo.
1. This is possible if you have one or more Projects selected and use Ctrl+F5, Alt+F5 or Shift+F5 hotkeys or the submenu of the Download button, which change the selected Projects settings and start the download.
2. All filters except Content Filters do not allow even downloading a file (crawling) when you disable certain URLs using them.
Best regards,
Oleg Chernavin
MP Staff
2. All filters except Content Filters do not allow even downloading a file (crawling) when you disable certain URLs using them.
Best regards,
Oleg Chernavin
MP Staff
Oleg, Thank you very much for your prompt answers.
I have posted more information about my questiones:
> 1. This is possible if you have one or more Projects selected and use Ctrl+F5, Alt+F5 or Shift+F5 hotkeys
> or the submenu of the Download button, which change the selected Projects settings and start the download.
The situation was like this: I have had 1 (one) project downloading. At the same time, I was reviewing the settings of some other projects (not running). In about 40% of these reviewed projects, I have found that the File Modification Check settings were changed (and I didn't change them).
I'm very worried about this, because now I don't feel confident that my project's settings are kept along the time, and I don't want to spend time everyday checking all the project'ts settings...
> 2. All filters except Content Filters do not allow even downloading a file (crawling) when you disable
> certain URLs using them.
Then I don't fully understand how OEP works. I have a project that downloads a news site in which most of the links go to a "redirect.html=variable number" file. After downloading it for several weeks, I realized that I have had about 175,000 redirect.html files stored in the project's folder (and also the related content files) !!!.
So I put "redirect.html" in the excluded filenames.
Now I don't have more redirect.html files stored, but I'm wondering if I am downloading all the site's content...
Thank you very much for your always excellent service !
Regards,
Pablo.
I have posted more information about my questiones:
> 1. This is possible if you have one or more Projects selected and use Ctrl+F5, Alt+F5 or Shift+F5 hotkeys
> or the submenu of the Download button, which change the selected Projects settings and start the download.
The situation was like this: I have had 1 (one) project downloading. At the same time, I was reviewing the settings of some other projects (not running). In about 40% of these reviewed projects, I have found that the File Modification Check settings were changed (and I didn't change them).
I'm very worried about this, because now I don't feel confident that my project's settings are kept along the time, and I don't want to spend time everyday checking all the project'ts settings...
> 2. All filters except Content Filters do not allow even downloading a file (crawling) when you disable
> certain URLs using them.
Then I don't fully understand how OEP works. I have a project that downloads a news site in which most of the links go to a "redirect.html=variable number" file. After downloading it for several weeks, I realized that I have had about 175,000 redirect.html files stored in the project's folder (and also the related content files) !!!.
So I put "redirect.html" in the excluded filenames.
Now I don't have more redirect.html files stored, but I'm wondering if I am downloading all the site's content...
Thank you very much for your always excellent service !
Regards,
Pablo.
Please answer my last post.
Thank you very much.
Thank you very much.
1. Do these 40% of projects have the same setting or they are different, but changed from the original?
2. In this case you have to use DeleteAfterParsing= command in the URLs field of the Project.
Oleg.
2. In this case you have to use DeleteAfterParsing= command in the URLs field of the Project.
Oleg.
Oleg, my answer to your first question:
> 1. Do these 40% of projects have the same setting or they are different, but changed from the original?
All the projects are different because they crawl a different URL. However, many of the settings of one project are derived from another project. Is this a problem ?
Please note that when I report that I found a project's setting changed was from a previous "known" state; they were not derived from a "wrong" master project.
> 2. In this case you have to use DeleteAfterParsing= command in the URLs field of the Project.
>
Thank you.
> 1. Do these 40% of projects have the same setting or they are different, but changed from the original?
All the projects are different because they crawl a different URL. However, many of the settings of one project are derived from another project. Is this a problem ?
Please note that when I report that I found a project's setting changed was from a previous "known" state; they were not derived from a "wrong" master project.
> 2. In this case you have to use DeleteAfterParsing= command in the URLs field of the Project.
>
Thank you.
Frankly, I have no idea why this may happen except that you selected several Projects and used Ctrl+F5 or Alt+F5 or Shift+F5 keys to start them - these keys change Project File Modification Check setting.
Oleg.
Oleg.
OK Oleg, thank you. I will follow watching if it happens again.
> Frankly, I have no idea why this may happen except that you selected several Projects and used Ctrl+F5 or Alt+F5 or Shift+F5 keys to start them - these keys change Project File Modification Check setting.
>
> Oleg.
> Frankly, I have no idea why this may happen except that you selected several Projects and used Ctrl+F5 or Alt+F5 or Shift+F5 keys to start them - these keys change Project File Modification Check setting.
>
> Oleg.
Yes, please keep me informed.
Oleg.
Oleg.