Problem with (i.e.) SkipParsingAfter and resumed downloads from file

05/01/2014 02:08 pm
Hi there,

if I have a project and I suspended the downloading queue to a file and I add in the project properties the command SkipParsingAfter (or i.e. Delay) and I resume the downloading queue from the file the afterward downloaded files are ignoring the added command(s) as if I wouldn't have entered it in the project properties. (When starting the download new and not resume it from a file it works as it should.)

I don't know (and did not test) which commands are working or not, but it would be great if you check this case and fix it, so that added project properties commands are working also on resumed downloads from file.

Thanks in advance,
Oleg Chernavin
05/05/2014 07:57 am
Delay= parameters are kept in the Suspend To File. The XXXParsing commands were just added.

We plan to release the new version today or tomorrow.

Thank you for the suggestion!

Best regards,
Oleg Chernavin
MP Staff
05/05/2014 01:13 pm
Hi Oleg,

thanks for the quick response - but I'm afraid I have to say to you: nothing changed.

Let me explain again a bit more in detail based on a theoretical example to avoid misunderstandings.

1.) I add a new project in OEP and start downloading.

2.) After downloading some pages I suspend the downloading queue to a file.

3.) I've seen that some things should be changed for this download, so I go to the project properties and...
3a.) in "URL Filters" - "Directories" I add in "Excluded keywords" the keyword "/print/"
3b.) in "Content Filters" - "Keywords" I add "unavailable" and check the box "Save these pages" in section "When keywords are not found in a page"
3c.) in "Project" - "Addresses (URLs)" I add the commands "Delay=5" and "SkipParsingAfter=Similar products"

4.) Now I resume the downloading queue from the file and...
4a.) URLs with directory "/print/" are ignored - OK
4b.) pages without "unavailable" will be saved, others not - OK
4c.) there is no delay between downloads and the whole page will be parsed - NOT OK

So I hope you know now what I mean: the changes made in 3c.) should also work after resuming the download like the changes made in 3a.) and 3b.)

It would be great, if you could add (or fix?) this functionality in OEP.

Oleg Chernavin
05/06/2014 03:49 pm

I just tested this on our MetaProducts site and the Delay= and SkipParsingAfter= were preserved in the .wdq file and after resuming (I quit and restarted OEP to make sure) both parameters continued working correctly.

Please make sure you have 6.8.4098 version installed now.

05/08/2014 12:57 pm

I tested again and again. Yes, I've got version 6.8.4098 installed and I restarted OEP and my computer several times - but it doesn't work as I intended.

Did you test it in the strict order as I described before?
Because: be sure, that you add the changes in 3.) during you suspended the download AFTER you started downloading the project. If you entered "Delay" and "SkipParsingAfter" commands right from the start the commands are ok after suspending and resuming to/from a file, but that's not what my question is about.

Again: the changes should work when made WHILE a download is executed, not when the download starts from the beginning.

I hope it's finally clear now, what I mean.

Oleg Chernavin
05/08/2014 01:28 pm
Yes, that's true. Changes to such parameters during the download do not affect it. Sorry, I don't have plans to make such changes now. I will consider them in future versions.

05/08/2014 02:05 pm
Ok, thanks for this (somewhat sad) information. :-(

But so I know now at least that I have to put more time in checking a website's structure before downloading huge website projects...

And, btw, OEP is still a GREAT product!! :-)

Looking forward for your implementation of this feature in the near future,
Oleg Chernavin
05/10/2014 08:05 am
Thank you very much! Yes, I will think about adding this.