More URL filter questions

Cami
11/07/2008 07:24 pm
Hi,

Thank you for your excellent recent help. I have developed an OE project that successfully downloads a gallery from deviantart.com. However there are just a couple of issues I would like to clear up - more a matter of tidiness than functionality:

1). I''m trying to filter out urls like:

http://noirfeu.deviantart.com/pageid=295245089/random=949431729/site=deviantart/area=deviation/aamsz=728x90

where noirfeu is the user name. I''ve been using a macro {:duser} in other filters, such as "included files keywords" with great success. However when I use

http://{:duser}.deviantart.com/pageid=*

in "excluded files keywords" it does not seem to exclude them. Any idea why it might not be working?


2). Sometimes OE comes up with urls like:

http://doomsday-dawn.deviantart.com/art/"http://www.deviantart.com/deviation/100455568/"

I want to use the substitution feature to just keep the http://www.deviantart.com/deviation/100455568/ portion.

I''ve tried to do this in two stages:
a). URL: http://{:duser}.deviantart.com/art/"http://www.deviantart.com/deviation/*/"
Replace: http://{:duser}.deviantart.com/art/"
With:
and
b). URL: */deviation/*/"
Replace: "
With:

I have the "apply all matching rules" box checked and have each substitution rule checked. However neither of these substitutions seem to work. I''m thinking that the quotes may be the problem. Any ideas?


3). Lastly it would be really nice to be able to pass my user macro {:duser} to OE in a command line. I don''t see any provision for doing so but thought I would ask in case I''m missing something.


Thanks for a brilliant product. It''s amazing what it can do.
Cami
11/07/2008 08:34 pm
Update:

I remembered the substitution test feature and have now resolved my substitution problem, though I''m not entirely sure how. My new (working) solution is:

a.) URL: /deviation/*/"
Replace: "
With:

This seems to successfully remove both quotation marks

b.) URL: http://*.deviantart.com/art/http://www.deviantart.com/deviation/*/
Replace: http://*.deviantart.com/art/
With:

Anyway, the substitution now seems to be working!

As for my URL filter, I tried changing {:duser} into *, but it still does not seem to work.

Thanks again,
Cami
Cami
11/07/2008 08:54 pm
Re-update:

Now I''m not so sure that my substitution is working. It passes the OE "test" ok, but the bad (pre-substitution) url is still showing up frequently in the queue. Assuming that the substitution is made before the url is added to the queue then I have a problem.

Help!!!!
Oleg Chernavin
11/08/2008 07:05 am
1. Please change the excluded filename keyword to:

http://{:duser}.deviantart.com/pageid=*/*

2. I made a fix and now a simple rule will be possible:

URL:
*
Replace:
**"*"
With:
*

Please update your oe.exe file with this one:

http://www.metaproducts.com/download/betas/OEP2870.ZIP

3. I will think on how to pass it, but so far I don''t have a good idea.

Best regards,
Oleg Chernavin
MP Staff
Cami
11/08/2008 01:15 pm
1. Your solution works perfectly. Thank you.

2. I have replaced the OE executable with the beta that you created for me and changed the substitution rule to what you suggested, but I''m afraid that it is still not ''fixing'' the urls. When I run the test feature at the bottom of the substitution page it seems to work fine - but not in the real world. I''m guessing that the ''bad'' urls are generated by javascript - might they be bypassing the substitution rules because of that? Should I send you a copy of my project file so that you can see the bad urls in the queue?

3. No problem about passing the macro value - it would just have been handy if the feature existed.

Thank you very very much for taking so much trouble over my ''problems''

Cami
Oleg Chernavin
11/08/2008 05:38 pm
You are welcome! I am glad to help you!

Can you please right-click the URL that should not be in the Queue and copy its Referer? Let me know it and I will see why such links bypass substitutes.

Oleg.
Cami
11/08/2008 11:23 pm
Here are a couple of examples:

URL: http://imko.deviantart.com/art/"http://www.deviantart.com/deviation/83080577/"
Referer: http://imko.deviantart.com/art/HDR-Wallpaper-Photo-Germany-2-83080577

URL: http://imko.deviantart.com/art/"http://www.deviantart.com/deviation/82922124/"
Referer: http://imko.deviantart.com/art/HDR-Wallpaper-Photo-Germany-82922124

Even if the fixed url turns out to be useless at least fixing it will avoid invoking too many errors on deviantart.

I hope this helps track down the problem. (And thanks, once again)

Cami
Oleg Chernavin
11/10/2008 09:00 am
I fixed this:

http://www.metaproducts.com/download/betas/opsetup.exe

You don''t need to use teh substitutes for this page now.

Oleg.
Cami
11/10/2008 12:59 pm
Thank you Oleg. It is now working perfectly. Your assistance has been far beyond anything I might have expected.
Oleg Chernavin
11/10/2008 01:01 pm
OK. Great!

Oleg.