Difference between url links in browser and OE interferes with downloads

Author Message
Jeff 04/24/2012 04:34 pm

The problem is the links are sometimes displayed differently in my Firefox browser than in the OE browser. And that cause the projects to not retrieve files properly.

This is may be tough to describe since the content of the site is constantly changing. If I could add a couple of pictures it might be easier to see the problem.

To see the problem go the http://www.reddit.com/r/pics in the OE browser. Mouse down the page over the links which show as titles. As you mouse down in OE, watch the URL that shows. Most point to a site called http://imgur.com/ and have a random file names that end with ".jpg" Keep going until you see one on that site that does not have that extension.

Then open the same page in Firefox and look at the same link. Chances are it will show the JPG extension. If not, try again and you soon will find one.

When this happens, OE does not recognize the link as an image so it does not try to download it because of the project settings. I need to limit it to images or I get way too much content.

A while ago, you had me change the OE browser to solve another problem. That may not be related to the problem but thought it was worth mentioning.

Oleg Chernavin 04/25/2012 05:08 am
I see now. Well, one idea is to limit the download to the www.reddit.com and imgur.com in URL Filters - Server section - Included servers list.

Would this work?

Best regards,
Oleg Chernavin
MP Staff
Jeff 04/25/2012 01:35 pm
Hi Oleg,

Always nice to get a response from you as I know the issue is in good hands.

Using filters may provide a solution, but I have tinkered with it for hours and haven't found a way to do it.

The problem I run into using filters is that the main page of reddit.com and has links to all of their other forums on that main page. So I get huge amounts of content I am not interested in unless I use directory includes. And if I limit them with directory includes, that messes up the ability to get the files off imgur since they appear at the root level.

Also, when the imgur.com link does not have a JPG extension, the imgur site loads a page with many other unrelated images. So I get all those too.

If I could figure out a way to know what the reddit.com page links would be for each of the /r/pics pages, I could probably figure out a way to do this. But unfortunately they change so I can't hard-code them or use variables.

So maybe I am using the wrong approach and should be asking this a different way. Can I create a project that will start on one site (reddit.com) and go down an unlimited number of pages under a single directory (/r/pics) on that site and have it open only one link to pages it finds on a different server (imgur.com), or short list of servers, that I can specify?

Oleg Chernavin 04/26/2012 06:12 am

it is quite hard to make the settings to get only and exactly what you want this way. However what about limiting to the starting server (URL Filters - Server) and specifying only the selected directories (URL Filters - Directory - Included list) on the starting site. Then allow to download up to 1 link on external server from all directories in URL Filters - Server?

Would this work?

Jeff 04/26/2012 08:18 am
Hi Oleg,

I kind of figured this would be really hard to do since I wasn't able to figure it out in a couple of hours. IT was in trying to figure this out that I came up with my original question asking about the difference between the Firefox and OE browser URLs. Because if those were consistent, it would get me part of the way there.

And I think I may have already tried what you suggested. But I will give it a try later on just to be sure and let you know how it goes.

Oleg Chernavin 04/26/2012 11:07 am
Jeff, I tried to find differencies in the links in browsers - they were the same for me.