Trying to download forum, but OE is downloading too much

Author Message
Kols 01/07/2011 09:15 pm
I'm trying to download a full forum, however the program is downloading too much irrelevant stuff that I don't need.

Basically all I want to download are the forum sections and all the threads.

However this forum has many other links, such as: new posts, calendar, community, members, archives, blogs, etc...that I don't need.

Also in each post it has this feature where you can click a buttom with a "thanks" link, and so what OE is doing is it's downloading each of the "thanks" in every single post of every thread.

The structure of the forum is like this:

Forum sections:

www.website.com/forum/forumdisplay.php?f=123

Forum threads:

www.website.com/forum/showthread.php?t=12345

I only want these two.

Is there a way I can setup OE so that it only downloads these?

If not is there a way I can setup OE so it doesn't download certain pages like for example the thanks links which look like this:

www.website.com/forum/post_thanks.php?do=blablabla
Oleg Chernavin 01/08/2011 10:09 am
It is easy to do. Please use the Project Properties dialog - URL Filters - Filename section. Add the following keywords to the Included list:

forumdisplay.php
showthread.php

This should be enough.

Best regards,
Oleg Chernavin
MP Staff
Kols 01/08/2011 10:13 am
Excellent, thank you.
Oleg Chernavin 01/08/2011 10:42 am
You are welcome!

Oleg.
Kols 01/08/2011 10:53 am
Well everything seems to be working well now, but I have one more question, this isn't that big of a deal but it'd still be nice to know how to filter this if possible.

I've set the file keyword filter up like you suggested and it works very well.

I'm still getting a few pages that I don't really need like for example:

forumdisplay.php?f=10&daysprune=-1&order=desc&sort=lastpost
forumdisplay.php?f=10&daysprune=-1&order=desc&sort=voteavg
forumdisplay.php?f=10&daysprune=-1&order=desc&sort=views
forumdisplay.php?f=10&daysprune=-1&order=desc&sort=replycount
forumdisplay.php?f=10&daysprune=-1&order=asc&sort=title
forumdisplay.php?f=10&daysprune=-1&order=asc&sort=postusername
forumdisplay.php?f=10&daysprune=-1&order=asc&sort=lastpost&pp=20&page=1

all I would need here is:

forumdisplay.php?f=10

is there anyway to filter this as well?
Oleg Chernavin 01/08/2011 11:02 am
What about adding

daysprune

to the Excluded list in the same section?

Oleg.
Kols 01/08/2011 11:06 am
I tried that, didn't work.

However I paused or "suspended" the current download and THEN added the exclusion. Would I have to start from scratch for it to kick in?
Oleg Chernavin 01/08/2011 11:37 am
It is better to stop. Then select "Do not load existing files" in the Project Properties dialog. And start again. It will get only the missing files.

Oleg.
Kols 01/08/2011 12:11 pm
Yup, that did it. Looks like you can't suspend a project download, change options and have it start working with the new settings. Like you said you have to stop it first or just make a new project. But ya that worked. Thanks.
Oleg Chernavin 01/08/2011 12:42 pm
Actually, you can. And filtered out links will not appear in the queue. However the links that are already there will be not removed automatically.

Oleg.
Kols 01/08/2011 01:14 pm
Ah yes, thats sounds like what happened.

Thanks.