These two are exactly identical pages with just different urls. How do I avoid it downloading the same pages over and over.
Uncheck the rule.
Because I think it`s still parsing the same pages with just different urls over and over.
Example of what I mean is:
It parses http://324lkj23l.sitetourl.com/watever.html
then it also parses
Which are identical files, but the url keep on changing every few hours meaning it will keep parsing the same urls over and over.
How do I stop it from parsing the same url over and over? My queue list has over 2,000,000 urls and I know for sure that the site doesn`t have that many.