> Thank you!
> Best regards,
> Oleg Chernavin
> MP Staff
Sometimes I will download http://www.website.com/ and it will also download http://website.com/ (which is a duplicate site) I know there is a URL filter for that kind of situation, but I would not know beforehand whether a site will do that or not, and I would have to literally go through every directory for every site I download to double check, then re-download if necessary
This will get rid of all duplicates when you download your site.