Robots.txt question

AN Other
11/28/2008 05:19 am
Does this program override www.SITENAME/robots.txt by default??

Interested to know, as I''m currently using a rival product which I have to set robots.txt to be ignored if I want to override the robots.txt.

Thanks
AN Other
Oleg Chernavin
11/28/2008 07:47 am
Offline Explorer uses its own exclusions in the URL Filters - Directory and Filename sections of the Project Propeties dialog. You can setup flexible keywords there to download or skip certain sections of Web sites. Therefore there is no need to use robots.txt files.

Best regards,
Oleg Chernavin
MP Staff
AN Other
11/28/2008 08:32 am
> Offline Explorer uses its own exclusions in the URL Filters - Directory and Filename sections of the Project Propeties dialog. You can setup flexible keywords there to download or skip certain sections of Web sites. Therefore there is no need to use robots.txt files.
>
> Best regards,
> Oleg Chernavin
> MP Staff

Thank you for that explanation. So it basically overrides robots.txt if I understand correctly?
Oleg Chernavin
11/28/2008 09:01 am
Yes, as a simple explanation, it works this way.

Oleg.