how to prevent loading html file except for starting url
|Mark Greenberg||12/02/2011 09:39 am|
I want to add a large number of hmtl urls to a single project in order to explore and download all links from them, without a depth limit (such as java, redirected media files, etc.), but I don't want to explore any other htmls. Is there a way to do this?
Also, is there a way to automatically have added urls contain the username:password@... format.
I am dealing with a large number of pages, so I need a way to do this in batch, but I can't simply crawl the whole site
|Oleg Chernavin||12/02/2011 10:41 am|
|Yes, sure. You may specify any number of URLs (even with password in the URL). Set Level to 1 and use URL Filters - Filename section to exclude most popular HTML extensions:
and so on.