I want to add a large number of hmtl urls to a single project in order to explore and download all links from them, without a depth limit (such as java, redirected media files, etc.), but I don't want to explore any other htmls. Is there a way to do this?
Also, is there a way to automatically have added urls contain the username:password@... format.
I am dealing with a large number of pages, so I need a way to do this in batch, but I can't simply crawl the whole site
and so on.