The way this would work within OE would be to place URLs extracted from a page at the top of the queue instead of at the bottom. To make it work perfectly, I would also have to download one page at a time and insert a delay between pages so that there was time to parse.
The reason I need it is that there are some websites that offer CGI links that create the requested content in a temporary file, but as soon as you hit the next CGI link, the previous temporary file gets deleted. OE handles one case of this automatically: when the CGI returns a redirect. But if the CGI instead returns a page with a link, and accessing another page invalidates the previous page`s link, I can`t download it.
Is there already an option to cause new URLs to go to the top? Or is there another way to make this happen?
Yes, there is such option. You need to select your Project, click the Properties button on toolbar and add the following line to the URLs field: