Dealing with dynamic URLs

Author Message
Thierry LEROUX 12/27/2014 06:57 am
Hello,
I'm trying to download information on a site which URL is changing at each logon.
For example, the site URL is whatever001.site.com at the first login and whatelse002.site.com later on.
How such a site can be dealt with, owing to the fact that the information must be donwloaded over several days and with several successive several login and that, as many ZIP files are downloaded, I shall avoid repeatedly downloading the same files.
With best regards,
Thierry
Oleg Chernavin 12/27/2014 05:34 pm
You may use URL Substitutes rules in the Project Properties dialog - Parsing section.

The idea is to cut the changing part from a URL and apply this rule to the filenames only. Offline Explorer would download the URL as it is and convert it to some fixed format for keeping on the hard disk.

Another thing to do is to correctly use the File Modification Check, so it downloads HTML files to check for not yet downloaded links and skips media and ZIP files that were previously downloaded.

It is hard to tell exact rules and settings without looking at the site and how it is organized. But you know the idea to start now.

Best regards,
Oleg Chernavin
MP Staff