Download Ordering

01/08/2012 05:21 am
A web site allows only 100 stories to be downloaded a day. Stories all have a particular format, with dependent files. For example:


Just to make it even more fun, the last 5 link to a getZip.php, that link to a completely different .txt, .prc, etc. file.

For example:

12345 contains links to 12345:67899, plus 53245.
12345:67899 contains links to 12345, and 12345:67899;1.
12345:67899;1 contains links to 12345, 12345:67899;0, and 59342

All the above pages link to storyInfo, and storyInfo links to the 12345.* files. Those files link to get getZip URL (with 12345 in its arguments) which links to a completely different file of the form author@title.txt.

Now, stories will also link to other stories at any level. What I'd really like to do is download all 12345 files, before continuing on with another set of numbers, but I haven't figured out how to do it. Depthfirst helps, but doesn't do it completely since other stories can appear at any level and interrupt the original download.

I'm also confused about level limits. I added the txt, prc, pdb, epub, and mobi into User Defined, and changed User Defined to be "Only from the starting server." Sometimes it appears those files are downloaded beyond the level limit (URL Filter Settings, I assume) and other times they don't...

Even with the project starting with 12345, and {:filename} in the Included files, if I go above level 1 in the above example, I'll still attempt to download file 53245.

So a couple of different questions:

1) Any way to order the files received, so I can download all the 12345 files before downloading any others?

2) Above would take care of it, but if it doesn't, is there any way to create a project to just download a SINGLE story? Preferably without needing to modify both the Included Files AND URL...

Thanks for any assistance you can provide,

Oleg Chernavin
01/08/2012 05:24 am
The above looks like some puzzle. A bit hard to understand how exactly it is organized without looking at the site.

The single story is probably easy to download. The {:filename} should work for Inclued (URL Filters - Filename). However you need to make sure that all File Filters have the Location field set to "Load using URL Filters".

If you need more assistance from me, please write to with the links to the actual site, so I could see it myself.

Thank you!

Best regards,
Oleg Chernavin
MP Staff