can I batch download URLs?

Author Message
Bob 08/31/2007 12:35 am

I'm trying to get access to bullet clipart off of MS site.


Basically, the URL stays the same and URI parameters change for the page number.

like

Page 1 URL is http://www.site.com?cat=1,size=l,page=1
Page 2 http://www.site.com?cat=1,size=l,page=2
etc etc
Page 9 http://www.site.com?cat=1,size=l,page=9

The only thing that would change is the page key, cat=1 couild mean something like

Can I set up an OE project that would let me just define the 9 urls and dload
whatever is on the page, and NOT process any page links on the dlownloaded page?

Make any sense?

Bob


Oleg Chernavin 08/31/2007 04:41 am
Yes, it is very easy. You will need to create a Project with the URL:

http://www.site.com?cat=1,size=l,page={:1..9}

and Level=0.

The above {:.....} in the URL is called URL Macro, you will be able to read in the Help file about them.

Best regards,
Oleg Chernavin
MP Staff
Bob 09/03/2007 11:41 pm
> Yes, it is very easy. You will need to create a Project with the URL:
>
> http://www.site.com?cat=1,size=l,page={:1..9}
>
> and Level=0.
>
> The above {:.....} in the URL is called URL Macro, you will be able to read in the Help file about them.
>
> Best regards,
> Oleg Chernavin
> MP Staff


Can I cointrol indexing based on # graphics on screen?

Page 1 starts at graphic 1.....
Page 2 starts at graphics 41.....
Page 3 at graphic 81....


Something like.... http://www.site.com?cat=1,size=l,page={:1...41...81..2001}?
Basically defining an incrememnt of 40 per page, instead of a page #?


note: 2001 would be last picture

Bob



Oleg Chernavin 09/04/2007 04:36 am
I think, it should be this way:

http://www.site.com?cat=1,size=l,page={:1..2001|40}

The above means from 1 to 2001 with the increment by 40.

Oleg.
Bob 09/12/2007 10:47 pm

Trying to use this URL macro thing.

At Clipart.com, I want to download pages 10-20 of a section (called cid)....

Here is the URL for Page 2:

http://www.clipart.com/en/search/split?page=2&ocols=6&width=600&a=w&rows=3&k_exc=&b=k&q_color=1&batch=18&cid=4h&q_bw=1&sub1=&date=&q_gif=1&orows=3&sh=&q_png=1&q=&total=569602&q_psd=1&q_jpeg=1&srch=Searching%2E%2E%2E&k_mode=all

I updated to do this....

http://www.clipart.com/en/search/split?
page={:10..20|1}&ocols=6&width=600&a=w&rows=3&k_exc=&b=k&q_color=1&batch=18&cid=4h&q_bw=1&sub1=&date=&q_gif=1&orows=3&sh=&q_png=1&q=&total=569602&q_psd=1&q_jpeg=1&srch=Searching%2E%2E%2E&k_mode=all


Using the wizard, I plug in the 2nd url here,
Set Levels=0
Load only from starting URL
and run,

Loaded over 3000 pages, which when viewed, are html pages. didn't capture a single image.

I'm dzed and cornfused!

Bob
Oleg Chernavin 09/13/2007 12:51 pm
OK. And what it you load one page with not URL Macros - would it load images?

Oleg.
Bob 09/13/2007 01:01 pm
> OK. And what it you load one page with not URL Macros - would it load images?
>
> Oleg.


If I am not mistaken....I can put in the first URL into the wizard, not change anything, and I get it all.

I was hoping (originally) of creating a .txt file, use a single line lines per poage and use that as input to the Pro program. Load a page/save, read record, load that page/save, etc etc

Bob
Oleg Chernavin 09/13/2007 01:17 pm
Yes, txt file way is also possible. You can use the {:file=c:\file.txt} macro for that.

Oleg.