Limit the harvest to a subdirectory

Manny
01/28/2006 07:22 pm
I need to down load pages
http://www.srigranth.org/servlet/gurbani.gurbani?Action=Page&Param=1 to
http://www.srigranth.org/servlet/gurbani.gurbani?Action=Page&Param=1430

and nothing else, how do i put url limit
i really need to extract the first 2 lines in unicode punjabi and hindi
thanks for help
Manny
Steve Sieloff
01/28/2006 10:23 pm
Manny --

I believe you will get what you want if you

1) set up the download to use a macro

http://www.srigranth.org/servlet/gurbani.gurbani?Action=Page&Param={:1..1430}

2) click the "level limit" on and set to 0 (this will have the effect of OE Pro not following any links on the single page ... it will only download param=1, param=2 ... param=1430

Let me know if this works!

Steve

> I need to down load pages
> http://www.srigranth.org/servlet/gurbani.gurbani?Action=Page&Param=1 to
> http://www.srigranth.org/servlet/gurbani.gurbani?Action=Page&Param=1430
>
> and nothing else, how do i put url limit
> i really need to extract the first 2 lines in unicode punjabi and hindi
> thanks for help
> Manny
Manny
02/04/2006 11:03 pm
Hi I am using this macro as suggested

http://www.srigranth.org/servlet/gurbani.gurbani?Action=Page&Param={:1..1430}
click the "level limit" on and set to 0 (this will have the effect of OE Pro not following any links on the single page ... it will only download param=1, param=2 ... param=1430

My Problems:
the server has some security built in that after 25-30 pages or so, i download only the footer of the page. There might be ip banning involved. how can i circumvent that or add pauses between down loads

How can i rename the files to .htm

thanks for all help
with regards
Manny

>
> > I need to down load pages
> > http://www.srigranth.org/servlet/gurbani.gurbani?Action=Page&Param=1 to
> > http://www.srigranth.org/servlet/gurbani.gurbani?Action=Page&Param=1430
> >
> > and nothing else, how do i put url limit
> > i really need to extract the first 2 lines in unicode punjabi and hindi
> > thanks for help
> > Manny
Oleg Chernavin
02/05/2006 10:30 am
I would suggest you to set 2-5 seconds delay between downloads in the Options dialog and use 1 connection there. This way you will emulate human browsing.

To add .htm extension to the downloaded files, use File | Export with the Standard Extensions box checked.

Best regards,
Oleg Chernavin
MP Staff