Help with download

Author Message
Steve Sieloff 05/21/2004 06:12 pm
Oleg --

I am trying to get OE Pro to download all the links for this site but OEP stops with just current pages {:a..z} and their embedded links -- it is not following the "Next 50" link to subsequent pages. I think I have done all the appropriate steps (file filters in play, level limit off, evaluate Javascript in advanced is on} but no luck. I could add another macro toincrement the Top=0&Pos=Next+50+ portion of the URL/Post like Top={:0..500000|50}&Pos=Next+50+ but I do not know how many subsequent pages are needed and do not want to put in an arbitrary number/limit. Here is my project setting:

[Object]
OEVersion=Pro 3.2.0.1583
Type=0
IID=353
Caption=http://www.sos.state.ms.us/Busserv/corp/soskb/searchresults.asp
URL=http://www.sos.state.ms.us/Busserv/corp/soskb/searchresults.aspPOST=FormName=CorpNameSearch&Words=Starting&searchstr={:a..z}&OnlyActive=&TopRec=0&Pos=Next+50+%3E%3EIgnoreLogOutLinksadditional=ConvertPOSTToFileNameReferer=http://www.sos.state.ms.us/Busserv/corp/soskb/SearchResults.asp?FormName=CorpNameSearch&Words=Starting&SearchStr=a&SearchType=SearchSetCookie=ASPSESSIONIDSADRASRC=KCHMJMNCJBEOJHLDOJHNNGOP
Lev=1000001
Weekday=257
LimTSize=10000
LimNumber=5000
LimTime=100
FTText.Exts=htmlhtmaspjspstmstmlidcshtmlhtxtxttextxspxmlrxmlcfmwmlphpphp3
FTImages.Exts=gifjpgjpegtiftiffxbmfifbmppngipxjp2j2cj2kwbmplwf
FTVideo.Exts=mpgavianimpegmovfliflcvivrmramrvasfasxwmvm1vm2vvob
FTAudio.Exts=wavriffmp3midmp2m3uravocwmaape
FTArchive.Exts=ziparcgzzarjlhalayleirarcabtarpakacejar
FTUDef.Exts=jscssssivbsdtdxsl
FTText.B=oooooo
FTImages.B=xooooo
FTVideo.B=xooooo
FTAudio.B=xooooo
FTArchive.B=xooooo
FTUDef.B=oooooo
FTOther.B=oooooo
FTSizes=0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,3,0,3,0
RFileBx=2
RFileIn=searchresults.aspCorp.asp?Names.asp?PItemID=SearchResults.asp? xxxx
RProt=63
LastStart=27:96:131:204:23:158:226:64:
LastEnd=76:240:135:220:23:158:226:64:
S200=61
SAbr=1302
SPar=61
SSav=61
SLast=200
SSiz=2103477
SMdf=61
LFiles=61
LSize=2103477
Stopped=True
ImgDim=0,0,0,0
PrevURL=http://www.sos.state.ms.us/Busserv/corp/soskb/searchresults.asp
ParseComplexScripts=True

Thanks for any assistance you can provide!

Steve
Oleg Chernavin 05/22/2004 08:24 am
Please try to enable the "Explore HTML forms" in the Project Properties dialog | Advanced section. The Next button on the page is an HTML form with just a button and few hidden fields.

This should work.

Best regards,
Oleg Chernavin
MP Staff
Steve Sieloff 05/22/2004 02:52 pm
Oleg --

Thanks! Worked like a charm ... your support and product come through again!!!!

I love this software!!

Steve

> Please try to enable the "Explore HTML forms" in the Project Properties dialog | Advanced section. The Next button on the page is an HTML form with just a button and few hidden fields.
>
> This should work.
>
> Best regards,
> Oleg Chernavin
> MP Staff
Steve Sieloff 05/22/2004 05:20 pm
Oleg --

I spoke too soon ... the project loaded the Next+50 links but when they were processed in the Queue none of the Corp.asp links were processed or added to the queue ... OEP just read thru the links as if they contained no subordinate links. The entire project downloaded 5300+ files .... I am expecting > 500,000 overall!

Any other ideas?

Thanks for your assistance!

Steve

[Object]
OEVersion=Pro 3.2.0.1583
Type=0
IID=353
Caption=http://www.sos.state.ms.us/Busserv/corp/soskb/searchresults.asp
URL=http://www.sos.state.ms.us/Busserv/corp/soskb/searchresults.aspPOST=FormName=CorpNameSearch&Words=Starting&searchstr={:a..z}&OnlyActive=&TopRec=0&Pos=Next+50+%3E%3EIgnoreLogOutLinksadditional=ConvertPOSTToFileNameReferer=http://www.sos.state.ms.us/Busserv/corp/soskb/SearchResults.asp?FormName=CorpNameSearch&Words=Starting&SearchStr=a&SearchType=SearchSetCookie=ASPSESSIONIDSADRASRC=KCHMJMNCJBEOJHLDOJHNNGOP
Lev=1000001
Weekday=257
LimTSize=10000
LimNumber=5000
LimTime=100
EnableForms=True
FTText.Exts=htmlhtmaspjspstmstmlidcshtmlhtxtxttextxspxmlrxmlcfmwmlphpphp3
FTImages.Exts=gifjpgjpegtiftiffxbmfifbmppngipxjp2j2cj2kwbmplwf
FTVideo.Exts=mpgavianimpegmovfliflcvivrmramrvasfasxwmvm1vm2vvob
FTAudio.Exts=wavriffmp3midmp2m3uravocwmaape
FTArchive.Exts=ziparcgzzarjlhalayleirarcabtarpakacejar
FTUDef.Exts=jscssssivbsdtdxsl
FTText.B=oooooo
FTImages.B=xooooo
FTVideo.B=xooooo
FTAudio.B=xooooo
FTArchive.B=xooooo
FTUDef.B=oooooo
FTOther.B=oooooo
FTSizes=0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,3,0,3,0
RFileBx=2
RFileIn=searchresults.aspCorp.asp?Names.asp?PItemID=SearchResults.asp? xxxx
RProt=63
LastStart=255:114:5:36:49:158:226:64:
LastEnd=68:61:185:165:52:158:226:64:
S200=5330
SPar=5330
SSav=5330
SLast=200
SSiz=115823999
SMdf=5308
LFiles=5330
LSize=115823999
Flags=1
ImgDim=0,0,0,0
PrevURL=http://www.sos.state.ms.us/Busserv/corp/soskb/searchresults.asp
ParseComplexScripts=True


> Oleg --
>
> Thanks! Worked like a charm ... your support and product come through again!!!!
>
> I love this software!!
>
> Steve
>
> > Please try to enable the "Explore HTML forms" in the Project Properties dialog | Advanced section. The Next button on the page is an HTML form with just a button and few hidden fields.
> >
> > This should work.
> >
> > Best regards,
> > Oleg Chernavin
> > MP Staff
Oleg Chernavin 05/24/2004 07:37 am
Steve,

It looks like you will have to use URL Macros. The Next button doesn`t contain information on from which item the search results page should display links. So, all subsequent requests to the server look absolutely the same and Offline Explorer loads pages with the same contents, because the server gets confused with so many simultaneous requests.

Also, there are many pages with an error like that:

HTTP 500.100 - Internal Server Error - ASP error
Internet Information Services

--------------------------------------------------------------------------------

Technical Information (for support personnel)

Error Type:
Microsoft VBScript runtime (0x800A01A8)
Object required: `mcApp.conn`
/busserv/corp/soskb/modUtility.asp, line 330


Browser Type:
Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt; MyIE2)

Page:
POST 90 bytes to /Busserv/corp/soskb/searchresults.asp

POST Data:
FormName=CorpNameSearch&Words=Starting&searchstr=k&OnlyActive=&TopRec=0&Pos=Next+50+%3E%3E

Time:
Monday, May 24, 2004, 6:23:23 AM

Oleg.