It has been a while, hope you are well.
I searched in the forums but didn't find an exact answer to my question. Does OEP have a setting to "ignore robots.txt" files? I have downloaded a site and it appears that the robots.txt at the host site has prevented OEP from grabbing style sheets. Also, when I tried to export it to see if that would improve the rendering, it froze after 5 files.
I've pasted the settings below. Thanks for your help!
[Object]
OEVersion=Pro 5.9.0.3254
Type=0
IID=7032
Caption=http://www.afghanistan.gc.ca/canada-afghanistan/menu.aspx
URL=http://www.afghanistan.gc.ca/canada-afghanistan/menu.aspx
Lev=1000001
Weekday=257
LimTSize=10000
LimNumber=5000
LimTime=100
FTText.Exts=htmlhtmaspaspxjspstmstmlidcshtmlhtxtxttextxspxmlrxmlcfmwmlphpphp3
FTImages.Exts=gifjpgjpegtiftiffxbmfifbmppngipxjp2j2cj2kwbmplwf
FTVideo.Exts=mpgavianimpegmovflvfliflcvivrmramrvasfasxwmvm1vm2vvobsmilmp4
FTAudio.Exts=wavriffmp3midmp2m3uravocwmaape
FTArchive.Exts=ziparcgzzarjlhalayleirarcabtarpakacejarpdftgzexe
FTUDef.Exts=jsaxdcssssivbsdtdxslswfclassent
FTText.B=ooxooo
FTImages.B=ooxooo
FTVideo.B=ooxooo
FTAudio.B=ooxooo
FTArchive.B=ooxooo
FTUDef.B=ooxooo
FTOther.B=ooxooo
FTSizes=0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,3,3,0,3,0
NotIgnoreLogout=False
RSrvsBx=1
RProt=255
LastStart=203:160:100:174:150:94:228:64:
LastEnd=59:68:24:54:153:94:228:64:
LastStarted=2014-03-18 5:00:39 PM
LastEnded=2014-03-18 6:54:30 PM
S200=4162
S400=66
SPar=3047
SSav=4162
SLast=200
SSiz=10301109123
SMdf=4105
SHTML=2399
SSuccDowns=1
LFiles=4226
LSize=10301164218
ImgDim=0,0,0,0
PrevURL=http://www.afghanistan.gc.ca/canada-afghanistan/menu.aspx
ConvertRSS=True
Exported=2014-03-19 9:19:47 AM - W:\patty.klambauer\Download\afghanistan-export-mar19\
LIndexed=False
IndexFiles=False
It was not related to robots.txt at all. I fixed an error in Offline Explorer. Here is the updated version:
http://www.metaproducts.com/download/betas/opsetup.exe
Thank you!
Best regards,
Oleg Chernavin
MP Staff
I'm curious: how does OEP normally treat robots.txt files? Does it ignore them? and is there a setting that we can switch on and off for robots.txt?
Oleg.
[Object]
OEVersion=Pro 6.8.4085
Type=0
IID=7032
Caption=http://www.afghanistan.gc.ca/canada-afghanistan/menu.aspx
URL=http://www.afghanistan.gc.ca/canada-afghanistan/menu.aspx
Lev=1000001
Weekday=257
LTExceptions=
LTExcMode=0
FTText.Exts=htmlhtmaspaspxjspstmstmlidcshtmlhtxtxttextxspxmlrxmlcfmwmlphpphp3
FTImages.Exts=gifjpgjpegtiftiffxbmfifbmppngipxjp2j2cj2kwbmplwf
FTVideo.Exts=mpgavianimpegmovflvfliflcvivrmramrvasfasxwmvm1vm2vvobsmilmp4
FTAudio.Exts=wavriffmp3midmp2m3uravocwmaape
FTArchive.Exts=ziparcgzzarjlhalayleirarcabtarpakacejarpdftgzexe
FTUDef.Exts=jsaxdcssssivbsdtdxslswfclassent
FTText.B=ooxooo
FTImages.B=ooxooo
FTVideo.B=ooxooo
FTAudio.B=ooxooo
FTArchive.B=ooxooo
FTUDef.B=ooxooo
FTOther.B=ooxooo
FTSizes=0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,3,3,0,3,0,0,0,0,0,0,0,0
NotIgnoreLogout=False
RSrvsBx=1
RProt=255
LastStart=252:188:247:12:112:95:228:64:
LastEnd=115:60:180:84:116:95:228:64:
PrjStart=148:125:165:232:78:95:228:64:
LastStarted=2014-03-25 12:02:16 PM
LastEnded=2014-03-25 3:14:53 PM
S200=4154
S400=19
SPar=3067
SSav=4154
SLast=200
SSiz=10299670534
SMdf=4096
SHTML=2414
SSuccDowns=4
LFiles=4170
LSize=10299688343
Flags=1
ImgDim=0,0,0,0
PrevURL=http://www.afghanistan.gc.ca/canada-afghanistan/menu.aspx
ConvertRSS=True
Exported=2014-03-25 3:25:34 PM - W:\patty.klambauer\Download\2014-03-25-afghanistan-export-1\
MapStats=1,199,1,199,0,0,0,0,0,0,0,0,0,0
Can you delete the downloaded Project and maybe download it to some other directory to check?
Oleg.
I don’t think the export is necessarily the issue because there are lots of downloaded files that indicate “not found” in their file path. I have pasted a small sample below. Do you have the same “not found” files in your download?
Patricia
notfound.aspx@404_253b_2fcanada_international_2fimportant_notices.aspx.htm
notfound.aspx@404_253bhttp_3a_2f_2fwww.afghanistan.gc.ca_2fcanada-afghanistan_2fassets_2fimages_2fanca-class1.jpg
notfound.aspx@404_253Bhttp_3A_2F_2Fwww.afghanistan.gc.ca_2Fcanada-afghanistan_2Fassets_2Fpdfs_2FCanadaCondemnsKabulAttackDari.pdf
notfound.aspx@404_253Bhttp_3A_2F_2Fwww.afghanistan.gc.ca_2Fiwglobal_2Fframeworks_2Fcss_2Fcustom.css
notfound.aspx@404_253Bhttp_3A_2F_2Fwww.afghanistan.gc.ca_2Fiwglobal_2Fframeworks_2Fjs_2Fcss_2Fpe-ap-min.css
notfound.aspx@404_253Bhttp_3A_2F_2Fwww.afghanistan.gc.ca_2Fcanada-afghanistan_2Fwet-boew.skipnav.js
Would it make a difference?
Oleg.
Patricia
Oleg.