cant download all pics

Author Message
RaTm7 12/13/2010 10:13 am
hello

i'm trying to download all pics from http://***/onlinereading/index.php (there are many directorys with other directorys behind, and it ends with embedded pic, which i want to download),but somehow it doesnt work, i removed ALL limitations (so OE only stay on this server/domain) so normally it should download all pics from this server/homepage

but it only loads the first pic of one directory(for example http://image.read.***/0001.jpg from http://read.****/0001.jpg&server=nas.html), but there are lot of more, up to over 200 in same directory, but OE dont load on, i ended with having one pic in every directory and dont know what to try now, i tryed lot of makros, but it still not work how it should

normally if i go to http://read.****/ch001/0001.jpg&server=nas.html, there is always link to next page (next page/next chapter button)
so my idea was to use first site as start-url and OE use then this "next" and "next chapter" links to go through all pages and save all pics, but like this really nothing happens, it only use "next" link once and stop then

i can save every pic manuell by right-klick and "save pic as", so its not locked or something like this

how could i solve this problem? to use the "next" link would be the best i think, and if i add "next" link manuelly it is used and i get 3 pics and not 2, but there are over 70000 directorys with over 20 pics in every one of them, so i would need weeks to make over million projekts to get really every pic...

the best would be if i could say "take this link of directory and download all pics there no metter which name they have", but there is no such makro...

it would make me really happy if someone could help me with that i spend already over a week to get 60000pics and there are still over 20 times more of them :(
Oleg Chernavin 12/13/2010 04:57 pm
Your description is very incomplete to do any advice. It only looks like maybe the Level limit is low. Maybe you should make it unlimited?

If this is not the case, please let me know the site you are downloading and your exact project settings - select it, use Export - Project Settings - Copy to Clipboard. Then paste to the forum message.

Best regards,
Oleg Chernavin
MP Staff
RaTm7 12/13/2010 07:46 pm
so here is the project(couldn't find how to export settings, so i write all below):
http://unixmanga.com/onlinereading/index.php
(i try to download all pics from this website, started with first http://image.read.homeunix.com/onlinereading/07%20Ghost/07%20Ghost%20c001/0001.jpg)

level limit: none
"download all files"

file filters: images only
URL exclusions: none
protocol: load all protocols
server: load files only within the starting "server" (no inclusions, no exclusions)
directory: no inclusions, no exclusions
filename: no inclusions, no exclusions

advanced: all limits are off
link translations: no translation

i thought maybe its because the start-URL is "unixmanga" and the URL with all pics is "image.read.homeunix", they are somehow different, but in every direcory i have already 1 pic, so it cant be the problem i think
i think all files should be about 200GB, but OE run through many times(4-5 times already) and all i get are ca 60000pics, most of them are like i already wrote in different directorys:
image.read.homeunix.com :
70.472 directorys
113.557 files (in every chapter folder is an descr.WD3 file, so there are about 60000pics at all)


if i go to:
http://read.homeunix.com/onlinereading/?image=07 Ghost/07 Ghost c001/0001.jpg&server=nas.html
then there is "next" below to show next pic, and if chapter is at the end, then there is "next chapter" link
(like here :
http://read.homeunix.com/onlinereading/?image=07 Ghost/07 Ghost c001/0046.jpg&server=nas.html)
but OE dont really use it, the best case i had til now is that OE downloaded 30+1 pics trough http://unixmanga.com/onlinereading/07_Ghost/07_Ghost_c001_nas.html
(there are only 30 desplayed + 1 through "next" button), but actually there are 46 in 07_ghost_c001 directory, so 15 are missing
Oleg Chernavin 12/14/2010 11:19 am
Please see this image on the Proect settings export:

http://www.metaproducts.com/files/OEProjectSave.png

Use Copy from this menu. Then paste in the forum message.

Oleg.
RaTm7 12/14/2010 11:27 am
Stream 1.2 File
[Object]
OEVersion=Standard 5.9.0.3283
Type=0
IID=7010
Caption=http://unixmanga.com/onlinereading/index.php
URL=http://unixmanga.com/onlinereading/index.php
Lev=1000040
Weekday=257
LimTSize=10000
LimNumber=5000
LimTime=100
EnableForms=True
FMGroup=1
LTMethod=2
FTText.Exts=htmlhtmaspaspxjspstmstmlidcshtmlhtxtxttextxspxmlrxmlcfmwmlphpphp3
FTImages.Exts=gifjpgjpegtiftiffxbmfifbmppngipxjp2j2cj2kwbmplwfwebp
FTVideo.Exts=mpgavianimpegmovflvfliflcvivrmramrvasfasxwmvm1vm2vvobsmilmp4m4v
FTAudio.Exts=wavriffmp3midmp2m3uravocwmaapeoggm4a
FTArchive.Exts=ziparcgzzarjlhalayleirarcabtarpakacejarpdftgzexe
FTUDef.Exts=jsaxdcssssivbsdtdxslswfclassent
FTText.B=xoxooo
FTImages.B=ooxooo
FTVideo.B=xoxooo
FTAudio.B=xoxooo
FTArchive.B=xoxooo
FTUDef.B=xoxooo
FTOther.B=xoxooo
FTSizes=0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,3,3,0,3,0
NotIgnoreLogout=False
RSrvsBx=1
RProt=127
LastStart=18:236:190:126:193:201:227:64:
LastEnd=0:3:85:139:193:201:227:64:
LastStarted=14.12.2010 01:07:16
LastEnded=14.12.2010 01:09:29
SAbr=56828
SLast=302
LFiles=15540
Stopped=True
Flags=1
ImgDim=0,0,0,0
ExploreDirs=True
ExploreSSMaps=True
ParseComplexScripts=True
ConvertRSS=True
IPAddr=819524781
Exported=14.12.2010 07:13:03 - C:\Users\RaTm7\Desktop\
LIndexed=False
IndexFiles=False
Oleg Chernavin 12/14/2010 11:46 am
I see now. It is easy. I used the Properties - URL Exclusions section to test the Next link. Its address is:

http://read.homeunix.com/onlinereading/?image=07 Ghost/07 Ghost c001/0002.jpg&server=nas.html

and as you see, it is located on another server. And you set the project to load only from the starting (unixmanga.com) server.

I would suggest you to uncheck that box and add the two servers to the URL Filters - Server - Included list:

unixmanga.com
read.homeunix.com

Oleg.
RaTm7 12/14/2010 11:55 am
actually all pics are saved at image.read.homeunix.com
so should i add this server too? or is it enough that pics are displayed at read.homeunix.com to save them?
Oleg Chernavin 12/14/2010 12:14 pm
You have "Load from any site" selected for File Filters - Images. So, no need to specify the separate server for them.

Oleg.
RaTm7 12/14/2010 12:16 pm
ok, then i try it like that now, i hope it will work

thanks and sorry for trouble :)
Oleg Chernavin 12/14/2010 01:12 pm
You are welcome!

Oleg.
RaTm7 01/02/2011 10:01 am
hello again and happy new year!

i tested it now how you told me, and now its loading much more, but the problem still the same somehow...

to run this project completely takes over 1day, so i did this:

=>i started project

=>as soon as the queue has over 6000 links i suspend

=>then i go to queue and delete as good as all links, i only let this one in queue:
http://unixmanga.com/onlinereading/Princess_Resurrection.html
(its one of these folders where should be over 31 pics in every chapter)

=>i click suspend again to go on with download

when its ready(there are only about 1500 URLs) there are only 31 pics instead of 203 in first folder, in other folders are missing some too...

what can i try now?

if you want to try to resume from file, there are only 55 chapter, so it only takes few minutes:

Stream 1.2 File
Count=2
[Object]
Type=3
IID=7010
KeepPrimary=False
[Object]
Type=2
IID=7010
URL=http://unixmanga.com/onlinereading/Princess_Resurrection.html
Img=True
Lev=1000038
AbsLev=2
FName=c:\download\unixmanga.com\onlinereading\Princess_Resurrection.html
LName=Princess Resurrection
Temp=
Path=c:\download\unixmanga.com\onlinereading
RFile=Princess_Resurrection.html
Depth=2
Referer=http://unixmanga.com/onlinereading/manga-lists.html
Retries=10
PFT=Text
LinkKind=2

i hope you can help me somehow, i dont want to copy every pic by myself or write over 100000 makros to get them :(

thank you
Oleg Chernavin 01/02/2011 10:20 am
What are the current project settings to reproduce?

Oleg.
RaTm7 01/02/2011 10:52 am
Stream 1.2 File
[Object]
OEVersion=Enterprise 5.6.0.3094
Type=0
IID=7010
Caption=http://unixmanga.com/onlinereading/index.php
URL=http://unixmanga.com/onlinereading/index.php
MVer=5
Lev=1000040
Weekday=257
LimTSize=10000
LimNumber=5000
LimTime=100
EnableForms=True
FMGroup=2
LTMethod=2
FTText.Exts=htmlhtmaspaspxjspstmstmlidcshtmlhtxtxttextxspxmlrxmlcfmwmlphpphp3
FTImages.Exts=gifjpgjpegtiftiffxbmfifbmppngipxjp2j2cj2kwbmplwfwebp
FTVideo.Exts=mpgavianimpegmovflvfliflcvivrmramrvasfasxwmvm1vm2vvobsmilmp4m4v
FTAudio.Exts=wavriffmp3midmp2m3uravocwmaapeoggm4a
FTArchive.Exts=ziparcgzzarjlhalayleirarcabtarpakacejarpdftgzexe
FTUDef.Exts=jsaxdcssssivbsdtdxslswfclassent
FTText.B=xoxooo
FTImages.B=ooxooo
FTVideo.B=xoxooo
FTAudio.B=xoxooo
FTArchive.B=xoxooo
FTUDef.B=xoxooo
FTOther.B=xoxooo
FTSizes=0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,3,3,0,3,0
NotIgnoreLogout=False
RSrvsIn=image.read.homeunix.comread.homeunix.comunixmanga.com xxx
RProt=127
LastStart=150:9:191:76:52:204:227:64:
LastEnd=170:6:116:1:54:204:227:64:
LastStarted=02.01.2011 15:13:29
LastEnded=02.01.2011 16:30:15
S200=1924
S304=1693
S400=117
SAbr=5277
SPar=1955
SLast=200
SSiz=13864037
LFiles=3634
LSize=5662477
Flags=1
ImgDim=0,0,0,0
PrevURL=http://unixmanga.com/onlinereading/index.php
ExploreDirs=True
ExploreSSMaps=True
ParseComplexScripts=True
ConvertRSS=True
IPAddr=819524781
Exported=14.12.2010 07:13:03 - C:\Users\RaTm7\Desktop\
LIndexed=False
IndexFiles=False
RaTm7 01/06/2011 05:26 pm
now i found a way to download them all, its not really nice solution, but it works

if i use "download all" and not "download only pics" then OE really download them all
i just have to delete all things which i dont need after dl is ready

i dont really know why it works that way but its just like it is :)

thank you for your time :)
Oleg Chernavin 01/07/2011 04:24 am
I am sorry for not answering in time. Was out of town and sick a little bit. Good that it works for you. Yes, sometimes it is easier to load more than fight with the site and different URLs trying to filter out.

Oleg.
RaTm7 01/07/2011 10:12 am
no need to feel sorry, its ok :)
hope you are feeling better now

too bad that i didn't tryed it earlyer...

thank you again, have a nice day!

P.S. OE is really amazing when you know how to handle it, you all have done really good job by programming it :)
Oleg Chernavin 01/07/2011 12:53 pm
Thank you very much for your warm words!

Oleg.