cant download all pics
|RaTm7||12/13/2010 10:13 am|
i'm trying to download all pics from http://***/onlinereading/index.php (there are many directorys with other directorys behind, and it ends with embedded pic, which i want to download),but somehow it doesnt work, i removed ALL limitations (so OE only stay on this server/domain) so normally it should download all pics from this server/homepage
but it only loads the first pic of one directory(for example http://image.read.***/0001.jpg from http://read.****/0001.jpg&server=nas.html), but there are lot of more, up to over 200 in same directory, but OE dont load on, i ended with having one pic in every directory and dont know what to try now, i tryed lot of makros, but it still not work how it should
normally if i go to http://read.****/ch001/0001.jpg&server=nas.html, there is always link to next page (next page/next chapter button)
so my idea was to use first site as start-url and OE use then this "next" and "next chapter" links to go through all pages and save all pics, but like this really nothing happens, it only use "next" link once and stop then
i can save every pic manuell by right-klick and "save pic as", so its not locked or something like this
how could i solve this problem? to use the "next" link would be the best i think, and if i add "next" link manuelly it is used and i get 3 pics and not 2, but there are over 70000 directorys with over 20 pics in every one of them, so i would need weeks to make over million projekts to get really every pic...
the best would be if i could say "take this link of directory and download all pics there no metter which name they have", but there is no such makro...
it would make me really happy if someone could help me with that i spend already over a week to get 60000pics and there are still over 20 times more of them :(
|Oleg Chernavin||12/13/2010 04:57 pm|
|Your description is very incomplete to do any advice. It only looks like maybe the Level limit is low. Maybe you should make it unlimited?
If this is not the case, please let me know the site you are downloading and your exact project settings - select it, use Export - Project Settings - Copy to Clipboard. Then paste to the forum message.
|RaTm7||12/13/2010 07:46 pm|
|so here is the project(couldn't find how to export settings, so i write all below):
(i try to download all pics from this website, started with first http://image.read.homeunix.com/onlinereading/07%20Ghost/07%20Ghost%20c001/0001.jpg)
level limit: none
"download all files"
file filters: images only
URL exclusions: none
protocol: load all protocols
server: load files only within the starting "server" (no inclusions, no exclusions)
directory: no inclusions, no exclusions
filename: no inclusions, no exclusions
advanced: all limits are off
link translations: no translation
i thought maybe its because the start-URL is "unixmanga" and the URL with all pics is "image.read.homeunix", they are somehow different, but in every direcory i have already 1 pic, so it cant be the problem i think
i think all files should be about 200GB, but OE run through many times(4-5 times already) and all i get are ca 60000pics, most of them are like i already wrote in different directorys:
113.557 files (in every chapter folder is an descr.WD3 file, so there are about 60000pics at all)
if i go to:
http://read.homeunix.com/onlinereading/?image=07 Ghost/07 Ghost c001/0001.jpg&server=nas.html
then there is "next" below to show next pic, and if chapter is at the end, then there is "next chapter" link
(like here :
http://read.homeunix.com/onlinereading/?image=07 Ghost/07 Ghost c001/0046.jpg&server=nas.html)
but OE dont really use it, the best case i had til now is that OE downloaded 30+1 pics trough http://unixmanga.com/onlinereading/07_Ghost/07_Ghost_c001_nas.html
(there are only 30 desplayed + 1 through "next" button), but actually there are 46 in 07_ghost_c001 directory, so 15 are missing
|Oleg Chernavin||12/14/2010 11:19 am|
|Please see this image on the Proect settings export:
Use Copy from this menu. Then paste in the forum message.
|RaTm7||12/14/2010 11:27 am|
|Stream 1.2 File
Exported=14.12.2010 07:13:03 - C:\Users\RaTm7\Desktop\
|Oleg Chernavin||12/14/2010 11:46 am|
|I see now. It is easy. I used the Properties - URL Exclusions section to test the Next link. Its address is:
http://read.homeunix.com/onlinereading/?image=07 Ghost/07 Ghost c001/0002.jpg&server=nas.html
and as you see, it is located on another server. And you set the project to load only from the starting (unixmanga.com) server.
I would suggest you to uncheck that box and add the two servers to the URL Filters - Server - Included list:
|RaTm7||12/14/2010 11:55 am|
|actually all pics are saved at image.read.homeunix.com
so should i add this server too? or is it enough that pics are displayed at read.homeunix.com to save them?
|Oleg Chernavin||12/14/2010 12:14 pm|
|You have "Load from any site" selected for File Filters - Images. So, no need to specify the separate server for them.
|RaTm7||12/14/2010 12:16 pm|
|ok, then i try it like that now, i hope it will work
thanks and sorry for trouble :)
|Oleg Chernavin||12/14/2010 01:12 pm|
|You are welcome!
|RaTm7||01/02/2011 10:01 am|
|hello again and happy new year!
i tested it now how you told me, and now its loading much more, but the problem still the same somehow...
to run this project completely takes over 1day, so i did this:
=>i started project
=>as soon as the queue has over 6000 links i suspend
=>then i go to queue and delete as good as all links, i only let this one in queue:
(its one of these folders where should be over 31 pics in every chapter)
=>i click suspend again to go on with download
when its ready(there are only about 1500 URLs) there are only 31 pics instead of 203 in first folder, in other folders are missing some too...
what can i try now?
if you want to try to resume from file, there are only 55 chapter, so it only takes few minutes:
Stream 1.2 File
i hope you can help me somehow, i dont want to copy every pic by myself or write over 100000 makros to get them :(
|Oleg Chernavin||01/02/2011 10:20 am|
|What are the current project settings to reproduce?
|RaTm7||01/02/2011 10:52 am|
|Stream 1.2 File
Exported=14.12.2010 07:13:03 - C:\Users\RaTm7\Desktop\
|RaTm7||01/06/2011 05:26 pm|
|now i found a way to download them all, its not really nice solution, but it works
if i use "download all" and not "download only pics" then OE really download them all
i just have to delete all things which i dont need after dl is ready
i dont really know why it works that way but its just like it is :)
thank you for your time :)
|Oleg Chernavin||01/07/2011 04:24 am|
|I am sorry for not answering in time. Was out of town and sick a little bit. Good that it works for you. Yes, sometimes it is easier to load more than fight with the site and different URLs trying to filter out.
|RaTm7||01/07/2011 10:12 am|
|no need to feel sorry, its ok :)
hope you are feeling better now
too bad that i didn't tryed it earlyer...
thank you again, have a nice day!
P.S. OE is really amazing when you know how to handle it, you all have done really good job by programming it :)
|Oleg Chernavin||01/07/2011 12:53 pm|
|Thank you very much for your warm words!