I have a problem downloading a website that is full of PHP (using OE E 3.6). When I go into the videos and pics that I want to download, it does a PHP type redirect to another site I believe where the pics and videos are actually located, without changing the URL.
So for example, when I click on a gallery of pics that I want to download it goes to this URL:
When I actually click on the JPG picture it goes to this URL:
Sorry I can`t give the site out because it`s a members only site, also the pics and videos are accessed via my username and password (which I have entered in the project).
It comes up with similar style urls when I click on the videos and streaming videos links which it also doesn`t download. For example, page with video files to download off:
When you actually click on the video file the URL becomes very long and werid!
I have tried numerous options in the project, I have levels unticked, tried checking just pictures and videos file types and specified keywords for the servers it goes to get the pictures and videos. However, this didn`t work properly, so I tried ticking all the file type options and all servers, but this just downloaded a lot of junk. Basically, lots of long file names of the url of 1kb. Loads and loads of them! I have also tried starting off the download with the Referer= line to the url also but this didn`t seem to help.
It seems to be able to download links/files from the PHP files, but for some reason has problems with those werid urls. They seem to have the path to the actual file embedded in the url with a LONG code (or mixture of letters) at the end.
Some of the files it has managed to download, end up named as the URL, but when I rename them to .jpg for example, they are the actual picture. However, it doesn`t seem to even do this when I start the download from the first page and there is far to many to input them all manually.
To make matters worse, the site has tried to stop auto downloaders such as OE, so when too many connections start to occur it just redirects to an error page and won`t let you to any of the pictures or videos. To get around this I have tried only using 1 connection and I put a 1 second gap between downloads, this is making the process EXTREMELY slow going as it all it seems to be doing is looking through loads of pages and not download anything from them, then will suddenly stop usually with "Aborted".
I noticed on the site that it says there is a download log to actually log why a file or page wasn`t downloaded, I can`t seem to find this or where to enable it, could you let me know this please. Would be very useful!
They really seem to have spent a long time on making this site offline downloader proof!
I really hope there is a way around it.
I would be very grateful of any help you could offer. I realise this will be difficult without the site details in question, but please suggest anything else I can try.
PS. Sorry for the long explanation and if you need any more info please let me know!
Therefore it`s obviously not attempting the login correctly for some reason!
Here`s the bit of code on the login page for the login box, so I hope this helps...
<form name="loginForm" method="post" action="login.php?ref=%2Fmembers%2Fs.php%3Fshootid%3D2522%26type%3Dvideo">
<input type="text" name="userName" value="" maxlength="32">
<input type="password" name="password" value="" maxlength="32">
<input type="submit" value=" Log in "><br><br>
Thanks for the reply :)
PS. Please could you let me know where the download log is.
You can access the download log by pressing Ctrl-W keys.
Thanks for letting me know about the download log, it seems to be going through pictures and links and after each one it says "Aborted" and then does the next one.... is this a problem?
Thanks for your help quick replies.
HTTP0: Delay 1 seconds before http://site.net/www.site.com/imagedb/2426/i/h/200/27.jpg.
HTTP0: Connecting to host site.net...
HTTP0: Host site.net connected. Waiting for http://site.net/www.site.com/imagedb/2426/i/h/200/27.jpg.
HTTP0: GET /www.site.com/imagedb/2426/i/h/200/27.jpg HTTP/1.0
HTTP0: Accept: */*
HTTP0: Accept-Language: en-gb
HTTP0: Referer: http://www.site.com/members/start.php
HTTP0: User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
HTTP0: Host: site.net
HTTP0: Transferring data from http://site.net/www.site.com/imagedb/2426/i/h/200/27.jpg.
HTTP0: HTTP/1.0 200 OK
HTTP0: Connection: close
HTTP0: Date: Thu, 03 Mar 2005 14:07:36 GMT
HTTP0: Age: 144475
HTTP0: Server: Apache/1.3.27 (Unix) (Red-Hat/Linux) PHP/5.0.3 mod_ssl/2.8.12 OpenSSL/0.9.6b
HTTP0: Accept-Ranges: bytes
HTTP0: Last-Modified: Wed, 22 Dec 2004 20:15:34 GMT
HTTP0: ETag: "2eb412e-10e0-41c9d5e6"
HTTP0: Content-Length: 4320
HTTP0: Content-Type: image/jpeg
HTTP0: Delay 1 seconds before http://site.net/www.site.com/imagedb/2426/i/h/200/29.jpg.
Now it doesn`t seem to be downloading the videos either:
HTTP0: 0 bytes of http://www.site.com/members/movie.php?server=ps1&movie=2468hi.rm.
HTTP0: Download complete. Status: 302 Object Moved.
Keeps coming up with that "Object Moved" error. Now, I believe this means that the video file is actually located on another server, but is there a way for it to say where? I have limited the server configuration, because if I don`t due to the massive amount of links it spends hours upon hours just looking at pages and goes completely off the website I want to download!
Regarding videos. You can adjust Project settings for that - open the Project Properties dialog, go to File Filters | Video and select "Load from any site" in the Location box. Click OK button.
Does this help?