Downloading Pics and Videos from PHP Redirect Site Not Working

Si
03/02/2005 08:01 pm
Hi,
I have a problem downloading a website that is full of PHP (using OE E 3.6). When I go into the videos and pics that I want to download, it does a PHP type redirect to another site I believe where the pics and videos are actually located, without changing the URL.

So for example, when I click on a gallery of pics that I want to download it goes to this URL:

http://www.site.com/members/s.php?shootid=2522&type=pictures&section=01%20Dining%20Room&target=hires&sessionToken=1147074f4b69ccb06a07df08d180cd0f613f449b&ticket=5HGbuxdVLnx/uJipWv5OBiiSFHykQplJwg+9cLkcAimvboCzLXZEV/eU10/+A/zo8hLdxUvM8ROVb5rR8Lm6KQUCkeJxMzlXdvmhq3WFk0j/N1zMiGgdUK8SC3pJj7CRLbLJemg8GnMnPqYxtDU29tCJTetE+2Bs2tGLT/pxhEc=

When I actually click on the JPG picture it goes to this URL:

http://www.site.com/members/g.php?l=site/members/2522/pictures/hires/01%20Dining%20Room/72CG3426_RT8.jpg&sessionToken=1147074f4b69ccb06a07df08d180cd0f613f449b&ticket=qiOSRUXGSkLjTYjspvW7Ur97X9LCPUf0r/6W72aEMWTSpzYd6pSLvHS7vfeq3LDZ5GDi9NU7si686PD+nNTWnOO5xiF8h1FIZyE7rjWznt0HsYfuBkpV6T2njXDiJtwsRzqNYp4Xc4JAq68PFaZ4JiOexuHGAsvXz5j+cHAsBKk=

Sorry I can`t give the site out because it`s a members only site, also the pics and videos are accessed via my username and password (which I have entered in the project).

It comes up with similar style urls when I click on the videos and streaming videos links which it also doesn`t download. For example, page with video files to download off:

http://www.site.com/members/s.php?shootid=2522&target=real&page=0&type=video&section=

When you actually click on the video file the URL becomes very long and werid!

I have tried numerous options in the project, I have levels unticked, tried checking just pictures and videos file types and specified keywords for the servers it goes to get the pictures and videos. However, this didn`t work properly, so I tried ticking all the file type options and all servers, but this just downloaded a lot of junk. Basically, lots of long file names of the url of 1kb. Loads and loads of them! I have also tried starting off the download with the Referer= line to the url also but this didn`t seem to help.

It seems to be able to download links/files from the PHP files, but for some reason has problems with those werid urls. They seem to have the path to the actual file embedded in the url with a LONG code (or mixture of letters) at the end.

Some of the files it has managed to download, end up named as the URL, but when I rename them to .jpg for example, they are the actual picture. However, it doesn`t seem to even do this when I start the download from the first page and there is far to many to input them all manually.

To make matters worse, the site has tried to stop auto downloaders such as OE, so when too many connections start to occur it just redirects to an error page and won`t let you to any of the pictures or videos. To get around this I have tried only using 1 connection and I put a 1 second gap between downloads, this is making the process EXTREMELY slow going as it all it seems to be doing is looking through loads of pages and not download anything from them, then will suddenly stop usually with "Aborted".

I noticed on the site that it says there is a download log to actually log why a file or page wasn`t downloaded, I can`t seem to find this or where to enable it, could you let me know this please. Would be very useful!

They really seem to have spent a long time on making this site offline downloader proof!
I really hope there is a way around it.

I would be very grateful of any help you could offer. I realise this will be difficult without the site details in question, but please suggest anything else I can try.

Thanks

PS. Sorry for the long explanation and if you need any more info please let me know!
Si
03/02/2005 08:19 pm
I have also just noticed that I don`t think it`s logging into the site properly with my username and password. I left it out the project and turned off IE cookies and turned on "prompt for user/pass", it goes through loads of URLs and I see "login.php?..... etc" come up a lot, but never prompts me for a username and password!

Therefore it`s obviously not attempting the login correctly for some reason!

Here`s the bit of code on the login page for the login box, so I hope this helps...

<form name="loginForm" method="post" action="login.php?ref=%2Fmembers%2Fs.php%3Fshootid%3D2522%26type%3Dvideo">
<table>
<TR>
<TD>Username:</TD>
<TD>
<input type="text" name="userName" value="" maxlength="32">
</TD>
</TR>
<TR>
<TD>Password:</TD>
<TD>
<input type="password" name="password" value="" maxlength="32">
</TD>
</TR>
</table>
<input type="submit" value="&nbsp;&nbsp;Log in&nbsp;&nbsp;"><br><br>


Thanks!
Oleg Chernavin
03/03/2005 08:06 am
I would suggest you to log on the site in the Internal browser and then start downloading the site. Please remove the username/password from the Project Properties.

Best regards,
Oleg Chernavin
MP Staff
Si
03/03/2005 08:47 am
How do you mean? By selecting each page in turn and choosing the "download this page" option? That would certainly take forever as there is so many. Plus, if I remove the user/pass it wouldn`t be able to access the files to download them?

Thanks for the reply :)

PS. Please could you let me know where the download log is.
Oleg Chernavin
03/03/2005 09:01 am
No. I mean to logon that site it is much easier to do it in the Internal browser. This way, Offline Explorer uses the logged session to load the Project . This is why username/password in the Project is not necessary.

You can access the download log by pressing Ctrl-W keys.

Oleg.
Si
03/03/2005 09:13 am
Ah I see. Sorry. I have browsed through the site and entered my username and password, then left that in the internal browser and started the project downloading. I hope this works, it seems to be doing more of the same, but I`ll see. I have it set to 1 channel and 1 second delay, so hopefully that will allow it to do stuff without blocking me for too many connections. Is there anyway to get around that?

Thanks for letting me know about the download log, it seems to be going through pictures and links and after each one it says "Aborted" and then does the next one.... is this a problem?

Thanks for your help quick replies.
Si
03/03/2005 09:21 am
Here`s some of the download log... is this right behaviour?

HTTP0: Aborted.
HTTP0: Delay 1 seconds before http://site.net/www.site.com/imagedb/2426/i/h/200/27.jpg.
HTTP0: Connecting to host site.net...
HTTP0: Host site.net connected. Waiting for http://site.net/www.site.com/imagedb/2426/i/h/200/27.jpg.
HTTP0: GET /www.site.com/imagedb/2426/i/h/200/27.jpg HTTP/1.0
HTTP0: Accept: */*
HTTP0: Accept-Language: en-gb
HTTP0: Referer: http://www.site.com/members/start.php
HTTP0: User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
HTTP0: Host: site.net
HTTP0: Transferring data from http://site.net/www.site.com/imagedb/2426/i/h/200/27.jpg.
HTTP0: HTTP/1.0 200 OK
HTTP0: Connection: close
HTTP0: Date: Thu, 03 Mar 2005 14:07:36 GMT
HTTP0: Age: 144475
HTTP0: Server: Apache/1.3.27 (Unix) (Red-Hat/Linux) PHP/5.0.3 mod_ssl/2.8.12 OpenSSL/0.9.6b
HTTP0: Accept-Ranges: bytes
HTTP0: Last-Modified: Wed, 22 Dec 2004 20:15:34 GMT
HTTP0: ETag: "2eb412e-10e0-41c9d5e6"
HTTP0: Content-Length: 4320
HTTP0: Content-Type: image/jpeg
HTTP0: Aborted.
HTTP0: Delay 1 seconds before http://site.net/www.site.com/imagedb/2426/i/h/200/29.jpg.
Oleg Chernavin
03/03/2005 11:10 am
Do you have File Filters | Images section checked in the Project Properties dialog?

Oleg.
Si
03/03/2005 11:17 am
Yes. I worked out what that was in the end, I have it set so that only images over 50kb are downloaded. Thats why it`s skipping them I believe. Maybe the log should say, image rejected "Minimum Image size" or something.

Now it doesn`t seem to be downloading the videos either:

HTTP0: 0 bytes of http://www.site.com/members/movie.php?server=ps1&movie=2468hi.rm.
HTTP0: Download complete. Status: 302 Object Moved.

Keeps coming up with that "Object Moved" error. Now, I believe this means that the video file is actually located on another server, but is there a way for it to say where? I have limited the server configuration, because if I don`t due to the massive amount of links it spends hours upon hours just looking at pages and goes completely off the website I want to download!

Thanks!
Oleg Chernavin
03/03/2005 11:36 am
This is a good idea about more information. I just implemented it. Thank you!

Regarding videos. You can adjust Project settings for that - open the Project Properties dialog, go to File Filters | Video and select "Load from any site" in the Location box. Click OK button.

Does this help?

Oleg.