I'm getting trouble with a site which use *session*

Author Message
Yudi 10/29/2006 09:23 am
Hi, recently I was downloading from imagegarden.net but those site use some *session-id* technique, so I want to ask, how can I get a different session-id for every download connection? Is it possible or not in OE? The image source isn't an image-type but rather a script (PHP-script). I'm using URL-Macros and it's succeeded for some link except that it cannot download an image which have a file-size above 710 KB (it keeps get a "status: 200 OK" error-message). A file which have file-size above those limit (710 KB) will get downloaded at first, but then it stops when the download has reach 75% - 87%. I don't know why...

And one more question, what's the meaning of the "status: 200 OK" error-message anyway?

Thanks!
Oleg Chernavin 10/30/2006 06:51 am
Can you try to open one image in the Internal browser of OE first, then copy the URL with the session ID, add URL Macro to load all images and download it. OE will use the same cookies as the internal browser (embedded MS IE).

Best regards,
Oleg Chernavin
MP Staff
Yudi 10/30/2006 09:22 am
> Can you try to open one image in the Internal browser of OE first, then copy the URL with the session ID, add URL Macro to load all images and download it. OE will use the same cookies as the internal browser (embedded MS IE).
>
> Best regards,
> Oleg Chernavin
> MP Staff

I've try that way before but still no clue. Here's the settings I use:
------------------------------------------------------------------------------------------------------------------------
[Object]
OEVersion=Enterprise 4.4.0.2478
Type=0
IID=7552
Caption=http://www.imagegarden.net/image.php?cataid=4&albumid=2795&type=jpeg&imageid={:1..64}
URL=http://www.imagegarden.net/image.php?cataid=4&albumid=2795&type=jpeg&imageid={:1..64}additional=DepthFirst
Weekday=257
User=chocobollz
Psw=aozora
LimTSize=10000
LimNumber=5000
LimTime=100
FMGroup=2
CheckSize=True
SkipMedia=True
pswMethod=2
FTText.Exts=htmlhtmaspaspxjspstmstmlidcshtmlhtxtxttextxspxmlrxmlcfmwmlphpphp3
FTImages.Exts=gifjpgjpegtiftiffxbmfifbmppngipxjp2j2cj2kwbmplwf
FTVideo.Exts=mpgavianimpegmovflvfliflcvivrmramrvasfasxwmvm1vm2vvob
FTAudio.Exts=wavriffmp3midmp2m3uravocwmaape
FTArchive.Exts=ziparcgzzarjlhalayleirarcabtarpakacejarpdftgz
FTUDef.Exts=jscssssivbsdtdxslswfclass
FTText.B=xoxooo
FTImages.B=ooxxoo
FTVideo.B=xoxooo
FTAudio.B=xoxooo
FTArchive.B=xoxooo
FTUDef.B=xoxooo
FTOther.B=xoxooo
FTSizes=0,0,0,10,0,0,0,0,0,0,0,0,0,0,0,3,3,3,0,3,0
RPathEx=showgallerythumbmediumtinhxuanavatarsmilie xxxxxx
RFileEx=showgallery.phpmember.phpindex.phpforumdisplay.phpshowthread.php xxxxx
RProt=127
LastStart=117:220:141:76:99:13:227:64:
LastEnd=172:64:203:142:99:13:227:64:
S200=127
SErr=1
SAbr=14
SPar=64
SSav=63
SLast=200
SSiz=14728141
SMdf=63
LFiles=127
LSize=15867906
NoFileCheck=True
Flags=1
ImgDim=0,0,0,0
PrevURL=http://www.imagegarden.net/image.php?cataid=4&albumid=2795&type=jpeg&imageid=1
NoIgnoreErrors=True
IPAddr=1613391306
------------------------------------------------------------------------------------------------------------------------

And by the way, after doing some more research, I guess the problems is unlikely caused by OE but because of an error on the site itself. I've found that even when I try to open the page in IE, the image's still don't get fully-loaded, so I've come into conclusion that the problem might be caused because of there was some form of *file-size limiter* on the site OR the admin itself had been block my IP-Address so I'll never get the full images =)

So, now I'm getting some workaround for this matter, here's what I want to ask (if you're willing ^^ ):
- I want to know if OE can save a *incomplete* files? For example, when an images is corrupted and the download was halted in the middle. Can I get those *incompleted* file or not?
- Second is, can you please browse the link I gave you and give me your thoughts about the link, and of course, any feedback on how to download it ^^.

I'll be very grateful if you're willing to do that ^^. In any case, thanks for the reply. Never been found anyone as kindly as you, Oleg. And one more thing, sorry for my bad English because I don't get too used to it =)
Yudi 10/30/2006 09:46 am
And now I've found something weird! I've trying to use another browser to browse the site in a normal way (I use Firefox 1.5.0.6) and get to the exact same pic-page. And guess what, the pic is fully loaded! The image-link in the 2 browser is:

IE: http://www.imagegarden.net/image_c.php?imagesession=dcede1e6233552fe195fe6614006c136&cataid=4&albumid=2896&imageid=9&type=jpeg
FFox: http://www.imagegarden.net/image_c.php?imagesession=17edeb2de254c2b75a7b49a405b679ee&cataid=4&albumid=2896&imageid=9&type=jpeg

Then I try to load another pic in Firefox (which is cannot be downloaded by OE before), and the result is, IT'S FULLY LOADED too! But I *did* notice that there are some lag to the download speed when the image is getting 80 - 90% loaded (it's loaded very fast at first, but when the image showed for about 80 to 90%, there are significant drop to the download speed). This is getting me even more confused =) Is the problem is caused by OE, IE, or the site itself? It's normally doesn't too hard to download some files from a website like this, doesn't it should be so?

And now i'm thinking about changing the [Agent Identification -> Identify As] option in [Options -> Internet -> Connection] option-page to something like "Firefox" but obviously I can't =P)
Yudi 10/30/2006 10:14 am
More info ^^. I've tested another method to download the site. Here's what I think: "IF Firefox IS ABLE to download the image, why don't we get OE identify itself as Firefox so the site will think that it's Firefox who open the page". So, I get the work up. I change the [Agent Identification] option to use the "Use this identification" and then entered "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.6) Gecko/20060728 Firefox/1.5.0.6" in the edit-box next to it (got this value from the [Help -> About Mozilla Firefox] in Firefox toolbar) and hope that OE will be successfully *recognized* as Firefox and guess what, it even take the Firefox session-id! (never thought that) Then I do one more thing to make sure that the site will recognize OE as Firefox: limit the connection to 1. Lastly, I clicked the download button and wait for the miracle to happen.

Several minutes has passed...

Guess what happen? It is still CANNOT download the image... T_T. How can it be... and now I get another error-message in the connection-panel, it is "Reget is not supported". Is there any probability that the site uses some mass-downloader protection by "intentionally breaking up the connection and auto-resume it again" so that a mass-downloader couldn't resuming (also with IE) and the Firefox is the only one who can resume the download? Don't know if it true...

Could you please help me Oleg? Please, give me your advice. Thanks.

Best regards.
Yudi 10/30/2006 10:35 am
Several more questions =)

- Can we get a different session-id for a different connection and append it automatically in the URL like those in the URL-Macros?

- And I'm curious, when I want to get a different session-id ([imagesession] parameter) value, it won't change even that I've already cleared all the caches, cookies, etc. (I'm using Firefox's [Clear Private Data] function to do this), and I'm even closing and re-opening Firefox, but the "image.php?imagesession=" parameter still won't change. I wonder why is this happened? Or is it me which mis-interpreted the "imagesession" parameter, thinking it's the same as "session-id"? I think there's probability that the site administrator does not use the PHP's/browser session-id as the [imagesession] parameter's value but rather choose a different method to obtain the value.

That's it for today, want to get sleep now =) Hope you reply my question above, though it's likely hella long to read (hope you're not too lazy to read it all =) Thanks.

Best regards.
Oleg Chernavin 10/31/2006 08:12 am
> - I want to know if OE can save a *incomplete* files? For example, when an images is corrupted and the download was halted in the middle. Can I get those *incompleted* file or not?

No, Offline Explorer was designed to save only fully-downloaded files.

> - Can we get a different session-id for a different connection and append it automatically in the URL like those in the URL-Macros?

Yes, you can use the Cookie= parameter in the URLs field of the Project to clear any cookie that was assigned before.

One idea about Offline Explorer is that probably you have too low Timeout setting in the Options dialog - set it to 600 seconds to see if it will be able to wait long enough to get the file.

Oleg.
Yudi 10/31/2006 09:04 am
> One idea about Offline Explorer is that probably you have too low Timeout setting in the Options
> dialog - set it to 600 seconds to see if it will be able to wait long enough to get the file.

I'm afraid this isn't the answer because I've already try that way (I'm setting up the timeout to 5 minutes (300 secs) and didn't work too). From my analysis, the download progress on each connection is actually works well, it fast at first but then it stopped at 70 - 90% just like when somebody is aborting the connection. The timeout setting will have nothing to do with this kind of case, didn't it?

> Yes, you can use the Cookie= parameter in the URLs field of the Project to clear any cookie that was
> assigned before.

And for this one, I guess it won't work, but I'll try it up anyway, just to make sure :) Why won't I believe this will work? Because as I already mentioned on my writing above, in the section like:

"And I'm curious, when I want to get a different session-id ([imagesession] parameter) value, it
don't change even that I've already cleared all the caches, cookies, etc. (I'm using Firefox's
[Clear Private Data] function to do this), and I'm even closing and re-opening Firefox, but
the "image.php?imagesession=" parameter still won't change. I wonder why is this happened?"

See? :) But I'll test it right away and by the way, thanks alot though your answer still not suffice to answer all of my questions :)
Oleg Chernavin 10/31/2006 09:14 am
Maybe the session ID depends on the browser and your IP address and the server assigns it to you?

Oleg.
Yudi 10/31/2006 09:22 am
> > - I want to know if OE can save a *incomplete* files? For example, when an images is corrupted and the download was halted in the middle. Can I get those *incompleted* file or not?
>
> No, Offline Explorer was designed to save only fully-downloaded files.
>
> > - Can we get a different session-id for a different connection and append it automatically in the URL like those in the URL-Macros?
>
> Yes, you can use the Cookie= parameter in the URLs field of the Project to clear any cookie that was assigned before.
>
> One idea about Offline Explorer is that probably you have too low Timeout setting in the Options dialog - set it to 600 seconds to see if it will be able to wait long enough to get the file.
>
> Oleg.

And about the "Cookie=" parameter, how can we clear the cookie? Is it like add this command: "http://www.imagegarden.net/image_c.php?imagesession=;" to the URLs? Or is it by another way?

And I don't know why is it works now! I'm trying to download just 1 file and it works! But then when I try it again the same way, it won't works again. Now where we go? :)
Oleg Chernavin 10/31/2006 09:33 am
Just try:

Cookie=a

This will not transfer the session ID information (if it is in the cookie).

Oleg.
Yudi 11/01/2006 03:46 am
> Maybe the session ID depends on the browser and your IP address and the server assigns it to you?
>
> Oleg.

It might be so, as I've already clearing up the cookies (and etc.) and closing and reopening the browser but it still have the same exact imagession ID, so your thoughts might be correct. I'm doing another research now, 2 more methods to try to.

Oleg, could you please opening the link and try to download the images please? But that if you have free-time. I'm waiting for your replies and by the way, thanks for the advices.
Yudi 11/01/2006 04:04 am
> Just try:
>
> Cookie=a
>
> This will not transfer the session ID information (if it is in the cookie).
>
> Oleg.

By the way, I'm not quite understand what you're (and the OE) refering as *cookies*. Is the cookies here is same as with a *parameter* (a PHP's variable passed on URL; or a POST or GET data)? Like I've read from the OE's Help-files, the explanations there is like "Cookie=param1=value&param2=othervalue"? So I've thoughts that the *cookies* mentioned in those help-files were the POST or GET data. And by the way, how can we know that *this* cookie is belongs to *this* website? Is it have to be viewed by some form of *cookie-viewer* software? Or you've just guess them up? :)

And also, I think MetaProducts have an excellent user-forum support (with you around, Oleg :), but unfortunately, the Help-files lacks of many *might-be-useful* informations. The help-files is of course have the most important informations in it, but I guess it lacks something else, like any command which likely hard to understand by the user should be equipped by an example. I know this is a bit abundant, but it's better than letting all the user to ask you 24-hours/7-days a week, right Oleg? But even so, I think this user-forum is really had been moderated by a very capable person like you :) You should ask for increment in salary :)

Oleg Chernavin 11/01/2006 07:38 am
I get the same result and I don't think this is because of cookies. It stops at around 70% because of the server request. Sorry, I don't have any other idea now.

Oleg.
Yudi 11/03/2006 12:54 pm
> I get the same result and I don't think this is because of cookies. It stops at around 70% because of the server request. Sorry, I don't have any other idea now.
>
> Oleg.

LOL, so you too? Anyway, thanks for your assist in this matter. I'll try some other workaround on this (currently I have 4 more methods to be tested :), so don't worry :)
Yudi 11/03/2006 01:42 pm
> I get the same result and I don't think this is because of cookies. It stops at around 70% because of the server request. Sorry, I don't have any other idea now.
>
> Oleg.

And by the way, what's error-message do you get? Is it "Status: 200 OK" or "Reget is not supported"? "Status: 200 OK" means that the server is telling us that the download is complete isn't it? (despite it isn't)

And also, how can I get the URL-Macros function to enumerate like this example:
(link1) http://www.a.com/b.php?from=white paper
(link2) http://www.a.com/b.php?from=gift
(link3) http://www.a.com/b.php?from=tea
(link4) http://www.a.com/b.php?from=golf
(link5) http://www.a.com/b.php?from=white paper
(link6) http://www.a.com/b.php?from=gift
(link7) http://www.a.com/b.php?from=tea
(link8) http://www.a.com/b.php?from=golf
(link9) http://www.a.com/b.php?from=white paper
... and so on

How can I achieve a result like the above? I've trying to use this command
"http://www.a.com/image.php?from={:white paper,gift,tea,golf}&imageid={:1..40}", but I can't get the result like above, the fact that, the result from the command is like this:
(link1) http://www.a.com/b.php?from=white paper .. (link10) http://www.a.com/b.php?from=white paper
(link11) http://www.a.com/b.php?from=gift .. (link20) http://www.a.com/b.php?from=gift
(link21) http://www.a.com/b.php?from=tea .. (link30) http://www.a.com/b.php?from=tea
(link31) http://www.a.com/b.php?from=golf .. (link40) http://www.a.com/b.php?from=golf

As you can see, that's not exactly I want it to be. In fact, I want to be reiterate on a 1-link basis (don't know the correct words in English to describe that but you should be understand from the examples above :)

Oleg Chernavin 11/06/2006 04:12 am
I get Status 200 first, then OE tries to reconnect to get the rest of the file, reget fails.

I understand the problem with the macro, but this is how Offline Explorer generates the links. Maybe only you would try this:

http://www.a.com/image.php?imageid={:1..40}&from={:white paper,gift,tea,golf}

Perhaps, the server would allow such URL.

Oleg.
Yudi 11/06/2006 09:08 am
> I get Status 200 first, then OE tries to reconnect to get the rest of the file, reget fails.
>
> I understand the problem with the macro, but this is how Offline Explorer generates the links. Maybe only you would try this:
>
> http://www.a.com/image.php?imageid={:1..40}&from={:white paper,gift,tea,golf}
>
> Perhaps, the server would allow such URL.
>
> Oleg.

Oh so the order is important? If it is, that's explain why did I get such a results! :) Yep, thanks again for the reply Oleg. I'll try this right away (though I'm not really sure it'll solve the problems :)