How force password to avoid "not logged in" shadow page

  • User Forums
  • How force password to avoid "not logged in" shadow page
ToolmakerSteve
12/21/2004 07:47 pm
Site: www.boxedart.com
Goal: download some .zip files.
Problem: Each download link is "shadowed" by a page which says "Your session has expired or you have not logged in.". That is, I`ve been unable to get OE to retrieve the "real" version of any page.
Things I`ve already verified:
* I can download each file in browser (IE) - requires clicking on a form button, THEN clicking on a text link;
* Options / "Use MSIE Cookies" IS checked;
* Downloaded a trial of IE PRO so I could add a "referer" line.
* I`ve successfully downloaded from other membership-protected sites.
* Tried on two different computers.
* Tried deleting files & starting over.
* Password is set.

* I even tried Passwords/"Prompt for user name and password on protected sites".
UNFORTUNATELY, OE NEVER ASKS for a password.
Reason: because there is a non-password-protected page that gets served instead.

Example:
http://www.boxedart.com/phpshop/members.php/info/mp_/15975
Referer=http://www.boxedart.com/
Level limit: 1

Look at the downloaded page (that was referenced from the above page,
via the "Member Download" button):

<download dir>/www.boxedart.com/phpshop/mem_download.php/down/mp_/15975

If successful, it is an HTML file that contains a text link "Download Strange New Worlds HTML directly here", which is href:
http://www.boxedart.com/phpshop/download.php?id=15975

If fails to provide password, it is an HTML file that says "Your session has expired or you have not logged in", and has a form for logging in.

From OE, I have been unable to get anything but the "expired" form.

I also tried, starting down one level (deleting the "expired" file first, to make sure I re-downloaded it):
http://www.boxedart.com/phpshop/members.php/info/mp_/15975
Referer=http://www.boxedart.com/
Level limit: 1

Nope, "download" of that URL results in the "expired` form again.

QUESTION: HOW FORCE correct use of session, cookie, password, so that the website won`t serve that "expired" form, but instead will do what browser does when a properly logged-in member accesses this field?

NOTE: this website doesn`t provide any place to log-in, until you actually try to download something, so it might be necessary to have a way to tell OE to fill in that login form on the first download -- apparently no "cookie" is created until that is done.

ToolmakerSteve
12/21/2004 08:40 pm
* I also tried Passwords/"Use this user name and password"/"in HTML forms".
Didn`t help; either with or without "referer" field.

NOTE: examining the HTML, I see that the fields that need filling in are described as:
input type="text" name="user"
input type="password" name="pass"

and the submit button is:
type="submit" name="downbutton" value="Download"

there also are a number of hidden fields, including one with a long hex string in it:
type="hidden" name="kid" value="ec2d78b1e36ad6c"

(That hidden value may be related to establishing a session id,
as it changed when I re-downloaded the exact same page, on a different computer.)
Oleg Chernavin
12/22/2004 03:55 am
Did you try to open the site online in the Internal browser first and logon there, then keep it logged on and start downloading your Project? In this case, no password should be entered in the Project settings.

Best regards,
Oleg Chernavin
MP Staff
ToolmakerSteve
12/22/2004 12:36 pm
OH, I have to login using the INTERNAL browser that`s in OFFLINE EXPLORER!!

Yes, that worked.

I had entered my username and password in a SEPARATE Internet Explorer window,
which didn`t do it.

It would be good to put a note in help for passwords to carefully explain this:
I`m so used to thinking of IE as my browser, that it didn`t sink in that I could (& should) use OE`s browser pane to do this.

Thank you.
Oleg Chernavin
12/23/2004 05:53 am
Yes, sorry for the confusion. The Internal OE`s browser shares cookies with the rest Offline Explorer, so after logging on the site, OE is able to download the pages.

Oleg.
Joma
05/13/2005 05:01 am
> Yes, sorry for the confusion. The Internal OE`s browser shares cookies with the rest Offline Explorer, so after logging on the site, OE is able to download the pages.
>
> Oleg.

Just wondering if you managed to download any content, i just download OE and am looking over it now. Can i get any advice on how to use it to download zip from from the above mentioned site ????
Oleg Chernavin
05/13/2005 06:40 am
Well, I think, this page contains all the information you need to download the site.

If it fails for you, let me know.

Oleg.
Joma
05/13/2005 06:10 pm
Being still a newbee at OE, it would be great. I am still trying to figure out all the controls and stuff, and what to actually do. SO anything you can tell me would be GREAT !!!
Oleg Chernavin
05/14/2005 06:38 am
Did you try the suggestions above? If yes, what exactly doesn`t work?

Oleg.
Joma
05/15/2005 10:24 pm
well to be quit honest, i dont know where to start. I only just started looking at OE so i got no idea what to do first, can you help me with this ?
Oleg Chernavin
05/23/2005 07:31 am
Please try the following - the right part of the main Offline Explorer window contains an Internal Browser. You should be able to browse to that site logon page in it. Then type your username and password there and logon. Once you get to the page you want to download with linked files, please copy its address (above the browser) and click the Paste button to create a new Project. Then click the Download button to start loading the Project.

Oleg.
Joma
05/23/2005 11:28 pm
I dont have a username and password. I have to buy those...i thought we could get the files without having to have a membership.
Oleg Chernavin
05/24/2005 03:52 am
I have no comments. :-)

Oleg.
Joma
05/24/2005 06:36 pm
no comments are no good to me, LOL, guess i will have to try and find another solution
Oleg Chernavin
05/25/2005 07:16 am
I understand. But you should know that trying to download a password-protected site without having the password means "hacking". Offline Explorer is not a tool to hack Web sites, so I doubt that it will help you.

Oleg.
sharia
08/27/2005 06:01 pm

suppose from this link I want to download a zip file. But it wants a username and password.

http://www.boxedart.com/phpshop/mem_download.php/down/mp_/16242


My question Is could you please inform me the stapes how I can move ahead to get a successful download zip file.

i`ll be happy if you provide thsi information. waiting for your reply

sharia






> Site: www.boxedart.com
> Goal: download some .zip files.
> Problem: Each download link is "shadowed" by a page which says "Your session has expired or you have not logged in.". That is, I`ve been unable to get OE to retrieve the "real" version of any page.
> Things I`ve already verified:
> * I can download each file in browser (IE) - requires clicking on a form button, THEN clicking on a text link;
> * Options / "Use MSIE Cookies" IS checked;
> * Downloaded a trial of IE PRO so I could add a "referer" line.
> * I`ve successfully downloaded from other membership-protected sites.
> * Tried on two different computers.
> * Tried deleting files & starting over.
> * Password is set.
>
> * I even tried Passwords/"Prompt for user name and password on protected sites".
> UNFORTUNATELY, OE NEVER ASKS for a password.
> Reason: because there is a non-password-protected page that gets served instead.
>
> Example:
> http://www.boxedart.com/phpshop/members.php/info/mp_/15975
> Referer=http://www.boxedart.com/
> Level limit: 1
>
> Look at the downloaded page (that was referenced from the above page,
> via the "Member Download" button):
>
> <download dir>/www.boxedart.com/phpshop/mem_download.php/down/mp_/15975
>
> If successful, it is an HTML file that contains a text link "Download Strange New Worlds HTML directly here", which is href:
> http://www.boxedart.com/phpshop/download.php?id=15975
>
> If fails to provide password, it is an HTML file that says "Your session has expired or you have not logged in", and has a form for logging in.
>
> From OE, I have been unable to get anything but the "expired" form.
>
> I also tried, starting down one level (deleting the "expired" file first, to make sure I re-downloaded it):
> http://www.boxedart.com/phpshop/members.php/info/mp_/15975
> Referer=http://www.boxedart.com/
> Level limit: 1
>
> Nope, "download" of that URL results in the "expired` form again.
>
> QUESTION: HOW FORCE correct use of session, cookie, password, so that the website won`t serve that "expired" form, but instead will do what browser does when a properly logged-in member accesses this field?
>
> NOTE: this website doesn`t provide any place to log-in, until you actually try to download something, so it might be necessary to have a way to tell OE to fill in that login form on the first download -- apparently no "cookie" is created until that is done.
>
>
Oleg Chernavin
08/31/2005 11:58 am
Can you please see the answers above about logging on the site using the Internal Browser of Offline Explorer first? This trick really helps.

Oleg.
djere
09/27/2005 06:08 am
What about web sites that don`t use cookies. That is, a web site that you must login to but when you browse outside the member area and then back, you have to login again. How do I get OE to handle these sites?

Thank You
Oleg Chernavin
09/27/2005 06:37 am
I think, you should create two Projects - one to download the public area and another - to download secured area only.

Oleg.
Shmulik
10/01/2005 12:13 pm
ToolmakerSteve, please contact me:
icq: 45863986
yahoo: booxa85
AIM: booxail


thanks
Pain
05/11/2006 03:43 am
> Did you try to open the site online in the Internal browser first and logon there, then keep it logged on and start downloading your Project? In this case, no password should be entered in the Project settings.

Hi!
This looks really exciting. I could indeed login on password protected web site via form in internal browser window. I got to see the password protected content, however, when starting to download from first link after login downloader was kicked out to login page again and thus nothing worked. Is there any setting which might help. Sure those people making web-sites aren’t stupid either, but would it be possible to escape to be detected to start downloading? As I said internal browser does work, but when I go to start project download, web-site detects that something is wrong.

The address is http://www. t e x t b o o k o f p a i n . c o m

Any ideas?
Oleg Chernavin
05/11/2006 03:54 am
Please try to add the following line to the Project's URLs field:

IgnoreLogoutLinks

Then logon the site in the Internal browser and start downloading it again.

Best regards,
Oleg Chernavin
MP Staff
Jay
06/02/2006 03:27 pm
So do you need to login first in order to then access the area through OE?
Oleg Chernavin
06/03/2006 04:37 am
Yes, you need to logon the site in the Internal browser of Offline Explorer. This will make all cookies of the logged session available for Offline Explorer downloads.

Oleg.
Seven
06/29/2006 09:38 pm
> Please try to add the following line to the Project's URLs field:
>
> IgnoreLogoutLinks
>
> Then logon the site in the Internal browser and start downloading it again.
>
> Best regards,
> Oleg Chernavin
> MP Staff

I have read this whole page because I am having this particular problem. I go to the page in OE's browser and I put in my un and pass and I can see that it's logged in but when I download the pages it always downloads the login page again... I have a list of direct urls to the exact pages I need to download and I'm putting those in the the project url fields and then I am going in the site and logging on and downloading from there but it does not work no matter what I do.. I have tried the password thing where you enter the un and pass to use in an html form.. and still the same thing I download like 5000 copies of the login page... I have at this point started to use only 3 at a time after downloading nothing a few times .. but I am extremely frustrated as we purchased this and textpipe for a very important project for my work and I just can't get it to work for me... simply because I can't get past this login issue and I don't know why... can someone please help me??
Oh and I tried the above mentioned IgnoreLogoutLinks ..... I put it in with the urls and it still made no difference... PLEASE HELP!!!!!
Oleg Chernavin
06/30/2006 08:20 am
Can you try the following trick:

You can record the logon form contents in a Project, so that Offline Explorer Pro will know how to log itself on whenever you wish to download the site. This is useful when you want to schedule the site download or perform it later, or if you want to update the downloaded site in the future.

Once you have entered your username and password on the logon page in the internal browser, press and hold the Alt + Ctrl keys on your keyboard, click the Logon (or Submit) button in the Web form and release the keyboard buttons. You should get a new Project that contains the Web form information recorded in the URL field.

Adjust the Project settings as you wish (set the Level and other parameters) and click the OK button to save the Project. You may begin downloading at any time.

Note: The form recording method is supported only in the Pro and Enterprise editions of Offline Explorer.

Oleg.
Seven
06/30/2006 02:28 pm
> Can you try the following trick:
>
> You can record the logon form contents in a Project, so that Offline Explorer Pro will know how to log itself on whenever you wish to download the site. This is useful when you want to schedule the site download or perform it later, or if you want to update the downloaded site in the future.
>
> Once you have entered your username and password on the logon page in the internal browser, press and hold the Alt + Ctrl keys on your keyboard, click the Logon (or Submit) button in the Web form and release the keyboard buttons. You should get a new Project that contains the Web form information recorded in the URL field.
>
> Adjust the Project settings as you wish (set the Level and other parameters) and click the OK button to save the Project. You may begin downloading at any time.
>
> Note: The form recording method is supported only in the Pro and Enterprise editions of Offline Explorer.
>
> Oleg.

Ok... I tried this and it did what you said it would do as far as creating a new project.. when I held alt and ctrl down while I clicked login (with the un and pass entered) it saved it into a new project.. but I immediately noticed that what it saved it as is the page that I keep getting in place of what I should be which is the access to the other side of that login... it saved in my new project the url that looks like this
www.whateverpage.com/login.aspx?ReturnUrl=%2ffind_business.aspx .... this is what tends to happen when I try to download direct urls that are located behind the login... I will get www.whateverpage.com/login.aspx?ReturnUrl=(then whatever the url is that I was trying to get)
So I'm still stuck in the same exact situation and I cannot understand why I would have so much trouble doing something that is supposed to be simple enough to do ... And I am using Offline explorer Pro...
This is driving me crazy! I feel bad because I downloaded the evaluation software and was able to do enough on this and the textpipe program that I told my boss that it would work perfect for what we were needing it for and so he purchased both of these programs and now we are going on a week without me being able to produce any result due to the fact that I can't download this data!! I'm started to feel a little worried that he is going to think I'm not doing my job!! I didn't pull data from this particular site when I was evaluating it.. I pulled from a site that did not have a un and pass issue but I did check and see that it said I could download pages like that so I figured it would be perfect for the things we were needing it for..
If there is anything else at all you can think of that I may be doing wrong please let me know.. I cannot for the life of me figure out what the issue is!
I did try to download again any of the direct urls behind that login page and what I got was the same thing I have been getting this whole time
Thanks for any help you may can provide on this issue.
Tori
Oleg Chernavin
07/03/2006 08:55 am
I think, we will be able to find the solution for you. Please write us to support@metaproducts.com and provide with the details about your download - site URL, what you need to download and logon information. I will look at it and provide you with the Project settings necessary to download it.

Oleg.