URL corrupted?

Author Message
Jason 09/12/2006 09:18 pm
Hi,

I am trying out Offline Explorer Pro 4.4 and am having difficulties downloading from a secure (https) site. It will download the start page I specify, however every link downloaded is a login page.

I have tried specifying the login information under the Project setup, and I have also logged in using the built in browser but still no luck.

One thing I noticed is that whenever I apply changes to the project, the URL is corrupted. The start URL I want is:

https://www.somedomain.com/dir1/dir2i/?l=3&id=100997&bt=2

however the URL gets changed in the project (and browser when I try to browse) to:

https://www.somedomain.com/dir1/dir2i/@l=3&id=100997&bt=2

The "?" gets changed to an "@"

Any ideas?
Jason 09/12/2006 09:20 pm
Also, in the URL I notice it is multi colored. The "?" is the start of purple so I don;t know if there is a MACRO or some kind of substitution happening.


> Hi,
>
> I am trying out Offline Explorer Pro 4.4 and am having difficulties downloading from a secure (https) site. It will download the start page I specify, however every link downloaded is a login page.
>
> I have tried specifying the login information under the Project setup, and I have also logged in using the built in browser but still no luck.
>
> One thing I noticed is that whenever I apply changes to the project, the URL is corrupted. The start URL I want is:
>
> https://www.somedomain.com/dir1/dir2i/?l=3&id=100997&bt=2
>
> however the URL gets changed in the project (and browser when I try to browse) to:
>
> https://www.somedomain.com/dir1/dir2i/@l=3&id=100997&bt=2
>
> The "?" gets changed to an "@"
>
> Any ideas?
Oleg Chernavin 09/13/2006 03:26 am
There are two kinds of password-protected Web sites. One type asks for a username and password in a standard Windows-type dialog (BASIC and NTLM authentication,) while the other type requires you to logon directly on a Web page.

1. Web sites that require BASIC and NTLM-authentication.

To download this type of site, either specify the username and password in the Project Properties dialog | Advanced | Password section, or type them directly in the URL, such as:
http://username:password@www.server.com/...

Notice that a colon separates the password from the username and the @ symbol separates them from the server name.

Some sites use NTLM authentication, which looks like the above, but with a third box labeled "DOMAIN" along with the username and password. Enter the domain name along with the username in the same field in the Project Properties dialog | Passwords this way:

DOMAIN\username

The backslash symbol separates the domain name from the username. When you are done with the Project Properties changes, click the OK button to save them.

Note: NTLM authentication is supported only in the Pro and Enterprise editions of Offline Explorer.


2. Authentication in a Web form.

It is easy to download this type of site.
You need to browse to the logon page of the site using the internal Browser of Offline Explorer Pro .

If you need to download the site immediately and only once, you can proceed with the logon and begin downloading the desired pages using Offline Explorer Pro . The program will use the session cookies of the logged on site from the internal browser.

You can also record the logon form contents in a Project, so that Offline Explorer Pro will know how to log itself on whenever you wish to download the site. This is useful when you want to schedule the site download or perform it later, or if you want to update the downloaded site in the future.

Once you have entered your username and password on the logon page in the internal browser, press and hold the Alt + Ctrl keys on your keyboard, click the Logon (or Submit) button in the Web form and release the keyboard buttons. You should get a new Project that contains the Web form information recorded in the URL field.

Adjust the Project settings as you wish (set the Level and other parameters) and click the OK button to save the Project. You may begin downloading at any time.

Note: The form recording method is supported only in the Pro and Enterprise editions of Offline Explorer.

Regarding the ? to @ symbols change - it is normal, because ? symbols are not allowed in filenames.

Best regards,
Oleg Chernavin
MP Staff
Jason 09/14/2006 01:51 am
Thanks for the info on the filename. I also found the real problem. There were several logout links and I had to explicitly excluded each one for it to work. The IgnoreLogoutLinks didn't.

> There are two kinds of password-protected Web sites. One type asks for a username and password in a standard Windows-type dialog (BASIC and NTLM authentication,) while the other type requires you to logon directly on a Web page.
>
> 1. Web sites that require BASIC and NTLM-authentication.
>
> To download this type of site, either specify the username and password in the Project Properties dialog | Advanced | Password section, or type them directly in the URL, such as:
> http://username:password@www.server.com/...
>
> Notice that a colon separates the password from the username and the @ symbol separates them from the server name.
>
> Some sites use NTLM authentication, which looks like the above, but with a third box labeled "DOMAIN" along with the username and password. Enter the domain name along with the username in the same field in the Project Properties dialog | Passwords this way:
>
> DOMAIN\username
>
> The backslash symbol separates the domain name from the username. When you are done with the Project Properties changes, click the OK button to save them.
>
> Note: NTLM authentication is supported only in the Pro and Enterprise editions of Offline Explorer.
>
>
> 2. Authentication in a Web form.
>
> It is easy to download this type of site.
> You need to browse to the logon page of the site using the internal Browser of Offline Explorer Pro .
>
> If you need to download the site immediately and only once, you can proceed with the logon and begin downloading the desired pages using Offline Explorer Pro . The program will use the session cookies of the logged on site from the internal browser.
>
> You can also record the logon form contents in a Project, so that Offline Explorer Pro will know how to log itself on whenever you wish to download the site. This is useful when you want to schedule the site download or perform it later, or if you want to update the downloaded site in the future.
>
> Once you have entered your username and password on the logon page in the internal browser, press and hold the Alt + Ctrl keys on your keyboard, click the Logon (or Submit) button in the Web form and release the keyboard buttons. You should get a new Project that contains the Web form information recorded in the URL field.
>
> Adjust the Project settings as you wish (set the Level and other parameters) and click the OK button to save the Project. You may begin downloading at any time.
>
> Note: The form recording method is supported only in the Pro and Enterprise editions of Offline Explorer.
>
> Regarding the ? to @ symbols change - it is normal, because ? symbols are not allowed in filenames.
>
> Best regards,
> Oleg Chernavin
> MP Staff
Oleg Chernavin 09/14/2006 02:17 am
Can you please post such logout links here, so I could improve the IgnoreLogoutLinks logic?

Thank you!

Oleg.