How to fast change proxy in OE with an available proxy list?

 
07/01/2004 04:22 pm
How to fast change proxy in OE with an available proxy list?

Can I change by editing the registry? Where`s the hive? Plizz help...

Thankz.
Oleg Chernavin
07/02/2004 09:19 am
You can change Registry:

HKEY_CURRENT_USER\Software\MetaProducts\Offline Explorer MX\Parameters
ProxyAddr, ProxyPort and ProxyType values (0 - no proxy, 1 - HTTP proxy).

However you have to quit Offline Explorer before making the Registry change, so this is probably not what yo need. I can offer you another way - to add a command-line parameter to change proxy settings. Each time you run a second instance of Offline Explorer, it will pass its command-line paremeters to the first instance of the program and quit.

If this works for you, just let me know.

Thank you!

Best regards,
Oleg Chernavin
MP Staff
Stefan
07/01/2005 12:27 pm
It would be nice to have a way to actually be able to set a proxy from the command line to make it possible to run different instances of OE with different proxy setting. Another way would be the option to use different global setting files, but I don`t know if that is possible right now.

Stefan
Oleg Chernavin
07/04/2005 08:05 am
What about using Proxy={:file=c:\proxyfile.txt} in the URLs field and changing that proxy line when necessary? If this would work for you, I will improve that URL Macro so it will work with the Proxy= parameter.

Oleg.
Stefan
07/04/2005 08:12 am
Would that OE allow to rotate through a list of proxies saved in that lst file ?

Thx

Stefan
Oleg Chernavin
07/04/2005 08:15 am
No. Rotation will be harder to implement. But each time you start a Project, it will get the information from the text file you supply.

Oleg.
Stefan
07/04/2005 08:17 am
ok, that would be definately somthing good to have. Please think about adding something like the rotation option, it could be very useful for certain sites blocking concurrent connections from the same IP.

Thx

Stefan
Oleg Chernavin
07/04/2005 08:29 am
I thought more and found a way to implement this quick. Now Proxy= parameter can use multiple proxy servers and use each line from the supplied file for the next URL the Project loads:

http://www.metaproducts.com/download/betas/oep2062.ZIP

Oleg.
Stefan
07/04/2005 08:35 am
so the new syntax for using this is "Proxy={:file=c:\proxyfile.txt}" ?

Thx

Stefan
Oleg Chernavin
07/04/2005 08:41 am
You can use this syntax and the old one:

Proxy=1,proxy.mycompany.com:3128

If you use Proxy={:file=c:\proxy.txt} then the file should contain lines in the same format:

1,proxy.mycompany.com:3128
1,proxy2.mycompany.com:3128
1,proxy3.mycompany.com:3128

Oleg.
Stefan
07/09/2005 04:50 am
To make me fully understand how this works now. If there is more than one proxy server given OE will only use one of these proxy servers for downloading files of a project? Or will OE actually use a different proxy from that list for each request sent out ?

Thx

Stefan
Oleg Chernavin
07/09/2005 05:08 am
It will use the next proxy for the next request looping through the whole list.

Oleg.
Stefan
07/09/2005 05:10 am
Thank you, that was what I was looking for. Just one more question: What happens if any of the proxies is unresponive/dead ?

Thx

Stefan
Oleg Chernavin
07/09/2005 05:27 am
There will be a timeout and then the URL will be added to the end of the queue to be retried later with another proxy.

Oleg.
Stefan
07/09/2005 05:35 am
Is the timeout setting the same for proxy and non proxy requests ?

Thx

Stefan
Oleg Chernavin
07/09/2005 05:42 am
Yes, there is a single timeout for all Internet operations.

Oleg.
Stefan
07/09/2005 06:01 am
I tried to use the new proxy list option, but can not get it to work. Here is the example project I created:

[Object]
OEVersion=Pro 3.9.0.2063
Type=0
IID=7089
Caption=http://www.spiegel.de/
URL=http://www.spiegel.de/Proxy={:file=C:\test\proxy.lst}
Lev=1
Weekday=257
LimTSize=10000
LimNumber=5000
LimTime=100
FTText.Exts=htmlhtmaspaspxjspstmstmlidcshtmlhtxtxttextxspxmlrxmlcfmwmlphpphp3
FTImages.Exts=gifjpgjpegtiftiffxbmfifbmppngipxjp2j2cj2kwbmplwf
FTVideo.Exts=mpgavianimpegmovfliflcvivrmramrvasfasxwmvm1vm2vvob
FTAudio.Exts=wavriffmp3midmp2m3uravocwmaape
FTArchive.Exts=ziparcgzzarjlhalayleirarcabtarpakacejarpdf
FTUDef.Exts=jscssssivbsdtdxslswfclass
FTText.B=ooxooo
FTImages.B=ooxooo
FTVideo.B=ooxooo
FTAudio.B=ooxooo
FTArchive.B=ooxooo
FTUDef.B=ooxooo
FTOther.B=ooxooo
FTSizes=0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,3,3,0,3,0
RProt=127
LastStart=67:198:25:231:207:209:226:64:
LastEnd=46:62:222:233:207:209:226:64:
S200=45
S304=1
SAbr=682
SPar=44
SSav=45
SLast=200
SSiz=2148173
SMdf=35
LFiles=46
LSize=2253590
Stopped=True
Flags=1
ImgDim=0,0,0,0
PrevURL=http://www.spiegel.de/


Contents of my poxy.lst file:
1,84.246.164.130:777

When starting this project, all requests are going directly to the webservers instead of being done through the proxy I gave.

Can you take a look at it and let me know what I am doing wrong ?

Thx

Stefan



Stefan
07/09/2005 06:04 am
In addition to my last post, when I set

Proxy=1,84.246.164.130:777

in the project urls, the setting get ignored as well and all requests are going directly to the webserver.

Stefan
Oleg Chernavin
07/09/2005 08:01 am
Thank you for noticing this! I just fixed it.

Oleg.
Stefan
07/09/2005 08:03 am
Would be wonderful if I could get the updated version where this is fixed.

Stefan
Oleg Chernavin
07/09/2005 08:11 am
Here it is:

http://www.metaproducts.com/download/betas/oep2063.ZIP

Oleg.
Stefan
07/09/2005 08:32 am
Thx a lot. Does now work fine. Already running a project with it.

Stefan
Oleg Chernavin
07/09/2005 08:38 am
Great! I have heard that there are some standards for proxy list files. If yes, I can add support of them to OE as well. Do you know them?

Oleg.
Stefan
07/09/2005 08:42 am
One standard that is widely in use the simple proxy list format:

some.proxy.com:2128
other.proxy.com:4234
192.168.1.1:80

and so on. There are a number of programs that can output lists like this. And as far as I know it is the most common way to list proxies.

Stefan
Stefan
07/09/2005 08:46 am
one more question about the proxy use: Does the retry count also apply to prjoects using proxies?
Oleg Chernavin
07/09/2005 08:52 am
OK. I added the standard format support as well - without the 1, prefix.

The retries count surely applies to all URLs - with and without proxy settings.

Oleg.
Stefan
07/09/2005 08:56 am
Is the new format ( without the "1,") automatically recognized?

Stefan
Oleg Chernavin
07/09/2005 08:59 am
Yes, without that 1, it will assume that the "1" format - HTTP Proxy type is used. But I haven`t uploaded that version yet.

Oleg.
Stefan
07/09/2005 09:19 am
This proxy list feature is absolutely great and helps a lot of simulating real world access to a website. You might think about customizing other options, like referer, AcceptedLanguage, BrowserID to be able to work with a file list as well. That would enable one to make it even more unlikely that any system can detect that the access comes from one computer only.

Thx

Stefan
Oleg Chernavin
07/09/2005 09:26 am
I think that if you use an empty referer and multiple proxies, the server will not catch you. Most used ways to detect a mass download are:

1. To see if there are too many simultaneous or too often requests from the same IP.
2. To use sessions.
3. To add a hidden link which will be not clicked by a human, but followed by an offline browser.

Oleg.
Stefan
07/09/2005 09:37 am
Basically I agree and disagree with you on at the same time. I am working on both sites (servers and clients) and while I try to actually analyze logs to find patterns that let allow for cathing mass downloaders and people trying to defaud websites, I am also working on the side trying to get information from websites that try their best to make it impossible. The more different the information being logged on the webserver side is the more difficult is it to find a pattern. E.g. using a large refer list (like 2000 or 3000 different refers) can help to actually confuse the website operator even more and make it more diffucult to see what is going on.

Stefan
Oleg Chernavin
07/09/2005 10:22 am
I understand. OK. I will plan to add this in future.

Oleg.
TJ
06/29/2007 05:46 pm
I have the same problem with proxies today. I downloaded the zip file but can not extract it. Is it possible that the zip file is damaged?

> Here it is:
>
> http://www.metaproducts.com/download/betas/oep2063.ZIP
>
> Oleg.
Oleg Chernavin
06/30/2007 07:26 am
This version is quite old already. Please download the latest 4.7 SR1 version:

http://www.metaproducts.com/download/opsetup.exe

Oleg.
TJ
07/01/2007 07:18 am
> This version is quite old already. Please download the latest 4.7 SR1 version:
>
> http://www.metaproducts.com/download/opsetup.exe
>
> Oleg.

I downloaded this new one and used the following command line:
"c:\\Program Files\\Offline Explorer Pro\\OE.exe" /new=url=www.yahoo.com,url=Proxy=1,122.2.2.2:8080

This command worked well before. But now it seems that the proxy can not be set. I know how to set up the proxy using the graphic interace, but I have to use commmand line for some other reason. Is there anything wrong in the command now? Thanks.
Oleg Chernavin
07/01/2007 02:59 pm
You can use the following command for that:

"c:\\Program Files\\Offline Explorer Pro\\OE.exe" /new=url=www.yahoo.com,url=Proxy=122.2.2.2:8080

This will mean HTTP proxy anyway.

Oleg.
TJ
07/01/2007 03:58 pm
This does work if all the proxies are Http ones. However, some proxy server types are socks4 and socks5. What should I do then with command line? Thanks.

> You can use the following command for that:
>
> "c:\\Program Files\\Offline Explorer Pro\\OE.exe" /new=url=www.yahoo.com,url=Proxy=122.2.2.2:8080
>
> This will mean HTTP proxy anyway.
>
> Oleg.
Oleg Chernavin
07/02/2007 06:43 am
So far, this cannot be done. Only if you would use a {:file=...} macros to load the list of proxies from a file.

Oleg.