do not download existing files problem

Author Message
Stefan 08/22/2005 09:33 am
With the following project I do have the problem that OE is starting to download existing files regardless of wether the files already exist in the project directory or not. Basically whenever I start the download all files are download again.


[Object]
OEVersion=Pro 3.9.0.2103
Type=0
IID=7080
Caption=PR
URL=http://www.nedstatbasic.net/catalogue/top1000?page={:1..20}&category={:file=c:\whois-records\nedststatcat.txt}&country=PRProxy={:file=C:\whois-records\proxy23.lst}#Additional=DoNotParseFiles#Channels=5#delay=2
Minute=10
Weekday=257
LimTSize=10000
LimNumber=5000
LimTime=1
FMGroup=2
FTText.Exts=htmlhtmaspaspxjspstmstmlidcshtmlhtxtxttextxspxmlrxmlcfmwmlphpphp3
FTImages.Exts=gifjpgjpegtiftiffxbmfifbmppngipxjp2j2cj2kwbmplwf
FTVideo.Exts=mpgavianimpegmovfliflcvivrmramrvasfasxwmvm1vm2vvob
FTAudio.Exts=wavriffmp3midmp2m3uravocwmaape
FTArchive.Exts=ziparcgzzarjlhalayleirarcabtarpakacejarpdf
FTUDef.Exts=jscssssivbsdtdxslswfclass
FTText.B=ooxooo
FTImages.B=xoxooo
FTVideo.B=xoxooo
FTAudio.B=xoxooo
FTArchive.B=xoxooo
FTUDef.B=xoxooo
FTOther.B=ooxooo
FTSizes=0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,3,3,0,3,0
RProt=127
LastStart=181:91:66:151:84:215:226:64:
LastEnd=4:111:229:139:84:215:226:64:
S200=660
S304=555
S400=22
SPar=1214
SSav=660
SLast=200
SSiz=10740584
SMdf=628
LFiles=1215
LSize=11043061
Flags=1
ImgDim=0,0,0,0
PrevURL=http://www.nedstatbasic.net/catalogue/top1000?page=1&category=2001&country=PR

Stefan 08/22/2005 09:39 am
Also a second question, I have set the level limit to 0 to make sure that only the urls are downloaded that are given by the macro I use, however OE still does download a couple of different pages from other domains in this project.

Thx

Stefan
Oleg Chernavin 08/22/2005 09:49 am
Strange. I replaced {:file=c:\whois-records\nedststatcat.txt} with auth and removed the Proxy= line. It didn`t attempt to load existing files.

OE loads the following URL:

http://www.nedstatbasic.net/i/b/maxonline_searchbox.html

This happens, because it is an IFRAME, which is the part of the page.

You can use the following to stop loading it:

Additional=SkipIFrames

Best regards,
Oleg Chernavin
MP Staff
Stefan 08/22/2005 09:54 am
Here are the contents of the nedstatcat.txt file:

2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
3001
3002
3003
3004
3005
3006
3007
3008
3009
3010
3011
3012
3013
3014
3015
3016
3017
18026
4001
4002
4003
4004
4005
4006
4007
4008
4009
4010
4011
18027
5001
5002
5003
5004
5005
5006
5007
5008
5009
5010
5011
5012
5013
5014
5015
5016
5017
6001
6002
6003
6004
6005
6006
6007
6008
6009
6010
6011
7001
7002
7003
7004
7005
7006
7007
8001
8002
8003
8004
8005
8006
8007
8008
8009
8010
8011
8012
8013
8014
8015
9001
9002
9003
9004
9005
9006
9007
9008
9009
9010
9011
9012
10001
10002
10003
10004
10005
10006
10007
10008
10009
10010
10011
10012
10013
10014
10015
11001
11002
11003
11004
11005
11006
11007
11008
11009
11010
11011
11012
11013
12001
12002
12003
12004
12005
12006
12007
12008
12009
12010
12011
13001
13002
13003
13004
13005
13006
13007
13008
14001
14002
14003
14004
14005
15001
15002
15003
15004
15005
15006
15007
15008
15009
15010
15011
15012
15013
15014
15015
15016
15017
15018
15019
15020
16001
16002
16003
16004
16005
16006
16007
16008
16009
16010
16011
16012
16013
16014
16015
16016
16017
16018
16019
16020
16021
16022
16023
16024
16025
16026
16027
16028
16029
16030
16031
16032
16033
16034
16035
16036
16037
16038
18019
18023
18024
18025
17001
17002
17003
17004
17005
17006
17007
17008
17009
17010
17011
17012
17013
18001
18002
18003
18004
18005
18006
18007
18008
18009
18010
18011
18012
18013
18014
18015
18016
18017
18018
Stefan 08/22/2005 09:55 am
contents of proxy23.lst (as of right now these are working proxies):

12.106.28.143:80
12.106.28.144:80
12.42.48.194:80
128.232.103.201:3128
129.132.57.4:3128
129.240.228.138:3128
130.208.18.29:3124
130.208.18.29:3127
130.208.18.30:3127
132.72.23.11:3124
134.2.205.228:3124
140.109.17.181:3128
140.112.107.80:3128
140.112.107.80:3127
140.112.107.80:3124
140.112.107.82:3128
140.112.107.82:3124
140.112.107.82:3127
142.103.2.2:3124
142.103.2.2:3128
143.248.139.169:3124
143.248.139.170:3124
148.223.234.130:444
148.244.150.52:80
148.244.150.57:80
148.244.150.58:80
150.165.15.18:3124
150.165.15.18:3128
150.165.15.19:3128
150.165.15.19:3124
161.53.156.3:80
193.136.157.20:80
193.136.157.37:80
193.252.28.111:80
194.89.17.4:80
195.116.244.17:80
195.55.222.19:80
195.87.69.242:80
195.96.195.26:80
200.129.0.162:3124
200.159.255.70:3128
200.159.255.80:3128
200.181.57.53:8080
200.189.74.118:8080
200.252.134.118:6588
201.15.75.226:3128
202.130.84.133:8080
202.143.150.142:8080
202.175.234.163:8080
202.175.60.218:80
202.79.220.51:3128
203.113.132.35:80
203.116.214.2:80
203.144.216.211:80
203.160.169.85:8080
203.199.92.158:80
207.248.240.118:80
207.248.240.119:80
210.0.200.3:80
210.212.176.228:80
210.243.16.125:80
211.25.50.156:80
212.175.113.52:3128
212.55.28.125:80
213.97.196.205:80
217.117.51.18:80
217.19.87.67:80
218.104.85.101:81
218.188.23.162:8080
218.244.225.180:80
218.93.119.83:8080
219.22.50.60:8080
221.10.55.203:8080
221.142.244.144:8080
221.212.177.97:80
222.240.128.4:8080
24.232.121.93:3128
24.232.95.53:8080
24.26.123.163:2301
24.6.111.200:8080
24.88.255.44:2301
61.0.62.4:8080
61.221.199.204:80
61.95.224.173:80
62.97.72.76:80
63.165.31.7:65208
63.218.109.130:8080
65.37.50.76:2301
65.43.209.233:3382
68.120.235.145:8080
68.121.75.73:80
68.193.81.129:3382
68.62.181.57:2301
69.57.138.6:80
80.237.140.233:8888
80.237.140.233:3127
80.237.140.233:8081
80.25.150.39:80
80.26.4.208:80
80.32.151.115:80
80.38.212.134:80
80.38.3.248:80
80.59.91.33:80
81.169.168.175:80
81.199.22.91:80
81.68.131.3:6588
82.157.33.132:63000
82.201.185.22:8080
82.77.200.162:3128
83.145.68.131:80
85.18.29.25:8080
192.16.125.12:3127
202.79.220.50:3128
81.199.24.18:80
Stefan 08/22/2005 09:56 am
Please give it a try as soon as possible (the proxies will expire very soon), it just does not work as I would expect it.
Oleg Chernavin 08/22/2005 10:20 am
I am sorry, but I will be able to try that only tomorrow.

Oleg.
Stefan 08/22/2005 10:22 am
Please leave a message here when you are ready to try it out and I will provide you with a working proxy list then. The current behaviour drives me a bit crazy.
Oleg Chernavin 08/22/2005 10:25 am
I just tried that quickly and I didn`t see any issues. I stopped and started the download several times and every time it started loading new links, not the ones that were already downloaded.

Oleg.
Stefan 08/22/2005 10:28 am
Then I honestly do not understand why it starts to download all files over and over again when using the setup as I have posted it here. Once I am finished with the project I can start it again and it will start to download the files one more time. Any idea why this is happening ?

Thx

Stefan
Stefan 08/22/2005 10:31 am
Addition to my last post:
The existing files are overwritten in the download directory. From my understanding the program should check if the file is in the download directory before it gets added to the queue, correct (with the settings I am using)?

Oleg Chernavin 08/23/2005 03:04 am
Yes, this is how it works - it checks every file before downloading in the Download Directory. It worked well for me.

Can you please try on another computer?

Oleg.