[Beta 4] 100% CPU and no GUI after some hours of downloading

Peter
07/20/2011 07:47 am
Hello,

after a few hours of downloading
(http://amp.dascene.net/, download only Archives, ca 220,000 .gz files, ca. 500,000 PHP links to parse)
OEE uses 100% CPU and the OEE window has no GUI anymore.

Is here a beta forum anywhere?

Peter
Oleg Chernavin
07/20/2011 07:48 am
Can you please look at the Status Bar - whether there is a Parsing (xx) message? If parsing goes on, then it will really occupy all CPU cores to process downloaded files. You may place the download(s) in Suspend mode (F9) and wait until parsing completes. Then resume (F9) the download.

Best regards,
Oleg Chernavin
MP Staff
Peter
07/20/2011 08:01 pm
There is no Status Bar visible. Just an empty window.
Pressing F9 doesnt do anything.
I even reduced the work to http://amp.dascene.net/downmod.php?index={:2..114919}
Level 2

The settings are:

[Object]
Type=0
IID=7010
Caption=http://amp.dascene.net/downmod.php?index={:2..114919}
URL=http://amp.dascene.net/downmod.php?index={:2..114919}
Lev=2
Weekday=257
LimTSize=10000
LimNumber=5000
LimTime=100
EnableForms=True
FTText.Exts=htmlhtmaspaspxjspstmstmlidcshtmlhtxtxttextxspxmlrxmlcfmwmlphpphp3
FTImages.Exts=gifjpgjpegtiftiffxbmfifbmppngipxjp2j2cj2kwbmplwf
FTVideo.Exts=mpgavianimpegmovfliflcvivrmramrvasfasxwmvm1vm2vvob
FTAudio.Exts=wavriffmp3midmp2m3uravocwmaape
FTArchive.Exts=ziparcgzzarjlhalayleirarcabtarpakacejarpdftgz
FTUDef.Exts=jscssssivbsdtdxslswfclass
FTText.B=xoxooo
FTImages.B=xoxooo
FTVideo.B=xoxooo
FTAudio.B=xoxooo
FTArchive.B=ooxooo
FTUDef.B=xoxooo
FTOther.B=ooxooo
FTSizes=0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,3,3,3,1,3,1
NotIgnoreLogout=False
RSrvsBx=1
RProt=255
LastStart=168:246:16:183:30:229:227:64:
LastEnd=169:69:87:108:20:229:227:64:
LastStarted=20.07.2011 23:02:10
LastEnded=20.07.2011 15:19:02
S200=109460
SPar=66708
SLast=200
SSiz=43108678
SHTML=109460
SSuccDowns=1
LFiles=109460
LSize=71447350
Flags=1
ImgDim=0,0,0,0
PrevURL=http://amp.dascene.net/downmod.php?index=2
ParseComplexScripts=True
IPAddr=337385310
LIndexed=False
IndexFiles=False
Peter
07/21/2011 02:41 am
I restarted the project and watched the parsing:
after about 40K downloads the files to parse went slowly up to about 3K and i paused the download.
Parsing these files was quite fast at this point and i resumed the download.
After about 60K downloaded files and pausing again the download the parsing is very slow (like about 1 file per second).
since then i have to pause the download after about every 1K files and wait for the parsing.

I also found 65,535 .tmp files in the Temp folder.
Peter
07/21/2011 03:37 am
Paused after another 1000 downloaded files (@71000 total downloaded by now): 930 files to parse.
Parsing speed is about 1 file/sec
Both CPU Cores are most of the time 100% used by oe.exe

AMD Athlon 64 X2 4200+, 2.20 GHz, 2 GB RAM (850 MB available), Windows 2003 Server SP2, no other tools are running.


With OEE 4.x i've never experienced such a slowdown
Oleg Chernavin
07/21/2011 04:55 am
Can you try to clean the Temp folder and start Offline Explorer from command line with the parameters:

oe.exe /NoDB /1parser

Would this improve speed of parsing and CPU usage?

Oleg.
Peter
07/21/2011 06:16 am
well...

after another 1000 downloaded files (950 to parse) i suspended again.
parsing was up to ca. 6 sec/file @ 72000 files downloaded so far (another ca. 100K to go).
I waited 1,5 hours till 0 files to parse.
Did "Suspend to File" which saved a 36MB .wdq file.
Closed OE with File/Exit and waited for a clean shutdown of OE
(it stayed active until the temp folder was empty).

After starting OE there was no Project anymore
I tried to Download/Resume from File but nothing was loaded (multiple times).
No Project, no Queue, nothing.
Maybe i should have saved the project changes too prior closing OE.

I'll create this project again and try your suggestion.

Peter
07/21/2011 08:10 am
OK
I restarted the server and recreated the Project.
Closed OE and restarted it with oe.exe /NoDB /1parser

Download runs quite smooth to 65K (I suspended it a few times to watch parsing some 100 files, which were very fast counting down).
The situation changed at about > 65,500 files downloaded: parsing takes some secs for each file while downloading is suspended.
Sysinternals Process Explorer shows Both CPU Cores are ca. 70%[green] (OE is accounted for ca. 40% [red]) utilized.
Oleg Chernavin
07/21/2011 09:16 am
Can you see if the Temp folder is occupied with files? It looks very much like files get stored somewhere and access to that folder slows down significantly.

Oleg.
Peter
07/21/2011 09:30 am
The OE Temp folder (configured to C:\OFFLINE\TEMP) is once again occupied with 65.535 htt[1-FFFF].tmp files.

These were created while OE was downloading.
E.g. when OE had downloaded x files the same amount of files were created in the Temp folder, until 65535 .tmp files, no more than this "limit" since then.
Oleg Chernavin
07/21/2011 12:49 pm
Thank you very much for analysing the problem! It helped me to identify and fix it. Please download the updated version:

http://www.metaproducts.com/download/betas/opbsetup.exe

Let me know how it works.

Oleg.
Peter
07/22/2011 07:30 pm
Sorry for the delayed answer.

The project was successfully downladed in time.
File processing seems now to be slower than in beta 4 so i had to change the pause between downloads from 0 to 1 sec. and the number of concurrent downloads from 8 to 25-30 to use the optimal ratio of file processing and download speed.
Oleg Chernavin
07/25/2011 06:34 pm
OK. I see. I will check why processing is slower now.

Oleg.