Mirroring a large website, 1 million+ files, 50GB+ html/png data.
OEP has consistently hung for me after 50k to 250k downloads. Completely random. It's hung about 6 times now and I was very close to giving up. Was just getting ready to try wget as a last resort. OS: Windows 8 Enterprise. I tried running OEP as Admin and running in Windows 7 compatibility mode, still hung. I attached Ollydbg to the hung process in a desperate attempt to fix it....threads 5 thru 13 were still running, and the main window thread was still running. The other threads were waiting around in Windows code(NTDLL). One of them had a PATH_NOT_FOUND error, but I wasn't successful in getting things moving again. I didn't dump the stacks of the suspect threads, sorry. It did look like 3 threads were waiting forever though.
I figured I'd give it one last shot, but for this run I disabled GZIP and disabled IE cookies, and now I'm at 600k downloaded files and 20GiB downloaded this session, and I'm certain it will finish this time around.
FYI here are my connection settings that prevent me from getting firewalled by the remote server, but also allow me to download at 8-10 megabits: 16 connections, .25 second delay.
OEP has consistently hung for me after 50k to 250k downloads. Completely random. It's hung about 6 times now and I was very close to giving up. Was just getting ready to try wget as a last resort. OS: Windows 8 Enterprise. I tried running OEP as Admin and running in Windows 7 compatibility mode, still hung. I attached Ollydbg to the hung process in a desperate attempt to fix it....threads 5 thru 13 were still running, and the main window thread was still running. The other threads were waiting around in Windows code(NTDLL). One of them had a PATH_NOT_FOUND error, but I wasn't successful in getting things moving again. I didn't dump the stacks of the suspect threads, sorry. It did look like 3 threads were waiting forever though.
I figured I'd give it one last shot, but for this run I disabled GZIP and disabled IE cookies, and now I'm at 600k downloaded files and 20GiB downloaded this session, and I'm certain it will finish this time around.
FYI here are my connection settings that prevent me from getting firewalled by the remote server, but also allow me to download at 8-10 megabits: 16 connections, .25 second delay.
Thank you very much for reporting the bug! Can you please tell me Offline Explorer Pro version you are using?
I will try to reproduce this and fix the error.
Best regards,
Oleg Chernavin
MP Staff
I will try to reproduce this and fix the error.
Best regards,
Oleg Chernavin
MP Staff
One other thing, I was noticing before that OEP was slowing down after a little while, too...but CPU usage wasn't increasing. I assumed it was my router, even though I have a very beefy router with a custom version of OpenWRT.
This time around though with GZIP/IE cookies disabled OEP has been running full speed the whole time, so that also points to a threading a bug IMO. Good luck on tracking it down, I have quite a bit of experience working with multithreading so I know how difficult it can be to track a bug down. I didn't bother to check which compiler you're using, but definitely use try/catch/finally pattern for acquiring and releasing locks if you can. I don't remember what the equivalent is in C++, but I know I used something like it on my last C++ threading adventure.
This time around though with GZIP/IE cookies disabled OEP has been running full speed the whole time, so that also points to a threading a bug IMO. Good luck on tracking it down, I have quite a bit of experience working with multithreading so I know how difficult it can be to track a bug down. I didn't bother to check which compiler you're using, but definitely use try/catch/finally pattern for acquiring and releasing locks if you can. I don't remember what the equivalent is in C++, but I know I used something like it on my last C++ threading adventure.
Using 6.5.3880.