Weird Disk Caching Issue
|Jerry||12/11/2015 01:42 pm|
|I am trying to use OE to download a website where the file sizes are typically huge (> 2GB). And a weird thing showed up, which I will describe shortly. I have tested on two of my laptops and both encountered the same issue. Below is the description using one of laptops.
In this case I have two hard drives. Disk 0: an internal SSD, partitioned into C: (system partition) and D:. Disk 1: an external portable drive, with only one partition (E:):
I set the temporary folder of OE to a folder in E: (the external drive), and I would like to have the contents downloaded also to somewhere in E:, as you can see in the screenshot:
Then I started to download the website using OE, with 8 channels:
Now the problem showed up, that is, OE consumed almost all of my C: partition.
When I looked into the windows task manager, I saw that both C:\ and E:\ are continuously writing, which means OE first caches in C:\ then move to the temp folder in E:\, and then finally move the downloaded file to the selected download folder. If I terminate OE in task manager, the available space in C:\ reverts to the normal state immediately.
I would greatly appreciate it if anyone has a good idea about what is happening here -- I honestly is far from skillful enough.
|Oleg Chernavin||12/16/2015 09:50 pm|
|Jerry, we tried to investigate what could be the reason for writing to disk C:\. So far, the best idea is the Windows virtual memory file, which gets cleared once the application exits.
Can I also ask you to run the DiskMon utility:
It allows to filter what applications to monitor and display exact paths and filenames accessed and written.
Please let me know what you find out. If it is Offline Explorer creating files on disk C:, please give me information on paths and filenames.
|Jerry||12/17/2015 04:38 am|
Thanks for your reply! DiskMon does not show the specific applications or files, so I am using ProcessMon instead:
And here is a log file that I captured using ProcessMon (you can use the same application to open it):
So the conclusion from the log file is -- OE does not directly create/write a caching file. The file caching is "invisible". I don't mean it's hidden; it's just not there. My available C: space is reduced but I cannot find the file that takes up the space. It is not logged in ProcessMon, nor can I find a related big file in C: using whatever file manager.
You can test it and download the following folder:
The nice thing about this folder is that the files are not too huge, and the server only allows about 4 downloading threads. Therefore, your C: won't be completely consumed up by the mysterious cache. However, you will still notice that around 1.5GB of space is taken when OE6 is downloading.
Please let me know what you think. Thanks!
|Oleg Chernavin||12/18/2015 08:00 pm|
|Can you see if it is c:\pagefile.sys that grows up? This file is hidden, but you can allow Windows Explorer to view hidden file and you will be able to see its size.
|Oleg Chernavin||12/18/2015 08:02 pm|
|Or another idea is to locate MS Internet Explorer cache directory and see it was grown up during the download?
|Jerry||12/20/2015 06:42 pm|
Very good, you get it :) YES, it is the IE cache directory. I made a screenshot:
I apologize that I am using a Chinese version of Win 8.1 in the screenshot, but I made a few translations and I guess you can understand.
So basically OE downloads everything to IE cache, and then moves things to OE temp folder, and finally moves things to OE download folder... The larger the files I want to download are, the larger the cache folder will grow (sometimes can be tens of gigabytes as I explained earlier).
|Oleg Chernavin||12/21/2015 07:40 pm|
|OK. This is quite strange, because I use instructions to prevent caching. I will try to reproduce this and monitor the cache. Then hopefully I will be able to find a way to fix that.
|Jerry||12/21/2015 10:46 pm|
Thanks. It happens to both of my laptops, as well as a native boot VHDX. Therefore it is due to either OE or my personal IE settings. I look forward to your investigations.
|Oleg Chernavin||02/13/2016 12:33 am|
I am sorry for answering late. I did tests on Windows 7 and Windows 10, but so far didn't reproduce this. I downloaded to disk D and monitored C with IE cache and system files. The download didn't take any additional space. Only what was downloaded to disk D:
I will keep monitoring this. If I get it reproduced, I am sure, I will be able to make improvements.
|Jerry||02/17/2016 07:32 am|
Thank you for getting back to me! So I guess it may be a problem with my IE settings. I have the same IE settings for all of my computers and all of them have this caching issue. Hopefully one day this mystery will be solved :)
Have a great day!