This amount of information and indexing process is becoming too big, so this is what I would like to know: I want to "freeze" those 100 sites stored on a folder, say, named '2006', index it, and then let that information and index freezed. Also, I would create a new folder '2007' for the pages downloaded from now over, with a new index only for the '2007' folder.
The problem is that I think that if I create the new '2007' folder (empty), the next time OE executes, it will download most of the 'old' pages again, so quickly the '2007' folder will be as big as the '2006' folder.
Is it possible for OE to "check already downloaded files" on the '2006' folder, but download the new ones in the '2007' folder? How could I do this ??
Thank you very much for your help.
I'm very sorry that can't be done.
I think this may be very common problem: folders that become huge to manipulate, index, store, back-up, etc.
Do you think that it would be very difficult to have a parameter that tells OE to check for existing files on a folder, but download the new ones on a different folder? That would be a great solution !
Thank you again.
> Well, if two folders use different download directories, this is impossible. Maybe only to use URL Filters to filter out pages that contain older dates (if URLs of the site contain date in some way).
> Best regards,
> Oleg Chernavin
> MP Staff
Let me ask a different question: what happens if I move all my downloaded directories, to a new location, completely deleting them from the place where OE downloads now. If now I execute OE will it download again most of that pages, or will use the information stored in \Documents and Settings\... to skip the already existing files ?
In different words: do OE maintain a "map" of all the files it has already downloaded from each site ?
I need to find a way to move those 80 Gigs to a different location, but I don't want OE to start downloading them again...!
Thank you very much for your help, and an excellent product.
However if you need to move previously downloaded files to another drive and download updated files to the previous directory, then Offline Explorer is unable to use files on the other drive to skip duplicates.
> However if you need to move previously downloaded files to another drive and download updated files to the previous directory, then Offline Explorer is unable to use files on the other drive to skip duplicates.
Dear Oleg; sorry to bother again with this issue, but I'm still trying to find the most simple solution for this problem. I think that, sooner or later, many OE user will need the ability to "freeze" and archive projects because they have grown just too big to move, index, and process.
This is "the best" solution I have found yet:
Create a new Url Field Parameter "CheckExistentAlsoOn=", followed of the path to the "project archive".
Like this: CheckExistentAlsoOn = d:\OldArchive\2006\www.domainname.com\
The way it should work is like this: when OE checks if a file exists, it should check BOTH at the usual projec'ts location, and ALSO at the path pointed to by the parameter CheckExistentAlsoOn=
This way, you can move huge amount of files to a "project archive", with a permanent, non-changing index, and the project will continue downloading at the usual folder, but checking for duplicates also in the project archive.
Also, it is very easy from time to time to move the "new downloaded" files, to the "project archive".
I hope this could be implemented... it doesn't seem too difficult... I hope.
I don't understand completely your point. In my proposal, OE should work just as it works now. The only addition is that when checking for the existency of files, it will check BOTH in the folder it checks now, and the folder indicated by CheckExistentAlsoOn=
It is the responsability of the system administrator to set the correct path in the CheckExistentAlsoOn= parameter.
Hope it helps...
I need this...
You will have to use the following URLs field command:
Unfortunately, I will not be able to test it until next monday...
Thank you very much for your great service and great product.
I will let you know the result of my tests...
> OK. I implemented this. The updated oe.exe file is here:
> You will have to use the following URLs field command:
Today I will start testing the new build 2602 I have just downloaded.
I want to confirm how to use the new OtherDownloadDir= parameter. I have two questions:
1) If I have a project currently downloading to D:\OE\PROJECTS\SITE\, and move it completely to an "old project archive" in D:\BACK.UP\PROJECTS\SITE\, then the right command would be:
2) I'm using the advanced options feature (checkbox) "prevent overloading Windows filesystem", so OE splits large folders with over 1000 files into smaller ones. Does the new OtherDownloadDir= keep this in mind? Old files might be in those subfolders named %&OvrX ...
2. I made a quick implementation of this feature and it doesn't support overloaded directories yet.
> 2. I made a quick implementation of this feature and it doesn't support overloaded directories yet.
Mmmm, I think that will have to wait until you have time to implement the checking of existing files in the \%&OvrX folders, because ALL my projects are big enough that they are stored in many \%&OvrX folders.
If I execute now, I will download zillons of duplicate files.
Thank you very much,
Any chance to finish the "OtherDownloadDir=" command with support for overloaded directories ??
GREAT ! I will start testing the new feature.
Thank you !
I have started to test the new OtherDownloadDir=, with some strange results:
A- In one of my sites, to test OtherDownloadDir=, I have moved all the files to a backup folder. Then started again the download, and... bingo! no new file was downloaded, since OEP checked for its existency in the backup folder, so there was nothing to download.
B- However, on several other projects, after moving its data and refreshing the project ... everything is downloaded again.
To search for the causes of this behaviour, I tryed to "refresh" these projects (B) without the OtherDownloadDir= command, and found that these projects download existing files once and again. So the problem is that it is not working properly the checking for existing files.
Since most of my projects are "news style projects" the settings I use is "Exclude existing files on levels over 0". This should mean that ANY EXISTING FILE will not be downloaded again, right ?
What might be the problem?
I think that the problem is not on the OtherDownloadDir= command, but on the checking for existing files. This is something I noted months ago, but didn't give it enough attention.
My theory is that when you check "Exclude existing files on levels over 0", OEP does "something else" than checking for the existence of the file; it is checking for the date or size or something...that causes the file to be downloaded again.
Thank you very much,
I think that I have found something:
If I update a project (Shift-F5), OEP does not keep in mind existing files, and downloads everything (or most) of the files again, not keeping in mind files existing neither in the current download folder, nor the OtherDownloadDir= folder. I think this is NOT correct.
If I update a project with F5, seems that OEP does keep in mind both existing files in the current download folder, and also in the "backup folder" (OtherDownloadDir=). This IS correct.
Hope this helps.
The project's info is the following:
Caption=Punto a Punto
I think that if a project includes the OtherDownloadDir=, OEP should prevent any action that could arise in duplicates...don't you think?
Now let me ask a different question about this: I have read that OE Enterprise creates a compact database of all downloaded files, and this database is used to prevent downloading duplicates. What if instead of using OEP, I use OEE... I could delete all the project's files periodically, moving them to the backup destination folder manually, and OEE will still download only new and modified files, correct?
Thank you for your help!
> Well, this looks as a correct behavior. When you press Shift- or Ctrl- or Alt-F5, it ignores the File Modification check settings in the Project Properties and uses the way you chose with the keystroke. And it looks like the server always reports these files as changed or new, so updating the site causes its redownload all the time.
Also want add now that
when put into the box just deletes everything (.jpg''s, .zip''s, etc.) after download before it can be moved to the new folder.
Oh, and just wanted to add that archive files like zip and rar always download, even if they are in the original download folder. From what I can see, they seem to download as a .php file, and are then are renamed by OE after download. It appears the program looks for the .php file when it is comparing and not the zip file. So they always download no matter what. Big problem when you update download content often.
> > Oleg.
> Oh, and just wanted to add that archive files like zip and rar always download, even if they are in the original download folder. From what I can see, they seem to download as a .php file, and are then are renamed by OE after download. It appears the program looks for the .php file when it is comparing and not the zip file. So they always download no matter what. Big problem when you update download content often.
Gah, no edit post button. Also wanted to say that I am using OE Enterprise 5.6, not pro. Its just that I found this message in the OE pro forum, and thought it was a good place to post and continue the discussion.
Hey Oleg, sent you an email with the login information.
Sorry, my Gmail account seems to be fried. I can''t access it at all. Could you please send email to Logan9773@hotmail.com.