Optimization of storing big amount of sites

Author Message
Andrey 01/19/2006 11:52 am
We use OLE while working with OEE. We need to download and keep up-to-date rather big quantity of sites but are simultaneousely limited in disk space. Does OEE give an opportunity to store sites` structure rather than storing all files downloaded from the WEB including htmls? What we need is to be able to download only new or modified files from the sites without physically storing the whole site locally.

We`ve tried to delete project files without deleting project itself (a project for downloading a site, which is managed by us), then we`ve modified few files on it and afterwards we`ve restarted the project download, but the modified files were not downloaded (actually there was no action at all).

Thanks in advance
Oleg Chernavin 01/20/2006 04:10 am
There is a way to do that. When you delete downloaded Project files (not the project itself), Offline Explorer Enterprise will ask you whether you want to keep the files information in a database. If you answer Yes, then the next download (using the Download only New and Modified files in the Project Properties dialog) will get only changed files.

Best regards,
Oleg Chernavin
MP Staff
Andrey 01/20/2006 05:15 am
Are you sure it works? As I said we had downloaded a site, modified some pages on it and tried to download it again, but nothing happened.

Can you explain how the OEE works when the project files were deleted ( without deleting information about them ) and user wants to download only new and modified files? If I am doing something wrong, it will help me to understand it and do it properly.

Thank you.
Oleg Chernavin 01/20/2006 05:25 am
Offline Explorer creates a database with the URL, modifucation date and original file size. Then it checks. Please make sure that If-Modified-Since command is correctly processed by your server. The server may not support it for ASP, PHP and other server-side scripts.

Oleg.
Andrey 01/24/2006 08:43 am
Hallo Oleg!

I`ve just tested deleting files with storing info about them. I`ve found an interesting thing - this mechanism doesn`t work when the folder`s location is marked as "default", but when I point the location of folder to any another directory, it works perfect.

But I didn`t find any way to do the same trick (delete project files without deleting information about them) via OLE. Is there any to do it?

Thank you
Oleg Chernavin 01/25/2006 11:16 am
I just uploaded the updated oe.exe file:

http://www.metaproducts.com/download/betas/oee2322.zip

You can now use IOEProject.DeleteFiles method. It has one integer parameter. Set it to 1 to create such database about deleted files.

Oleg.
Andrey 01/26/2006 06:06 am
Thank you!

I`ve just tested a new version of OE. And faced a problem: deleting files works perfect, but when I try to update project, or download missing files - nothing happens. The only one type of download that works is "restart download" :).

And another feature - I`d like to export files of project, before deleting them. When I write something like:

Project.Export("C:\\somefile.zip", 1, 1, 1, 1, 0, 0);
Project.DeleteFiles(1);

The file "somefile.zip" contains just "content.html". But using just

Project.Export("C:\\somefile.zip", 1, 1, 1, 1, 0, 0);

saves the project files into "somefile.zip".

What should I do to import project files before deleting them? (using "Sleep(TickNum)" is problematically, because the size of sites varies greatly )

Thanks for youe attention.




Oleg Chernavin 01/26/2006 07:02 am
You can use IMainOE.Exporting property to see if the Export still goes or not.

Regarding the site update - can you please give me more details on how to reproduce this?

Oleg.
Andrey 01/26/2006 07:26 am
> You can use IMainOE.Exporting property to see if the Export still goes or not.

Thank you, that is it.

> Regarding the site update - can you please give me more details on how to reproduce this?

Ok, that is what I`ve tested:
I`ve downloaded a site, delete its files ("yes" when OE asked me whether I want to download this files again) and tried to update it. nothing happend at all. It is very curiously, because yesterday, the same sequence of operations gaved the desired result.

Originally, I`ve performed the same sequence, but had to delete files via OLE - project update didn`t worked.


> Oleg.
Oleg Chernavin 01/26/2006 07:28 am
You should have "Download only new and modified files" set in the Project Properties.

Oleg.
Andrey 01/26/2006 07:35 am
> You should have "Download only new and modified files" set in the Project Properties.

It was set.
And when i said "nothing happened at all", I meant that download hasn`t even started. As if I didn`t push any button - no reaction at all.
Oleg Chernavin 01/26/2006 07:40 am
Can you please quit Offline Explorer and start it again - would this change anything? Also, can you please enable logging (Ctrl+W) and see if there will be any message?

Oleg.
Oleg Chernavin 01/26/2006 07:41 am
Can you please quit Offline Explorer and start it again - would this change anything? Also, can you please enable logging (Ctrl+W) and see if there will be any message?

Oleg.
Andrey 01/26/2006 07:48 am
> Can you please quit Offline Explorer and start it again - would this change anything? Also, can you please enable logging (Ctrl+W) and see if there will be any message?

No affect after restatring program and no messages in Log.

> Oleg.
Oleg Chernavin 01/26/2006 08:54 am
I can make a version that logs in details what exactly happens there. But please try to reboot your Windows before that.

Oleg.
Andrey 01/26/2006 09:58 am
> I can make a version that logs in details what exactly happens there. But please try to reboot your Windows before that.
>
> Oleg.

Reboot didn`t changed the situation
Oleg Chernavin 01/27/2006 08:33 am
OK. I will add more logging information and post a new version this Monday. Sorry that it takes some time.

Oleg.
Andrey 02/01/2006 07:08 am
> OK. I will add more logging information and post a new version this Monday. Sorry that it takes some time.
>
> Oleg.

Hallo Oleg!

I am waiting a new version of OE since this monday. Did you forget about me? :)

Andrey.
Oleg Chernavin 02/01/2006 10:03 am
I am sorry for the delay. Can you please update your oe.exe file to this one:

http://www.metaproducts.com/download/betas/oee2328.ZIP

Start the download. You should have the oe.log file after that. If it is small, please post its contents here. Otherwise, please zip it and send to support@metaproducts.com.

Thank you!

Oleg.
Andrey 02/01/2006 12:57 pm
> I am sorry for the delay. Can you please update your oe.exe file to this one:
>
> http://www.metaproducts.com/download/betas/oee2328.ZIP
>
> Start the download. You should have the oe.log file after that. If it is small, please post its contents here. Otherwise, please zip it and send to support@metaproducts.com.
>
> Thank you!
>
> Oleg.

Well? thats it is. I had to download a site, deleted its files and tried to update it or to load a missing files.

ds1
ds2
FMGroup=1
FN=E:\Offline Explorer Data\DB\P7292.dat
False
ds1
ds2
FMGroup=0
FN=E:\Offline Explorer Data\DB\P7292.dat
True
ds20
ds21
ds22
ds23
ds24
ds25
ds26
ds1
ds2
FMGroup=2
FN=E:\Offline Explorer Data\DB\P7292.dat
True
ds20
ds21
ds22
ds23
ds24
ds25
ds26
Oleg Chernavin 02/02/2006 03:10 am
Can you please send me the following files (if they exist):

E:\Offline Explorer Data\DB\P7292.dat
E:\Offline Explorer Data\DB\P7292.idx
E:\Offline Explorer Data\DB\P7292.blb

to support@metaproducts.com ?

Please zip the files before sending. Something is wrong with the database.

Oleg.
Andrey 02/06/2006 06:38 am
Hallo Oleg!

What`s about my problem?
Oleg Chernavin 02/06/2006 07:14 am
I am still waiting for the files. Have you sent them already?

Oleg.
Andrey 02/06/2006 10:40 am
> I am still waiting for the files. Have you sent them already?
>
> Oleg.

I sent the letter again. Did you get it?
Oleg Chernavin 02/07/2006 03:35 am
No yet. Can I write you myself directly to the E-mail you used in this forum?

Oleg.
Andrey 02/07/2006 04:32 am
Yes, of course. I was sending letters to support@metaproducts.com
Oleg Chernavin 02/07/2006 05:39 am
I just sent you an E-mail.

Oleg.
Andrey 02/07/2006 05:59 am
Did you receive my email?
Oleg Chernavin 02/07/2006 06:08 am
Yes, thank you! I will reply it once I reproduce and fix the issue.

Oleg.