Compact webdown.dat

Nick
03/22/2006 08:44 am
I filled in the "New Features Request Form" on the OE page a while back and did not get any feedback so I thought I`d post the ideas here too…

If you have a large number of projects then the webdown.dat file can become very large and rather cumbersome. OE can then take a long time to read/write this file which is not ideal. However, if you look at what is stored in the webdown.dat file, most of it is common to a great deal of projects. This is especially true if you use templates. For example about 90% of my projects are common to about 5 templates. If webdown.dat had a default setup for a project (like a default "template") it could significantly reduce the data stored by storing just the differences between each project and this template. This could be taken further if the template.dat and webdown.dat files were merged. A project could them be stored as the template that was applied last plus any modifications. In the worst case where all projects were unique this would result in a single extra line per project (for example “Template=default”). In the best case it would result in a significant saving. Consider the following extract from webdown.dat (www.google.co.uk downloaded with my default template)

[Object]
Type=0
IID=283
Caption=http://www.google.co.uk/
URL=http://www.google.co.uk/
Weekday=257
LimTSize=10000
LimNumber=5000
LimTime=100
FTText.Exts=html htm asp jsp stm stml idc shtml htx txt t ext xsp xml rxml cfm wml php php3
FTImages.Exts=gif jpg jpeg tif tiff xbm fif bmp png ipx jp 2 j2c j2k wbmp lwf
FTVideo.Exts=mpg avi ani mpeg mov fli flc viv rm ram rv asf asx wmv m1v m2v vob
FTAudio.Exts=wav riff mp3 mid mp2 m3u ra voc wma ape
FTArchive.Exts=zip arc gz z arj lha lay lei rar cab tar p ak ace jar
FTUDef.Exts=js css ssi vbs dtd xsl swf
FTText.B=oooooo
FTImages.B=oooooo
FTVideo.B=oooooo
FTAudio.B=oooooo
FTArchive.B=oooooo
FTUDef.B=oooooo
FTOther.B=oooooo
FTSizes=0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,3,0,3,0,3,0
RProt=63
LastStart=66:187:166:89:209:241:226:64:
LastEnd=180:14:190:89:209:241:226:64:
S200=2
SPar=1
SSav=2
SLast=200
SSiz=12585
SMdf=2
LFiles=2
LSize=12585
ImgDim=0,0,0,0
PrevURL=http://www.google.co.uk/

Using my idea this could be compressed to:

[Object]
> new Template=default
Type=0
IID=283
Caption=http://www.google.co.uk/
URL=http://www.google.co.uk/
LastStart=66:187:166:89:209:241:226:64:
LastEnd=180:14:190:89:209:241:226:64:
S200=2
SPar=1
SSav=2
SLast=200
SSiz=12585
SMdf=2
LFiles=2
LSize=12585
PrevURL=http://www.google.co.uk/

This represents compaction to below 30% of the original size. If a new template was applied to the project (completely changing the setup), only the “Template=…0” line would need to change.

The only complication I can think of is that when a template is changed it means every project which was based on that template will need to be re-evaluated. Alternatively, when a template is being changed/deleted, OE could check if any project was “assigned” to that template and, if it was, create its own hidden copy of the template which is about to be changed and assign the projects involved to the hidden version. The hidden version would need a name that is guaranteed to be unique from any user defined template. You could for example use the syntax ~X where X is a number (ie 1 for the first hidden template, 2 for the next etc) and if “~” was made illegal to use as the first character for any user defined template then there could never be a mix up.

I would love a feature like this as it would cut my webdown.dat file down and also waste less space with all the backup files of webdown.dat the OE creates every time you startup.

This could also be extended further as follows:
- only store “PrevURL” if it is different from “URL”
- I am not sure what all of the statstics stored are used for. Ie

Oleg Chernavin
03/22/2006 09:01 am
I thought about this, but there are more problems. Many people transfer webdown.dat file to another computer, which may have different templates. Many people copy Projects to the clipboard and send them by E-mail, which will mean that the template might be missing on the other end or has changed settings.

I am afraid, this change in webdown.dat will cause many other problems which are more serious than wasted disk space.

Best regards,
Oleg Chernavin
MP Staff
Nick
03/22/2006 09:16 am
> Many people transfer webdown.dat file to another computer, which may have different templates.
Why not assign a hidden template name to each template used and store these hidden templates at the start of the webdown.dat file. That way the file can be moved and will take the record of the templates applied with it. Every time you apply a template to a project OE could search the hidden templates for one that is identical and if it finds one assign it to that template. If it doesn`t exist OE could then create a new hidden template and assign it to that. To help with the search the hidden template could keep track of the users name for the template when it was last applied. This would speed up the search for existing templates if this field was searched first.

> Many people copy Projects to the clipboard and send them by E-mail, which will mean that the template might be missing on the other end or has changed settings.
Could you not either copy the associated template with the project or alternatively expand the project to its full version using the template when you copy. When you paste back in, OE could search the available templates for one that matches and assign it to any one that matches.

I can`t believe many people have huge numbers of templates so searching through them should be pretty quick.

Do you have any idea how large a webdown.dat file OE can support reliably? Is there a limit?



Oleg Chernavin
03/22/2006 09:37 am
OK. I will think about this. I have also to worry about backwards-compatibility issues.

Oleg.