pipe downloaded files through a filer

Author Message
Stefan 07/09/2005 04:12 am
Is there any way to send each retrieved file thorugh a pipe filter upon download?

e.g. use of an external program to remove certain tags from the html files before it gets saved to disk

If you need more info on how that would be usefull let me know.

Stefan
Oleg Chernavin 07/09/2005 04:24 am
Yes, you can use the DataMining feature with the TextPipe Pro application for that.

Oleg.
Stefan 07/09/2005 04:30 am
But how about using my own utilities? Is that possible as well ? There are lots of ported utlis from the Unix world that can be used for filtering files on windows as well. Some kind of generic postprocessing after each download that could be called up would be great.

Stefan
Oleg Chernavin 07/09/2005 05:09 am
Well, you can take the Project Map in your utilities and process all of the files in it.

Oleg.
Stefan 07/09/2005 05:12 am
ok, that will be a some way to do actually do it for now. The reason I was looking for a pipe/filer functions after each download is for projects that are running for days/weeks and downloading 100.000 of files. I was actually thinking about even filtering these files upon retrieval by OE and then inserting them into a database while OE is still busy downloading the next 100.000 files.

Just so that you know why I am looking for that functioniality.

Thx

Stefan
Stefan 07/09/2005 05:14 am
ok, that will be a some way to do actually do it for now. The reason I was looking for a pipe/filer functions after each download is for projects that are running for days/weeks and downloading 100.000 of files. I was actually thinking about even filtering these files upon retrieval by OE and then inserting them into a database while OE is still busy downloading the next 100.000 files.

Just so that you know why I am looking for that functioniality.

Thx

Stefan
Oleg Chernavin 07/09/2005 05:28 am
I see. You can use Enterprise edition for such purposes - it will report you about each downloaded file.

Oleg.
Stefan 07/09/2005 05:36 am
Great I will look into the Enterprise version. Somehow you should make it more clear what the advantages of the Enterprise version are, so far I was under the impression that the only advantages are more URLs per project and OLE automation.

Stefan
Oleg Chernavin 07/09/2005 05:42 am
Yes, OLE Automation will help you to be notified about the files.

Oleg.