will massdownloader support rar preview?

Author Message
jiggy 01/11/2006 04:10 am
hi all, 3 questions:
1) will massdownloader support rar preview in the future?
2) what does FTP connection refused mean when trying to use an FTP link?
3) does FTP folders view work, tried it on a few links and it just comes up empty?!

Bonus questions:
why is that some sites refuse to honor "downloader tools." I don`t get it, what dif does it make how you get it? why should they care. And is there a workaround?

thanks for a great tool.
jigs
Alexander 01/11/2006 11:53 am
jiggy,

Thank you for writing us.

> hi all, 3 questions:
> 1) will massdownloader support rar preview in the future?

No, rar archives format does not support preview/partial download feature.

> 2) what does FTP connection refused mean when trying to use an FTP link?

This means that server does not allow to establish a connection (possible reasons: overloaded server, limited number of simultaneous users, blocked IP address, etc.)

> 3) does FTP folders view work, tried it on a few links and it just comes up empty?!

Could you please send me some example FTP links for a test? Thank you!

> Bonus questions:
> why is that some sites refuse to honor "downloader tools." I don`t get it, what dif does it make
> how you get it? why should they care. And is there a workaround?

The cause is that almost all modern download tools uses several connections. Some site administrator prohibit them because they consider that this will lower the server load.
This is the administrator`s right and we should not search for the workaround here.

> thanks for a great tool.

Thank you for your kind words.

Sincerely,
Alexander.
| Alexander Bednyakov
| MetaProducts Corporation
jiggy 01/12/2006 09:23 am
> jiggy,
>
> Thank you for writing us.
>
> > hi all, 3 questions:
> > 1) will massdownloader support rar preview in the future?
>
> No, rar archives format does not support preview/partial download feature.
>
> > 2) what does FTP connection refused mean when trying to use an FTP link?
>
> This means that server does not allow to establish a connection (possible reasons: overloaded server, limited number of simultaneous users, blocked IP address, etc.)
>
> > 3) does FTP folders view work, tried it on a few links and it just comes up empty?!
>
> Could you please send me some example FTP links for a test? Thank you!
>
> > Bonus questions:
> > why is that some sites refuse to honor "downloader tools." I don`t get it, what dif does it make
> > how you get it? why should they care. And is there a workaround?
>
> The cause is that almost all modern download tools uses several connections. Some site administrator prohibit them because they consider that this will lower the server load.
> This is the administrator`s right and we should not search for the workaround here.
>
> > thanks for a great tool.
>
> Thank you for your kind words.
>
> Sincerely,
> Alexander.
> | Alexander Bednyakov
> | MetaProducts Corporation
>

alexander, well i don`t think any of the ftp is working, i paste this link into the massdownloader 3.2 window: ftp://24.222.39.104/Shane/Lost.S01.DVDRip.XviD-WAT/lost.s01e10.dvdrip.xvid-wat.avi
and it just loops around connecting and reconnecting, but if i paste the same link into my ANCIENT version of AceFTP v1.22, all is well. Sorry, I lost the folder link I was trying the other day, but i bet the problem is related. Do u have a public ftp folder link i can try? thanks!!!! support here is grand.
jigs

Alexander 01/12/2006 10:23 am
Please try the following:

1. Go to the Mass Downloader Options dialog, Proxy server | FTP section
2. Check the "FTP passive mode" box
3. Click the OK

Then please try your FTP downloads and write me about the result.

Thanks,
Alexander
jiggy 01/12/2006 09:34 pm
> Please try the following:
>
> 1. Go to the Mass Downloader Options dialog, Proxy server | FTP section
> 2. Check the "FTP passive mode" box
> 3. Click the OK
>
> Then please try your FTP downloads and write me about the result.
>
> Thanks,
> Alexander

yes, that box was/is checked, same result. interesting, AceFTP died after about 45meg on that link. Perhaps the problem is these public ftp sites...but mad won`t even start on this link...does it work for you?
thx, jigs
Alexander 01/13/2006 02:45 am
> yes, that box was/is checked, same result. interesting, AceFTP died after about 45meg on
> that link. Perhaps the problem is these public ftp sites...but mad won`t even start on this
> link...does it work for you?
> thx, jigs

Please switch the Log on (Project Properties dialog, Log tab -> set all boxes, click the OK button) and take a look at the Log View. What does it tell you?
In my case it writes

FTP0: FTP server access denied.

This may means: server is overloaded, numbel of users exceeds the limit, IP address is blocked, etc.

Alexander
naaloh 04/08/2008 01:39 pm
> No, rar archives format does not support preview/partial download feature.

That's complete BS. Any RAR archive can be previewed and partial download and decompression is possible for any non-solid RAR archive. If you're too lazy to add it to your product, admit it and don't lie to your customers >:-E
Alexander 04/08/2008 03:12 pm
naaloh,

Thank you for writing us.
Could you please name the application that have RAR preview feature?

Best regards,
Alexander
naaloh 04/09/2008 05:41 am
> naaloh,
>
> Thank you for writing us.
> Could you please name the application that have RAR preview feature?
>
> Best regards,
> Alexander

Sure. WinRAR, for example :-)) Of course, it doesn't work with files on the Internet, it reads them from a disk, but it doesn't change anything. You see, the structure of a RAR archive is basically HEADER-DATA repeated as many times as there are files in the archive. The "HEADER" part contains info about the size of the "DATA" part, so by adding the size to the offset of the HEADER's end we'll get the offset of the next header. This is how the preview done. Of coure, if we download the "DATA" part of an archived file, we are guaranteed to have the ability to extract the file (provided the archive is not a solid one). If you are stubborn enough to insist that some kind of black magic will prevent us from doing it with a file on the Internet, I'll tell you what I did when I needed to see what's inside of a RAR archive and download/extract only what I needed. I used the program called Segment Downloader, which allows to manually specify which part(s) of a file should be acquired (would be VERY nice to have this ability in Mass Downloader, BTW). First I downloaded the first few kilobytes of a file, then opened this incomplete archive in Total Commander (although virtually any program that works with RAR archives can be used) and looked at the COMPRESSED size of the last file whose HEADER was in the archive. Adding the size to the offset of the end of the last HEADER I got the offset from which to download the next HEADER, so I entered it in the download program and downloaded another 500 bytes from this offset, repeating three previous steps until I got to the file I needed, for which I downloaded the "DATA" part as well. After that I could extract the file. If you think that the operations I described cannot be done automatically, you are hopeless.
Alexander 04/09/2008 08:01 am
naaloh,

Thank you for your reply.
Now I see what you mean. This isn't a "preview" exactly, this is RAR "scanning." Yes, this can be implemented automatically. Thank you for your persistence, we added the feature you request to the Mass Downloader development file.

Sincerely,
Alexander
naaloh 04/09/2008 10:05 am
> Thank you for your persistence, we added the feature you request to the Mass Downloader development file.

You mean there is a REAL chance that this will be in Mass Downloader? If you are really going to add it, then I have to apologize for my arrogant tone. You see, I was 99% sure that in the end I'd get some lame excuse of why it "can't" be done.

But what do you mean by "This isn't a "preview" exactly this is RAR "scanning""? Scanning is surely a wrong term for this, scanning would mean looking through every bit of something in order to find what's needed, while here we're just looking through headers in order to get the complete picture of what's inside.
Alexander 04/09/2008 12:15 pm
> But what do you mean by "This isn't a "preview" exactly this is RAR "scanning""? Scanning is surely a wrong term for this, scanning would mean looking through every bit of something in order to find what's needed, while here we're just looking through headers in order to get the complete picture of what's inside.

This looks like scanning in opposite to the current zip preview feature. Just imagine that you preview rar archive with, say, thousand of small files. This will take the huge amount of time and the corresponding amount of traffic. The "traditional" zip preview does this just in two steps.

We will try to implement this soon.

Sincerely,
Alexander
naaloh 04/09/2008 01:19 pm
Well, the header without filename (and, possibly, relative path) is only 32 bytes. Even if we assume we are previewing an archive with 5000 files with an average filename length of 200 characters this only gives us about 1.1 MB of traffic. Of course, the actual traffic will be greater, but not huge by anyone's standarts.

Anyway, creating separate utilities for each archive type is not a good solution. What's really needed is a tool that can allow to preview ANY type of archive (or non-archive). How's this possible? Through a virtual disk with a virtual file linked to an URL. The virtual disk program would download data from the Internet according to a system read request, transferring the data to the process which initiated such a request (for example an archiver utility). This would allow to get only the needed data regardless of the type of file previewed/decompressed. It all doesn't have much to do with Mass Downloader, but maybe you know a software which can do this (map a virtual file to a URL)?
Alexander 04/09/2008 02:43 pm
No, I don't.

Alexander
naaloh 05/18/2008 05:10 am
> We will try to implement this soon.
>
> Sincerely,
> Alexander

How are the trials going?
Alexander 05/21/2008 02:34 am
We work on releasing of the 3.5 version now. After this we plan to start implementing this feature.

Thanks,
Alexander
naaloh 10/02/2008 05:13 am
> We work on releasing of the 3.5 version now. After this we plan to start implementing this feature.
>
> Thanks,
> Alexander

I see you are no better at keeping your promises than I am :-/
Alexander 10/03/2008 03:07 am
Naaloh,

We are working under this feature! :-)
The first prototype already works.

I hope the developer will finish this task to the next release.

Sincerely,
Alexander
Alexander 11/06/2008 07:43 am
Naaloh,

Could you please test the latest release of Mass Downloader?
It has the feature you requested.

Best regards,
Alexander Bednyakov
MP Staff
naaloh 11/06/2008 03:24 pm
I''ll test it on Saturday. Big thanks for incorporating this feature :)
Alexander 11/07/2008 02:31 am
You are welcome!
Please let me know how does it work for you. Send us problem links (if any,) your comments, etc.

Thank you for your help.
Alexander
naaloh 11/08/2008 04:39 am
Well, let''s start with the fact that the newest version doesn''t have the feature I requested, since it only supports preview, but not partial download, which is the main reason for previewing, don''t you agree?

Now to bugs: when the "Enable archives preview" option is enabled in the "Create new Project" dialog box, Mass Downloader starts downloading from the beginning after I click the "Start" button and simultaneously displays the "The file ''*'' appears to be a zip archives. Do you want to browse it?" dialog box (a RAR archive will be called "a zip archives" as well). If I answer "Yes", it retrieves the directory of the archive parallel to downloading it whole.

Now I''ll make a very easy-to-do, but also very significant improvement request. You see, one of the usual reasons to preview an archive is to see if certain files in it are changed (or equal to the ones you already have). So, would you be so kind as to make this possible to do with Mass Downloader by making the "Archives listing" window show file sizes in bytes and display CRC-32 checksum of each file. Thanks in advance.
AlexanderBa 11/11/2008 03:10 am
It''s very strange. Mass Downloader supports partial downloading of RAR files. Could you please describe exactly what steps do you do to download some files from archive. Please send the link to your rar-file for a test.
naaloh 11/11/2008 03:18 pm
> It''''s very strange. Mass Downloader supports partial downloading of RAR files. Could you please describe exactly what steps do you do to download some files from archive. Please send the link to your rar-file for a test.

File > New > entered link > OK > clicked on the task > Tools > Preview archive > clicked on the "manual.txt" file > Download. The file started downloading from the beginning.

The link: http://p2ktools.motofan.ru/download/FlexTools.rar

The log

TEST0: Testing for preview...
TEST0: Looking up host p2ktools.motofan.ru...
TEST0: Connecting to host p2ktools.motofan.ru...
TEST0: Host p2ktools.motofan.ru connected. Waiting for http://p2ktools.motofan.ru/download/FlexTools.rar.
TEST0: GET /download/FlexTools.rar HTTP/1.0
TEST0: Accept: */*
TEST0: Range: bytes=0-63
TEST0: Request-Range: bytes=0-63
TEST0: User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
TEST0: Host: p2ktools.motofan.ru
TEST0: Connection: Keep-Alive
TEST0: HTTP/1.1 206 Partial Content
TEST0: Server: nginx/0.6.13
TEST0: Date: Tue, 11 Nov 2008 20:17:33 GMT
TEST0: Content-Type: application/x-rar-compressed
TEST0: Content-Length: 64
TEST0: Last-Modified: Sun, 23 Oct 2005 14:10:11 GMT
TEST0: Connection: close
TEST0: Content-Range: bytes 0-63/236991
TEST0: Download complete.
TEST0: Close connection. Status - 206, Reason - 0
TEST0: Connecting to host p2ktools.motofan.ru...
TEST0: Host p2ktools.motofan.ru connected. Waiting for http://p2ktools.motofan.ru/download/FlexTools.rar.
TEST0: GET /download/FlexTools.rar HTTP/1.0
TEST0: Accept: */*
TEST0: Range: bytes=64-64
TEST0: Request-Range: bytes=64-64
TEST0: User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
TEST0: Host: p2ktools.motofan.ru
TEST0: Connection: Keep-Alive
TEST0: HTTP/1.1 206 Partial Content
TEST0: Server: nginx/0.6.13
TEST0: Date: Tue, 11 Nov 2008 20:17:33 GMT
TEST0: Content-Type: application/x-rar-compressed
TEST0: Content-Length: 1
TEST0: Last-Modified: Sun, 23 Oct 2005 14:10:11 GMT
TEST0: Connection: close
TEST0: Content-Range: bytes 64-64/236991
TEST0: Download complete.
TEST0: Close connection. Status - 206, Reason - 0
TEST0: Found 1 files
TEST0: Found name: FlexTools.exe
TEST0: Connecting to host p2ktools.motofan.ru...
TEST0: Host p2ktools.motofan.ru connected. Waiting for http://p2ktools.motofan.ru/download/FlexTools.rar.
TEST0: GET /download/FlexTools.rar HTTP/1.0
TEST0: Accept: */*
TEST0: Range: bytes=231728-231771
TEST0: Request-Range: bytes=231728-231771
TEST0: User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
TEST0: Host: p2ktools.motofan.ru
TEST0: Connection: Keep-Alive
TEST0: HTTP/1.1 206 Partial Content
TEST0: Server: nginx/0.6.13
TEST0: Date: Tue, 11 Nov 2008 20:17:33 GMT
TEST0: Content-Type: application/x-rar-compressed
TEST0: Content-Length: 44
TEST0: Last-Modified: Sun, 23 Oct 2005 14:10:11 GMT
TEST0: Connection: close
TEST0: Content-Range: bytes 231728-231771/236991
TEST0: Download complete.
TEST0: Close connection. Status - 206, Reason - 0
TEST0: Found 2 files
TEST0: Found name: manual.txt
TEST0: Connecting to host p2ktools.motofan.ru...
TEST0: Host p2ktools.motofan.ru connected. Waiting for http://p2ktools.motofan.ru/download/FlexTools.rar.
TEST0: GET /download/FlexTools.rar HTTP/1.0
TEST0: Accept: */*
TEST0: Range: bytes=232543-232586
TEST0: Request-Range: bytes=232543-232586
TEST0: User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
TEST0: Host: p2ktools.motofan.ru
TEST0: Connection: Keep-Alive
TEST0: HTTP/1.1 206 Partial Content
TEST0: Server: nginx/0.6.13
TEST0: Date: Tue, 11 Nov 2008 20:17:34 GMT
TEST0: Content-Type: application/x-rar-compressed
TEST0: Content-Length: 44
TEST0: Last-Modified: Sun, 23 Oct 2005 14:10:11 GMT
TEST0: Connection: close
TEST0: Content-Range: bytes 232543-232586/236991
TEST0: Download complete.
TEST0: Close connection. Status - 206, Reason - 0
TEST0: Found name: Seem
TEST0: Co
AlexanderBa 11/12/2008 09:31 am
I was able to reproduce your issue. I think I fixed it. Please download and test:
http://www.metaproducts.com/download/betas/mdsetup.exe
Please write about the result.
izkhan 11/14/2008 08:24 am
> I was able to reproduce your issue. I think I fixed it. Please download and test:
> http://www.metaproducts.com/download/betas/mdsetup.exe
> Please write about the result.

I downloaded Massdownloader and was surprised to see that it worked perfectly by providing RAR file preview. I was also able to download individual files from rar a file which I never expected before. That is very rare feature and I was awaiting for this for ages.
naaloh 11/15/2008 03:34 am
test
naaloh 11/15/2008 03:50 am
> test

Really what''s this "checking by a moderator" crap??? Some messages post normally, others don''t.
naaloh 11/15/2008 03:58 am
> test

Anyway, I posted the reply to AlexanderBa yesterday and re-posted it today, but each time got the dumb "checking by a moderator" message and nothing was added to this thread yet. Maybe only short messages are coming through. In any case THE PROBLEM STILL PERSISTS IN THE BETA VERSION.