5 questions
1) i get error messages that crash the program.
1 of them was "division by zero" and it would keep popping up multiple times
another was something about a thread, but it only happened once
2) suspend to file
A) used to save as a huge 1GB+ file but now only 30-45 MB's?
B) cant seem to get the function to work properly in order to reload the queue files
i suspend manually, then i suspend to file and replace the file i was working on
Is this correct Procedure? If Not, how should i do it?
How Long does it take to restore the project, when i had the huge 1GB+ queue files it took 2-3 hours to "suspend to file" and 30 minutes to "resume from file" but now i have a faster computer
Intel Pentium 4 1.8 GHz to a AMD 64 x2 4400+ and you cant even tell when its completed
I need some kind of log entry when you start the "suspend to", and "resume from file", because lately it hasnt been working and i have to use the start/download missing files all the way from the begginning
3) Project Map/Site Map feature
I have been ignoring it, but maybe i should be generating a site map. Would this be a good thing to do.
4) How do i download everything but the "forum" which would allow me to save more files since the limit of enterprise is 100,000,000. ?
5) Also i cant seem to get the program to download just the *.torrent files of
isohunt.com 4.1Million torrents
demonoid.com 267,300 torrents
i tried only including *.torrent in the filename filter but the suspend to file is not working properly and its endless circle, also even though i disable bit torrent protocol, it is still winding up downloading the contents of the torrents file, instead of just the torrents themselves
CONCLUSION:
If i could get the program to correctly save the queue files i might not lose all the time spent
Maybe i need to lokk into creating/generating the site map. I have now 6or7 sets of .wdq and .wdqh files and none of them seem to give me back the list of over 2 million in queue i had when i saved them. Strangely they range from 15 MB's up to 45MB's in size. Did youy change the format to be more efficient or is something totally wrong 45 MB's for 2 million in queue??
ALso if i could just figure out, or you could incorporate a function to just download the torrents from a torrent website, that would save me a lot of trouble. I still have to get used to tweaking the application with the filters and disabling bit torrent protocol, i should be able to do it.
I would really like to help you, in the way of plugins.
A program that is doing very well with plug-ins right now is JDownloader, and they have their users help them to make configurations for diffewrent websites. Maybe you could add this function to your application, where people could figure out the best ways to download each section of a website seperately or entirely and donate a configuration file to your database on your site. PLUG-INS are awesome, and very powerful, but in the case of JDOWNLOADER it is open source so maybe thats why they do that-----just an idea to help make everybodys days easier in the end
common sections of websites that could be useful using plugins or config files
TO COPY or NOT TO COPY:
Forums
Advertisements
blogs
*.torrents (IS THERE A BETTER WAY THEN THE WAY IM DOING IT NOW??)
etc.
Maybe by check mark
1) i get error messages that crash the program.
1 of them was "division by zero" and it would keep popping up multiple times
another was something about a thread, but it only happened once
2) suspend to file
A) used to save as a huge 1GB+ file but now only 30-45 MB's?
B) cant seem to get the function to work properly in order to reload the queue files
i suspend manually, then i suspend to file and replace the file i was working on
Is this correct Procedure? If Not, how should i do it?
How Long does it take to restore the project, when i had the huge 1GB+ queue files it took 2-3 hours to "suspend to file" and 30 minutes to "resume from file" but now i have a faster computer
Intel Pentium 4 1.8 GHz to a AMD 64 x2 4400+ and you cant even tell when its completed
I need some kind of log entry when you start the "suspend to", and "resume from file", because lately it hasnt been working and i have to use the start/download missing files all the way from the begginning
3) Project Map/Site Map feature
I have been ignoring it, but maybe i should be generating a site map. Would this be a good thing to do.
4) How do i download everything but the "forum" which would allow me to save more files since the limit of enterprise is 100,000,000. ?
5) Also i cant seem to get the program to download just the *.torrent files of
isohunt.com 4.1Million torrents
demonoid.com 267,300 torrents
i tried only including *.torrent in the filename filter but the suspend to file is not working properly and its endless circle, also even though i disable bit torrent protocol, it is still winding up downloading the contents of the torrents file, instead of just the torrents themselves
CONCLUSION:
If i could get the program to correctly save the queue files i might not lose all the time spent
Maybe i need to lokk into creating/generating the site map. I have now 6or7 sets of .wdq and .wdqh files and none of them seem to give me back the list of over 2 million in queue i had when i saved them. Strangely they range from 15 MB's up to 45MB's in size. Did youy change the format to be more efficient or is something totally wrong 45 MB's for 2 million in queue??
ALso if i could just figure out, or you could incorporate a function to just download the torrents from a torrent website, that would save me a lot of trouble. I still have to get used to tweaking the application with the filters and disabling bit torrent protocol, i should be able to do it.
I would really like to help you, in the way of plugins.
A program that is doing very well with plug-ins right now is JDownloader, and they have their users help them to make configurations for diffewrent websites. Maybe you could add this function to your application, where people could figure out the best ways to download each section of a website seperately or entirely and donate a configuration file to your database on your site. PLUG-INS are awesome, and very powerful, but in the case of JDOWNLOADER it is open source so maybe thats why they do that-----just an idea to help make everybodys days easier in the end
common sections of websites that could be useful using plugins or config files
TO COPY or NOT TO COPY:
Forums
Advertisements
blogs
*.torrents (IS THERE A BETTER WAY THEN THE WAY IM DOING IT NOW??)
etc.
Maybe by check mark
1. Can I make a version that creates detailed error logs? You would make a download using this version and when the crash happens, I would need a file with the details. This should help me find out and fix the error.
2.
A. The size totally depends on the number of URLs in the queue. The more files, the bigger .wdq file will be. .wdqh size depends on how long download worked before the suspend to file. It contains lists of URLs (encoded) that were already downloaded, so duplicates are ignored.
B. Yes, this is the correct sequence. But the time is really too long. The newest version allows to abort long parsing process and save to file immediately. This is quite safe to do.
3. No, it will not influence the behavior.
4. Please use Project Properties - URL Filters sections to exclude the desired URLs. Various forums have different URLs, so you can give me a site example and I will suggest you what to do exactly.
5. You need to allow HTML files and .torrent files, because HTMLs are necessary to download to get links from them.
Yes, I am thinking on such universal templates feature now.
Best regards,
Oleg Chernavin
MP Staff
2.
A. The size totally depends on the number of URLs in the queue. The more files, the bigger .wdq file will be. .wdqh size depends on how long download worked before the suspend to file. It contains lists of URLs (encoded) that were already downloaded, so duplicates are ignored.
B. Yes, this is the correct sequence. But the time is really too long. The newest version allows to abort long parsing process and save to file immediately. This is quite safe to do.
3. No, it will not influence the behavior.
4. Please use Project Properties - URL Filters sections to exclude the desired URLs. Various forums have different URLs, so you can give me a site example and I will suggest you what to do exactly.
5. You need to allow HTML files and .torrent files, because HTMLs are necessary to download to get links from them.
Yes, I am thinking on such universal templates feature now.
Best regards,
Oleg Chernavin
MP Staff
Oleg,
It looks like Paul has been having the same problems to me.
Even thought 'Suspend to File' is better in 5.8, it will still crash from time to time. I still find it safer to Suspend first and then Suspend to File second. (Then 'Unsuspend' by re-clicking the Suspend button.)
For the record, his file sizes are similar to mine so very large sites still seem to be a problem.
Wozza
It looks like Paul has been having the same problems to me.
Even thought 'Suspend to File' is better in 5.8, it will still crash from time to time. I still find it safer to Suspend first and then Suspend to File second. (Then 'Unsuspend' by re-clicking the Suspend button.)
For the record, his file sizes are similar to mine so very large sites still seem to be a problem.
Wozza
I made a new version that should create an error log once the crash happens - c:\download\oe.log.
Please download it here:
http://www.metaproducts.com/download/betas/oep3212.zip
Unzip the file and replace the old oe.exe file with the new one.
When it crashes, please post in the forum and send the log file to support@metaproducts.com. Thank you!
Oleg.
Please download it here:
http://www.metaproducts.com/download/betas/oep3212.zip
Unzip the file and replace the old oe.exe file with the new one.
When it crashes, please post in the forum and send the log file to support@metaproducts.com. Thank you!
Oleg.