Queue Lenght Limit

Author Message
ashura 01/09/2005 09:23 pm
I`ve noticed Brad Konia`s "Restrict connections per server" topic.

My problem is, I`ve tried to download a database on a web site, but after a certain unknown number of contemporary queries the pages of database are obscured, so OE download blank pages (you can see only the same message on every page).

So, number of connections is indifferent, the problem is the number of queries.

Can I limit the lenght of queue? I need the program to automatically limit queries to the number I`ve chosen. Is it possible to implement this feature in a future relase?

Thank you! excuse me for my english, your program is the best in its category.
ashura
Oleg Chernavin 01/10/2005 10:42 am
I think, you can do it right now - please see the Limits section in the Project Properties dialog | Advanced.

Best regards,
Oleg Chernavin
MP Staff
ashura 01/11/2005 11:25 am
> I think, you can do it right now - please see the Limits section in the Project Properties dialog | Advanced.
>
> Best regards,
> Oleg Chernavin
> MP Staff

I have tried your suggestion. It just stop downloading the project when the number of files exceeds.
When I update the project it resumes the list of files in queue, then the server stops my ip.

My necessity, I think, downloading a file should be resolved before launching next query.
I don`t know if that`s feasible, but I think it could resolve many problems with many sites.

Maybe I have to use a dynamic ip?
Even I could try to download site part by part.

Thanks a lot

ashura
Oleg Chernavin 01/11/2005 02:47 pm
I think, it is possible. Please try running the Project in the "Do not load existing files" mode (in its Properties dialog). This way, each new download will load only new files.

Regarding the server which bans your IP - perhaps you can work this around by setting less connections (2-3) in the Options dialog.

Maybe also setting random delays between downloads, like 2-4. This will simulate human browsing.

Of course, this slows down the download, but it may solve the banned IP problem.

Oleg.
ashura 01/11/2005 08:03 pm
> I think, it is possible. Please try running the Project in the "Do not load existing files" mode (in its Properties dialog). This way, each new download will load only new files.
>
> Regarding the server which bans your IP - perhaps you can work this around by setting less connections (2-3) in the Options dialog.
>
> Maybe also setting random delays between downloads, like 2-4. This will simulate human browsing.
>
> Of course, this slows down the download, but it may solve the banned IP problem.
>
> Oleg.

It seems work.
I have download the whole site with missing pages, then I have download missing sections using "do not load existing files" feature..

thanks a lot,

ashura
Oleg Chernavin 01/13/2005 05:14 am
Great!

Oleg.