Few things...

Author Message
Eros 09/25/2012 10:59 pm
Well, I've done some spidering...

I got 1 crash, Windows fault I guess...
Faulting application oe.exe, version 6.4.0.3842, faulting module kernel32.dll, version 5.1.2600.5781, fault address 0x00012afb.

I got 1 crash due to OE itself, since there was no Event log. I remember seeing 0x00000000 error message.

Both of course happened after 3-4h of spidering and involves hundreds of sites so it is hard to trace the problem...


I've done some picture grabbing with OE. I set a limit to not to download pictures smaller than 800x600, but I still sometimes get pictures smaller than that. Happened with 3 different targets already. May this be because server denies to tell picture resolution and OE can't do anything, but to just download it?
I haven't encountered this problem before, but may there be a connection with Server excluded keywords list? Since I have over 500 keywords in there.


I also have a suggestion. I want to use my custom browser ID, just the updated one, but why do I have to set it manually every damn time? I want it to remember it and stay like that.


Once last thing.
photos.igorbass.com - uses flash to show images. Isn't there really any way to make OE to "click" and "right-click" on that SWF so I could get the image?
Eros 09/25/2012 11:17 pm
I just found something odd in my last picture grabbing project.

I found 2 files with PHP3 extension, both containing lines about streaming errors.
Oleg Chernavin 09/26/2012 08:35 am
I am sorry for the problems!

1. Next time during lengthy downloads can you please watch the System Monitor tab on the bottom of OE screen - the memory it can use is limited by 2 GBs. If it gets close to that, it will surely crash.

I want to understand, whether this issue is memory-related or a bug inside my code.

2. Maybe the format is not among the suppored. Can you give me several URLs of such images?

3. Please give me the ID you are using. I typed several words in the field and it was preserved in several exits and starts of OE.

4. Can you try to use Browse with AutoSave to load all images from the Flash applets?

5. I also need more details about this.

Best regards,
Oleg Chernavin
MP Staff
Eros 09/29/2012 11:11 am
About identifier, I'm using this:
Mozilla/5.0 (Windows NT 5.1; rv:15.0) Gecko/20120819 Firefox/15.0 PaleMoon/15.0

On every new project it resets back to FireFox 10.0 - it's not even the correct name for that browser, it's Firefox, without second capital F!


About images...
Already fourth project where some images bypass 800x600 image resolution limit.

Here are few images from a lot:

http://assets1.desktopnexus.com/thumbnails/36638-thumbnail.jpg
http://assets2.desktopnexus.com/thumbnails/1031813-thumbnail.jpg
http://cuddlycomments.com/icon.bmp
http://g38.picoodle.com/ltd/img38/5/10/23/loki/f_1eol_649_ucbpu.gif
http://img1.ak.crunchyroll.com/i/spire1/07232008/0/d/2/8/0d28f0b38199c0_full.jpg
https://img4.custompublish.com/getfile.php/1350288.464.dfuybybxra/740x0/4826221_1350288.jpg
http://img5.visualizeus.com/thumbs/c3/01/c3017ad909b9b6324e2a2b1b9fa3bb0b_h.jpg
http://img.fotocommunity.com/photos/27409186.jpg
http://isanam.com/scraps/thank-you/thank-you-84.gif
https://tracking.hostgator.com/img/Shared/300x250.gif
https://www.flobbo.de/banner/flobbo_150x150__1.gif
http://www.freshscraps.com/thank-you-scraps/Thank-You-Scraps-08.gif
http://www.goodlightscraps.com/content/thinking-of-you/thinking-of-u-26.gif
https://www.imageshotel.org/images/nkerbrat/decoq.gif
http://www.orkut7.com/orkut_scraps/weekend_scraps/weekend_scraps_3.gif
http://www.ubercomments.com/icon.bmp

Here's a really weird one:
http://static.desktopnexus.com/thumbnails/171528-thumbnail.jpg

When viewing that thumbnail in Windows Explorer, it appears rotated left.

As if I never set the minimum limit at all...


And about other issues, I still have to collect more information.
Oleg Chernavin 09/29/2012 11:13 am
Thank you! I fixed the user-agent issue.

I also improved dimensions detection for most JPEG files and added support for BMP.

One JPG cannot be resolved still.

Here is the new oe.exe file version:

http://www.metaproducts.com/download/betas/OEP3850.zip

Oleg.
Eros 09/29/2012 05:51 pm
Thank you!

One guestion, though. How does OE prioritize downloads?
For example, I set a target and I allow project to load up to 2 links on other servers, does it just grab everything it encounters or does it prioritize, so that first it gets everything from main target and then it goes spidering on other servers?
Eros 09/29/2012 07:24 pm
And what's the purpose of WSOE.exe? Why does it want to connect to internet?

The scary part is this information:
http://www.isthisfilesafe.com/sha1/9A2BC96D5C4D7B3B65981EAC93A2F0A7FA70FD97_details.aspx
Oleg Chernavin 09/30/2012 04:33 pm
There is no special priority for downloads - all files get added to the queue and downloaded one by another.

WSOE.exe - it is a module to make screenshots of the downloaded Projects. It simply loads the URL into an embedded MS IE window (invisible), makes its screenshot, saves the .PNG file and exits.

You may delete WSOE.exe if it causes inconvenience. It will not influence Offline Explorer.

Oleg.
Eros 09/30/2012 05:13 pm
Well, it would be nice to have priority as an option.
Oleg Chernavin 10/01/2012 05:35 am
So far, we plan priority for file types, not servers. This will appear in 7.0 version.

Oleg.