images are not visible on retrieved pages

Marek
06/17/2013 01:52 pm
Hi Oleg,

I'm trying to retrieve the following part of the website with OEE 6.6.0.3944:
http://lessons.runrev.com/

After finishing the project I checked few pages in the internal browser as well as Firefox and no images are visible. I know that the image files have been downloaded and the s3.amazonaws.com folder (175MB) is there into which the images have been downloaded. The images are on some other server http://s3.amazonaws.com/ and have that kind of very long URLs:
http://s3.amazonaws.com/screensteps_live/step_images/runrev/30491/media_1303300801336_display.png?AWSAccessKeyId=AKIAJRW37ULKKSXWY73Q&Expires=1371436177&Signature=CEGMKxF8hYKOEX%2F7q689yBFmr+I%3D

The downloaded image files have this kind of names:
media_1268654745764.png@AWSAccessKeyId=AKIAJRW37ULKKSXWY73Q&Expires=1371471412&Signature=9HDGrPT0YY0865NF+2IGPpc_252FPVw_253D

In the project properties I have of course selected to Load images from any site

I have tried use the Browse with AutoSave feature but it does not work either.

I also tried to retrieve just a part of these lessons here:
http://lessons.runrev.com/s/lessons/m/4069/ and it's the same problem.

The only thing that works OK is retrieving just 1 page, just 1 lesson, then the images are downloaded and visible.

In all these projects I have selected to Load images from any site

I've spent several hours now on downloading and don't know if this is a bug or if I'm doing something wrong.

Please let me know how to fix this problem.

Thanks and regards,

Marek

Oleg Chernavin
06/17/2013 02:06 pm
I made a download with the latest version. I used the URL:

http://lessons.runrev.com/s/lessons/m/4069/

Level=1. Allowed to download from the starting server and directory in URL Filters. I browsed pages and all images were there after the download. Maybe it was some bug that is recently fixed.

Can you please update your OE.exe file to the latest version:

http://www.metaproducts.com/download/betas/OEE3965.zip

Thank you!

Best regards,
Oleg Chernavin
MP Staff
Marek
06/18/2013 10:38 am
Thanks Oleg,

I updated the exe file and run the same project as you did:
http://lessons.runrev.com/s/lessons/m/4069/
Level=1. Allowed to download from the starting server and directory in URL Filters. I browsed pages and all images were there after the download. Maybe it was some bug that is recently fixed.

Few pages have images most of them don't. Some pages have 1-2 images and the remaining 1- or 2 are missing.
But I know that lots of images have been downloaded.

Any ideas what to do? Why is it happening?

Marek
Marek
06/18/2013 12:18 pm
Oleg,

Perhaps we can compare how many images you've got and how many I've got running the same project.
So in the s3.amazonaws.com folder there are 86 Files, 20 Folders with total of 7.11 MB (7,457,336 bytes)

How many you have?

Marek
Oleg Chernavin
06/18/2013 12:21 pm
Could you take a look at the Properties of a missing image and its location and see if it is named differently on the hard disk?

Oleg.
Marek
06/18/2013 12:39 pm
I checked for example the first image on this page:
using-the-hardware-back-button-on-android

In the Properties it shows:
media_1303300801336_display.png@AWSAccessKeyId=AKIAJRW37ULKKSXWY73Q&Expires=1371559896&Signature=T%2BIqCicZEQL1Q7cAQnyR74JoJpM_253D

and

URL
http://127.0.0.1:800/7021/s3.amazonaws.com/screensteps_live/step_images/runrev/30491/media_1303300801336

But I don't find it on the disk. And there are only 52 files on the disk with that kind of name starting with "media_"
But when I was running the project there were few hundred images listed in the Que and being downloaded, showing the percentage and eventually message Downloaded. This is very strange!

Marek


Marek
06/18/2013 12:50 pm
Another missing image is:
media_1306239951061_display.png@AWSAccessKeyId=AKIAJRW37ULKKSXWY73Q&Expires=1371561425&Signature=0k_252F08lww5kyP%2B1ul%2BPe%2B%2B6jZYAg_253D

http://127.0.0.1:800/7021/s3.amazonaws.com/screensteps_live/step_images/runrev/32692/media_1306239951061_display.png@AWSAccessKeyId=AKIAJRW37ULKKSXWY73Q&Expires=1371561425&Signature=0k_252F08lww5kyP%2B1ul%2BPe%2B%2B6jZYAg_253D

So perhaps the images were listed and shown as downloaded but not saved on the disk.

Why?

Another strange thing that I just discovered is when I open the Properties of the project, in the Addresses (URLs) box I found this:

http://lessons.runrev.com/s/lessons/m/4069/
SingleURL=http://s3.amazonaws.com/screensteps_live/step_images/runrev/94412/media_1358336633556_display.png?AWSAccessKeyId=AKIAJRW37ULKKSXWY73Q&Expires=1371560052&Signature=1fdUHaWvu%2F24F1TIV9vAuzAepGs%3D
SingleURL=http://s3.amazonaws.com/screensteps_live/step_images/runrev/94412/media_1358336745692_display.png?AWSAccessKeyId=AKIAJRW37ULKKSXWY73Q&Expires=1371560052&Signature=AYX4%2F4UR6J05ocNTPvNffzpzLJA%3D
SingleURL=http://s3.amazonaws.com/screensteps_live/step_images/runrev/58669/media_1332260124532_display.png?AWSAccessKeyId=AKIAJRW37ULKKSXWY73Q&Expires=1371559922&Signature=Dg+S7hAmQqVENKMhpFmMao3ZJb8%3D
SingleURL=http://s3.amazonaws.com/screensteps_live/step_images/runrev/58669/media_1332260792441_display.png?AWSAccessKeyId=AKIAJRW37ULKKSXWY73Q&Expires=1371559922&Signature=fXLZG7If3McS1+RSLCpf2A4KxE0%3D
SingleURL=http://s3.amazonaws.com/screensteps_live/step_images/runrev/58669/media_1332260836641_display.png?AWSAccessKeyId=AKIAJRW37ULKKSXWY73Q&Expires=1371559922&Signature=rtGDSZa4Cn0B9MJBVr+wglGgv4s%3D
SingleURL=http://s3.amazonaws.com/screensteps_live/step_images/runrev/48552/Screen_shot_2011-10-26_at_14.37.51_display.png?AWSAccessKeyId=AKIAJRW37ULKKSXWY73Q&Expires=1371560038&Signature=NLdQVRMTJ+BvuMWJI+pMLlcTUv4%3D
SingleURL=http://s3.amazonaws.com/screensteps_live/step_images/runrev/48552/Screen_shot_2011-10-26_at_18.20.32_display.png?AWSAccessKeyId=AKIAJRW37ULKKSXWY73Q&Expires=1371560038&Signature=JJexhpGV7QFbdi7wWzI6QvDgh9o%3D
SingleURL=http://s3.amazonaws.com/screensteps_live/step_images/runrev/48552/Screen_shot_2011-10-26_at_16.09.57_display.png?AWSAccessKeyId=AKIAJRW37ULKKSXWY73Q&Expires=1371560038&Signature=rQAStt14FC1FSgCisZ3EgeQj7CE%3D
SingleURL=http://s3.amazonaws.com/screensteps_live/step_images/runrev/48552/Screen_shot_2011-10-26_at_19.19.24_display.png?AWSAccessKeyId=AKIAJRW37ULKKSXWY73Q&Expires=1371560038&Signature=sKJHjrY91lWPJxw4gFv0FzmIFz0%3D
SingleURL=http://s3.amazonaws.com/screensteps_live/step_images/runrev/12161/media_1269946411890.png?AWSAccessKeyId=AKIAJRW37ULKKSXWY73Q&Expires=1371559845&Signature=o46g1j1N5EqY8JBujDfq68Y1nxE%3D
SingleURL=http://s3.amazonaws.com/screensteps_live/step_images/runrev/12161/media_1269945707622.png?AWSAccessKeyId=AKIAJRW37ULKKSXWY73Q&Expires=1371559845&Signature=qwjSGsIDgVLywZpadgFDyrKFoD0%3D
SingleURL=http://s3.amazonaws.com/screensteps_live/step_images/runrev/12161/media_1269947189330.png?AWSAccessKeyId=AKIAJRW37ULKKSXWY73Q&Expires=1371559845&Signature=RIJ8xvBLYIObNmEvJrWeRox%2FZ+g%3D
SingleURL=http://s3.amazonaws.com/screensteps_live/step_images/runrev/12161/media_1269947844357.png?AWSAccessKeyId=AKIAJRW37ULKKSXWY73Q&Expires=1371559845&Signature=S7U7vhgqSn0eE+aj8cVeYWCGlrc%3D
SingleURL=http://s3.amazonaws.com/screensteps_live/step_images/runrev/33065/MetaData1_display.png?AWSAccessKeyId=AKIAJRW37ULKKSXWY73Q&Expires=1371560038&Signature=dLbWZ+J+BUPg8Wj%2FdoEsTxAgfD8%3D
SingleURL=http://s3.amazonaws.com/screensteps_live/step_images/runrev/33065/Summary1_display.png?AWSAccessKeyId=AKIAJRW37ULKKSXWY73Q&Expires=1371560038&Signature=lmTWVRj5c5ZOqI5CD4yL9RulkG0%3D

Why are these lines listed there? and how did they got there? Is it as it should be?


Marek
Oleg Chernavin
06/19/2013 05:31 am
These SingleURL= lines are from using the AutoSave feature. It is OK.

I would suggest you to turn on logging and see what happens with files. Maybe there are error messages.

Oleg.
Marek
06/19/2013 06:16 am
Thanks Oleg,

I tried to find where to turn on the logging, but could not find it.
Once I turn it on should I use Download Missing Files, Ctrl+F5 ?

Regards,

Marek
Marek
06/19/2013 07:10 am
OK, I found where to start logging.
Will let you know what it shows soon.

Marek
Marek
06/19/2013 11:46 am
Hi Oleg,

So I've run the project again with logging on. Settings were Download only new and modified files and Skip existing media files.
77 files were downloaded, only 5-6 pages show images. I'm emailing you the log as attachment.

What do you suggest next?

Thanks.

Marek
Oleg Chernavin
06/19/2013 11:50 am
Marek,

What address did you use to E-mail the log?

Oleg.
Marek
06/19/2013 11:55 am
support@metaproducts.com

Is it OK? or should I use some other email address?

Marek
Oleg Chernavin
06/19/2013 12:02 pm
I just sent you an E-mail to the address you left in this forum.

Oleg.
Marek
06/21/2013 12:38 am
Hi Oleg,

Any news? Did you figure out from the log how to fix it?

Regards,

Marek
Oleg Chernavin
06/25/2013 05:52 am
Marek,

I am sorry for the late answer. It looks like these images have expiration time and if OE downloads them a few seconds later, they output an error.

Please try to add the following line to the Project's URLs field to make images load before HTML files:

Additional=DepthFirst

Would it help?

Oleg.
Marek
06/25/2013 07:09 am
Thanks Oleg,

I tried to run just a portion of these lessons with the URL http://lessons.runrev.com/m/2592/ and added the code you've mentioned (added it just below the URL - hope it's correct)

http://lessons.runrev.com/m/2592/
Additional=DepthFirst

I rerun the whole project and this time no images at all were retrieved.

Any other ideas how to fix it?

Marek
Oleg Chernavin
06/25/2013 07:25 am
I downloaded it with Level=1 and all images are there. What is your download speed? Maybe it loads too slowly and the images get expired too fast.

Oleg.
Oleg Chernavin
06/25/2013 07:25 am
My average speed on this download was 75 kb/second (right-click a Project, View - Statistics).

Oleg.
Marek
06/25/2013 07:44 am
I've run it again from scratch. No images at all.

The average download speed is 3.5kB/s
I'm using a mobile internet connection in India and mobile in India is far slower than what you have in the West.

The highest speeds that I've occasionally noticed here were 25 kB/s, usually it's about 6-12 kB/s

Marek


Marek
06/25/2013 07:46 am
Are there any settings I should change to adjust for that kind of low download speed?

Marek
Oleg Chernavin
06/25/2013 08:05 am
I think, this is the reason why it happens this way. Too slow download because of your provider.

I can only suggest to use the Browse With AutoSave feature to get the images semi-manually.

Oleg.
Marek
06/25/2013 11:56 am
I tried with AutoSave mode selected. The pages load properly in the internal browser, but neither the pages nor any images are saved.

What to do?

Marek
Oleg Chernavin
06/26/2013 05:15 am
Did you browse them OFFLINE using the AutoSave mode?

Oleg.
Marek
06/26/2013 11:33 am
No, I was connected to internet, browsing was online and that's why the pages and images did show up, but nothing was autosaved.

Marek
Oleg Chernavin
06/26/2013 02:12 pm
Yes, you need connect to Internet, select the partially downloaded Project, click the Browse With AutoSave button and start visiting the pages. The images should be loaded and saved automatically.

Oleg.
Marek
06/28/2013 06:56 am
Yes, the AutoSave works OK. But it's way less convenient than retrieving the website or parts of it and having it ready for browsing offline.

Marek
Oleg Chernavin
06/28/2013 08:30 am
Yes, it is less convenient, but it is the only way to download this site on a slow internet connection, because it has expired images.

Oleg.
Marek
06/30/2013 06:18 am
Thanks Oleg for all your help.

Regards,

Marek
Oleg Chernavin
07/01/2013 05:28 am
You are welcome!

Oleg.