How to download search results at www.mobile.de?

Author Message
Matthias Meisner 09/11/2004 12:46 pm
Hi there!

I`m trying to download search results (that is, survey and detailed pages) at www.mobile.de (for instance, search results for "Audi Allroad").

I`ve entered the following parameters with regard to "URL Filters":

- "Load only the starting protocol"
- "Load files only within the starting Server"
- "Custom directories configuration" selected; "http://www.mobile.de/sid*/cgi-bin/" entered within the input field of "View included directories keywords"
- "Custom filenames configuration" selected; keywords "da.pl" and "searchpublic.pl" entered within the input field of "View included files keywords"

The result is that OE downloads only one page (i.e. the starting survey page of the search).

What did I wrong? Or isn`t OE able to download the requested pages?

I will highly appreciate if someone can tell me the solution.


Yours sincerely,

Matthias Meisner
09/13/2004 03:19 am
> Or isn`t OE able to download the requested pages?

OEP can download the requested pages. But in most cases you will get multiple identical pages (double, triple, ... pages, the more resultpages your searchresult offers). This is caused by the SID in the directory structure:
www.mobile.de/SID*/*

Perhaps there is a way to avoid this. I didn`t have luck with my first tests with URL Substitutes, Cookies ...
I wonder if Oleg can find a way to easily avoid the multiple (nearly) identical page downloads?
@ Oleg: A good starting point for tests would be a search request with 2 or 3 resultpages, i.e.:
Model: Porsche

or try:
Marke: Ligier

But let`s get back to your question, Matthias.

Try the following to search for your PKW:

Browse: www.mobile.de
Click on "Suchen"
Modell: Audi Allroad
Hit CTRL+ALT when clicking on "Suche starten"

OEP will create a new Project, something like:

http://www.mobile.de/cgi-bin/searchPublic.pl?_form=search&sr_make=-2&sr_model=Audi+Allroad&sr_priceFrom=-2&sr_priceTo=-2&sr_category=1100&sr_powerRange=-2&sr_registrationDateFrom=-2&sr_registrationDateTo=-2&sr_mileageFrom=-2&sr_mileageTo=-2&sr_engineType=-2&sr_doorCount=-2&sr_color=-2&sr_country=-2&sr_zip=&sr_zipRadiusTo=-2&sr_damaged=0&sr_daysOldTo=-2&sr_sortOrder=0&doSearch.x=54&doSearch.y=11
Referer=http://www.mobile.de/cgi-bin/searchPublic.pl?bereich=pkw&sprache=1

(I`ve cut out the SID-directory: http://www.mobile.de/SID*/ )

This is the starting URL.


The other settings:

No "Level limit"


File Filters:

Text:
Load using URL filters settings

Images:
Add the following keyword to the Extensions list: bild
Load from any site

User defined:
Load only from the starting server

Other:
Load using URL filters settings


URL filters:

Protocol:
Load all protocols

Server:
Load files only within the starting Server

Directory:
Load files from all directories

Filename:
Custom filenames configuration

View included files keywords:
^displayaddpics.pl*
^da.pl*
http://www.mobile.de/sid*/cgi-bin/searchpublic.pl?bereich=pkw&top=21&
http://www.mobile.de/sid*/cgi-bin/searchpublic.pl?bereich=pkw&top=41&
http://www.mobile.de/sid*/cgi-bin/searchpublic.pl?bereich=pkw&top=61&
http://www.mobile.de/sid*/cgi-bin/searchpublic.pl?bereich=pkw&top=81&
http://www.mobile.de/sid*/cgi-bin/searchpublic.pl?bereich=pkw&top=101&

-----------------------------------

I won`t recommend to download a searchresult with more than 5 resultpages (120 PKW`s), because you will get a lot of junk (multiple nearly identical pages with different SID`s).

(Note: Some links won`t be useable offline; i.e. use BACK and FORWARD instead)

I would also be glad if Oleg finds a better setting, without getting those junk pages... ;-)
I`m sure that there are a lot more sites around with those /SID*/ directories.
Oleg Chernavin 09/13/2004 07:40 am
Thank you for helping!

I think that the only way to get rid of the SID* directory is to use URL Substitutes.

Best regards,
Oleg Chernavin
MP Staff
09/13/2004 04:39 pm
> Thank you for helping!

No problem, although the settings aren`t offering really good results (duplicates)...

> I think that the only way to get rid of the SID* directory is to use URL Substitutes.

Just FYI:
Most files are stored on disk as "sid..." under the topmost directory www.mobile.de.
It`s easy to avoid the download of links with other SIDs than in the starting URL or cut off completely the SID-directory. But then some files would be missing. The download process on the site produces many different temporary SID-directories (www.mobile.de/SID*/*). I can`t find a way to download the search survey completely with only one SID.

But there is another issue:
The Export process doesn`t convert links like the following correctly:

Stored on disk (before and after the Export process):
javascript:Morepics(`SIDwSWumu8ER.30t2IOvUIfJA-t-vaN_25E3DexlCsAsCsK_25F3P_25F3R~BmSB10Bda_2581_257C-t-vCaCiCoMkMoPESeTUZ020C5780DD`);

The original URL was something like:
javascript:Morepics(`../../cgi-bin/displayAddPics.pl?bereich=pkw&DatasetNumber=11111111140167580`);

The only way you will be able to see the additional pictures (Details ansehen... Weitere Bilder (above the car image):
Filenames format: Keep as-is
Uncheck "Use standard extensions for files with no file types"
Oleg Chernavin 09/14/2004 06:20 am
Yes, I am afraid, it is almost impossible to convert such scripted links correctly. Sorry.

Oleg.
Matthias Meisner 09/16/2004 02:17 pm
Hallo Anonymous and Oleg!

Thanks for your answers.

Hallo Anonymous, I tested the configuration described in your first reply, and it worked on principal (but there are many junk pages, as you mentioned).

I`ll put the topic how to possibly suppress the download of these irrelevant pages in another question.

Matthias