Spiders

[mememe]

I posted this earlier:

viewtopic.php?f=6&t=6390

That was before I found this thread. Tips and Tricks seems like an odd area for a spider development thread. Anyway, I'm guessing this is where I should have posted it. I would appreciate any help.

[cinOxen]

I posted this earlier:

http://xlobby.com/forum/viewtopic.php?f=6&t=6390

That was before I found this thread. Tips and Tricks seems like an odd area for a spider development thread. Anyway, I'm guessing this is where I should have posted it. I would appreciate any help.

TVMan does what you want only with tv.com. It is a great utility.

[cinOxen]

I do think we have a bug then. I mean if it works under F2 but not in the skin we're in trouble. Wes can u confirm also and maybe hand over to Steven?


//Lasse

I've been trying to get a spider function in my skin and have been looking at the default skin for guidance. When the skin brings up the spider overlay is sets the spider as "dvd*" which means all the dvd spiders are loaded. I had the same problem as you though that the working imdb spider wasn't coming up. In fact it seems only spider results with coverart are displayed. If you set the skin to only load the imdb spider there are no results at all.

This does seem like sort of a bug. I have no need to find the coverart through the skin as I find it easy to add myself when ripping my movie (which is done at the office computer). It's more of a hassle to load the plot and other info because I have to remotely connect to the htpc or pull out the keyboard and mouse to hit F2 in the living room.

I've looked through some other skins as well and it seems this behaviour is typical.

Edit: Just want to say I don't know if my assumption was right about only spider results with coverart being displayed through the skin. I added coverart to my imdb spider and it still didn't come up but if I ensure there are two results it does. So I'm not sure if the spider needs a coverart entry to be added to the spider category or if it simply needs more than one entry to work properly but my guess is it's both.

[slaman]

My IMDB spider stopped working for a bit, but now it's back... changed the first two lines to this:

url=http://us.imdb.com/find?s=all&q=%searchstring%
results=<a href="(?<url>/title/.*?/).*?">(?<display>.*?)</a>

[lar282]

My IMDB spider stopped working for a bit, but now it's back... changed the first two lines to this:

url=http://us.imdb.com/find?s=all&q=%searchstring%
results=<a href="(?<url>/title/.*?/).*?">(?<display>.*?)</a>

and it works both in the skin and in the F2 database section?

//Lasse

[slaman]

My IMDB spider stopped working for a bit, but now it's back... changed the first two lines to this:

url=http://us.imdb.com/find?s=all&q=%searchstring%
results=<a href="(?<url>/title/.*?/).*?">(?<display>.*?)</a>

and it works both in the skin and in the F2 database section?

//Lasse

I only use F2... I guess I should use the skin to add new items...

[tswhite70]

Just saw this tonight and couldn't help myself... database is probably small right now although it did have covers for the half dozen movies I tried. Plus an added bonus available as PNG dvd case images, if you like that sort a thing. I uploaded my whole DVD spiders collection as most of them have been updated since my last upload. New spider is "dvd - nicecovers.com - coverartonly.txt". It's set to get the raw cover in JPG format by default, follow directions in the spider to get the PNG dvd case image.

http://home.comcast.net/~twhite644/spid ... piders.zip

good luck,
tsw

[beingblueeyes]

Is it possible to make a movie spider for cover art and info from the Netflix site? I ask because they have a huge library of movies all with cover art and info.

[beingblueeyes]

Now we need to get the source from the search url and design our regex for the "Results=" line of the spider. So run a search of DVDtown for "Troy" and choose "View\Source" in IE to see the source code for the URL. Holding the mouse pointer over the details links in the search results web page shows us that we are looking for a link with the words "discdetails"

is it still posable to make a spider for a page (netflix) even if there is no "discdetails" part of the surch?

[tswhite70]

Yes, it's possible to create a spider for Netflix. Just follow the instructions on my earlier post or check out one of the other spiders. There is a disc details page for every movie on Netflix - it's the page you go to when you click on the movie title after a search.

The reason no one has bothered writing a netflix spider is because the coverart image quality sucks and the info is nothing special - but don't let that stop you! If you are a netflix user check out the netflix plugin for managing your account (that will give you a good idea of how bad the coverart is)...

good luck,
tsw

[slaman]

Hey tswhite70 - you're the king of the spiders... can you fix the impawards.com one please? That's the only site that has decent coverart!

[sven37]

Hey I was wondering if someone could help me with a spider. I can't figure it out. I got a database of drinks from billbert and I'm trying to get a spider for images. http://www.drinkswap.com has pictures for every drink. as far as the code from the search page goes...

Code: Select all

 <table width="100%" align="center">
                                      <tr>
                                         <td valign="top" height="30" width="30">
                               
                                              <a href="../drinks/detail.asp?recipe_id=7469"><img src="../images/da/tequila_sunrise.jpg" height="60" border="0"></a>
                            
                                         </td>
                                         <td colspan="3" valign="middle" style="padding:10px;">
                                            <a href="../drinks/detail.asp?recipe_id=7469">Triple Sec Sunrise</a>

                                         </td>
                                      </tr>
                                   </table>


As for what I have thus far for regex.....

Code: Select all

url=http://www.drinkswap.com/search-for-drinks/index.asp?searchfor=%searchstring%
results=<a href="../drinks/detail.asp\?recipe_id=.*?"><img src="../images/da/.*?.jpg" .*?></a></td><td .*?><a

href="../drinks/detail.asp\?recipe_id=.*?">(?<display>)</a>

I really don't know what I am doing. I'm trying to figure it out, I've been running it through expresso and no luck.

Helps???

[sven37]

So I'm still looking for some help. I changed my code a bit. Here is what it is so far.

Code: Select all

url=http://www.drinkswap.com/search-for-drinks/index.asp?searchfor=%searchstring%
results=<a href="..(?<url>http://www.drinkswap.com/drinks/detail.asp\?recipe_id=.*?)"><img src=".*?".*?></a></td><td colspan="3" valign="middle" style="padding:10px;"><a href="../drinks/detail.asp\?recipe_id=.*?">(?<display>.*?)</a>

The source is still the same as above. That is a section from the source that the relevant data is from.

Anyone?

[sven37]

<bump>

[HDFan]

Hey I'm trying to create a spider for blu-ray.com but with little success. Is there anyone that can help with my code. Thanks