Spiders

title says it all

Re: Spiders

Postby mememe on Fri May 30, 2008 10:40 pm

I posted this earlier:

viewtopic.php?f=6&t=6390

That was before I found this thread. Tips and Tricks seems like an odd area for a spider development thread. Anyway, I'm guessing this is where I should have posted it. I would appreciate any help.
mememe
 
Posts: 76
Joined: Thu May 29, 2008 12:23 am

Re: Spiders

Postby cinOxen on Sun Jun 01, 2008 7:04 pm

mememe wrote:I posted this earlier:

http://xlobby.com/forum/viewtopic.php?f=6&t=6390

That was before I found this thread. Tips and Tricks seems like an odd area for a spider development thread. Anyway, I'm guessing this is where I should have posted it. I would appreciate any help.


TVMan does what you want only with tv.com. It is a great utility.
cinOxen
 
Posts: 53
Joined: Thu Sep 14, 2006 6:19 pm

Re: Spiders

Postby cinOxen on Sat Jun 07, 2008 6:23 pm

lar282 wrote:I do think we have a bug then. I mean if it works under F2 but not in the skin we're in trouble. Wes can u confirm also and maybe hand over to Steven?


//Lasse


I've been trying to get a spider function in my skin and have been looking at the default skin for guidance. When the skin brings up the spider overlay is sets the spider as "dvd*" which means all the dvd spiders are loaded. I had the same problem as you though that the working imdb spider wasn't coming up. In fact it seems only spider results with coverart are displayed. If you set the skin to only load the imdb spider there are no results at all.

This does seem like sort of a bug. I have no need to find the coverart through the skin as I find it easy to add myself when ripping my movie (which is done at the office computer). It's more of a hassle to load the plot and other info because I have to remotely connect to the htpc or pull out the keyboard and mouse to hit F2 in the living room.

I've looked through some other skins as well and it seems this behaviour is typical.

Edit: Just want to say I don't know if my assumption was right about only spider results with coverart being displayed through the skin. I added coverart to my imdb spider and it still didn't come up but if I ensure there are two results it does. So I'm not sure if the spider needs a coverart entry to be added to the spider category or if it simply needs more than one entry to work properly but my guess is it's both.
cinOxen
 
Posts: 53
Joined: Thu Sep 14, 2006 6:19 pm

Re: Spiders

Postby slaman on Mon Jun 09, 2008 11:46 pm

My IMDB spider stopped working for a bit, but now it's back... changed the first two lines to this:

url=http://us.imdb.com/find?s=all&q=%searchstring%
results=<a href="(?<url>/title/.*?/).*?">(?<display>.*?)</a>
slaman
 
Posts: 147
Joined: Sat Oct 14, 2006 10:30 pm

Re: Spiders

Postby lar282 on Tue Jun 10, 2008 1:37 pm

slaman wrote:My IMDB spider stopped working for a bit, but now it's back... changed the first two lines to this:

url=http://us.imdb.com/find?s=all&q=%searchstring%
results=<a href="(?<url>/title/.*?/).*?">(?<display>.*?)</a>



and it works both in the skin and in the F2 database section?

//Lasse
Image
lar282
 
Posts: 1682
Joined: Thu Apr 01, 2004 4:13 pm
Location: Helsingborg, Sweden

Re: Spiders

Postby slaman on Tue Jun 24, 2008 9:00 pm

lar282 wrote:
slaman wrote:My IMDB spider stopped working for a bit, but now it's back... changed the first two lines to this:

url=http://us.imdb.com/find?s=all&q=%searchstring%
results=<a href="(?<url>/title/.*?/).*?">(?<display>.*?)</a>



and it works both in the skin and in the F2 database section?

//Lasse


I only use F2... I guess I should use the skin to add new items...
slaman
 
Posts: 147
Joined: Sat Oct 14, 2006 10:30 pm

Yummm - Nicecovers.com

Postby tswhite70 on Sun Jul 27, 2008 4:09 am

Just saw this tonight and couldn't help myself... database is probably small right now although it did have covers for the half dozen movies I tried. Plus an added bonus available as PNG dvd case images, if you like that sort a thing. I uploaded my whole DVD spiders collection as most of them have been updated since my last upload. New spider is "dvd - nicecovers.com - coverartonly.txt". It's set to get the raw cover in JPG format by default, follow directions in the spider to get the PNG dvd case image.

http://home.comcast.net/~twhite644/spid ... piders.zip

good luck,
tsw
Image
tswhite70
 
Posts: 319
Joined: Tue Jan 06, 2004 3:44 pm
Location: Houston, Tx

Re: Spiders

Postby beingblueeyes on Thu Aug 28, 2008 4:12 pm

Is it possible to make a movie spider for cover art and info from the Netflix site? I ask because they have a huge library of movies all with cover art and info.
beingblueeyes
 
Posts: 5
Joined: Sun Aug 24, 2008 3:15 am

Re: Learning How to Build Spiders

Postby beingblueeyes on Sat Aug 30, 2008 2:06 am

tswhite70 wrote:
Now we need to get the source from the search url and design our regex for the "Results=" line of the spider. So run a search of DVDtown for "Troy" and choose "View\Source" in IE to see the source code for the URL. Holding the mouse pointer over the details links in the search results web page shows us that we are looking for a link with the words "discdetails"

is it still posable to make a spider for a page (netflix) even if there is no "discdetails" part of the surch?
beingblueeyes
 
Posts: 5
Joined: Sun Aug 24, 2008 3:15 am

Re: Spiders

Postby tswhite70 on Sat Sep 20, 2008 7:27 pm

Yes, it's possible to create a spider for Netflix. Just follow the instructions on my earlier post or check out one of the other spiders. There is a disc details page for every movie on Netflix - it's the page you go to when you click on the movie title after a search.

The reason no one has bothered writing a netflix spider is because the coverart image quality sucks and the info is nothing special - but don't let that stop you! If you are a netflix user check out the netflix plugin for managing your account (that will give you a good idea of how bad the coverart is)...

good luck,
tsw
Image
tswhite70
 
Posts: 319
Joined: Tue Jan 06, 2004 3:44 pm
Location: Houston, Tx

Re: Spiders

Postby slaman on Tue Jan 13, 2009 4:29 am

Hey tswhite70 - you're the king of the spiders... can you fix the impawards.com one please? That's the only site that has decent coverart!
slaman
 
Posts: 147
Joined: Sat Oct 14, 2006 10:30 pm

Re: Spiders

Postby sven37 on Wed Apr 15, 2009 9:51 pm

Hey I was wondering if someone could help me with a spider. I can't figure it out. I got a database of drinks from billbert and I'm trying to get a spider for images. http://www.drinkswap.com has pictures for every drink. as far as the code from the search page goes...

Code: Select all
 <table width="100%" align="center">
                                      <tr>
                                         <td valign="top" height="30" width="30">
                               
                                              <a href="../drinks/detail.asp?recipe_id=7469"><img src="../images/da/tequila_sunrise.jpg" height="60" border="0"></a>
                            
                                         </td>
                                         <td colspan="3" valign="middle" style="padding:10px;">
                                            <a href="../drinks/detail.asp?recipe_id=7469">Triple Sec Sunrise</a>

                                         </td>
                                      </tr>
                                   </table>


As for what I have thus far for regex.....
Code: Select all
url=http://www.drinkswap.com/search-for-drinks/index.asp?searchfor=%searchstring%
results=<a href="../drinks/detail.asp\?recipe_id=.*?"><img src="../images/da/.*?.jpg" .*?></a></td><td .*?><a

href="../drinks/detail.asp\?recipe_id=.*?">(?<display>)</a>


I really don't know what I am doing. I'm trying to figure it out, I've been running it through expresso and no luck.

Helps???
sven37
 
Posts: 8
Joined: Sun Apr 12, 2009 6:00 am

Re: Spiders

Postby sven37 on Mon Apr 20, 2009 4:50 am

So I'm still looking for some help. I changed my code a bit. Here is what it is so far.
Code: Select all
url=http://www.drinkswap.com/search-for-drinks/index.asp?searchfor=%searchstring%
results=<a href="..(?<url>http://www.drinkswap.com/drinks/detail.asp\?recipe_id=.*?)"><img src=".*?".*?></a></td><td colspan="3" valign="middle" style="padding:10px;"><a href="../drinks/detail.asp\?recipe_id=.*?">(?<display>.*?)</a>


The source is still the same as above. That is a section from the source that the relevant data is from.

Anyone?
sven37
 
Posts: 8
Joined: Sun Apr 12, 2009 6:00 am

Re: Spiders

Postby sven37 on Wed Apr 29, 2009 3:22 am

<bump>
sven37
 
Posts: 8
Joined: Sun Apr 12, 2009 6:00 am

Re: Spiders

Postby HDFan on Tue Nov 10, 2009 12:53 am

Hey I'm trying to create a spider for blu-ray.com but with little success. Is there anyone that can help with my code. Thanks
HDFan
 
Posts: 1
Joined: Tue Nov 10, 2009 12:24 am

Previous

Return to Tips and Tricks

Who is online

Users browsing this forum: No registered users and 1 guest

cron