Spiders

title says it all

Another German dvd spider

Postby tswhite70 on Sat Jul 28, 2007 6:56 pm

I got a chance to write a spider for buch.de. I think it's working correctly based on my "incredible" German language skills (aka Babefish). I did have to do a little something weird on this one. Apparently Xlobby sends 2 seperate http get request when spidering, the first does not include the %searchstring% variable, I'm guessing Xlobby uses this to verify the site is available before sending a second get request with the %searchstring% included. Buch.de returns a 404 error if you don't have a value for the search instead of the typicall HTTP 200/OK message. This causes Xlobby not to send the second request, bummer. I figured out that I could include a default "1" in the search and Buch.de would respond correctly to the first query and it doesn't seem to mess up the search in the testing I've done - of course I'm sure it will screw some search (ie searching for 10 - Bo Derek movie would probably get you 101 Dalmations instead...).

I've put the two German spiders in a single zip...
http://home.comcast.net/~twhite644/spid ... German.zip

good luck,
tsw
Last edited by tswhite70 on Tue Nov 06, 2007 5:37 pm, edited 1 time in total.
Image
tswhite70
 
Posts: 319
Joined: Tue Jan 06, 2004 3:44 pm
Location: Houston, Tx

Re: Spiders

Postby CaM on Mon Aug 20, 2007 8:52 am

hi there all, does anyone know why the amazon music spider is not currently working?

thanks

CaM
CaM
 
Posts: 97
Joined: Tue Apr 06, 2004 11:05 am
Location: NZ - living in France

Re: Spiders

Postby tswhite70 on Fri Aug 24, 2007 9:57 pm

Updated the Amazon.com music spider... (Pre-Calrad tested only)

http://home.comcast.net/~twhite644/spiders/UpdatedMusicSpiders.zip

Contents:
music - amazon.com
music - buy.com
music - cdbaby.com
music - cduniverse.com
music - elusivedisc.com
music - walmart.com

good luck,
tsw
Image
tswhite70
 
Posts: 319
Joined: Tue Jan 06, 2004 3:44 pm
Location: Houston, Tx

Updated DVD Spiders

Postby tswhite70 on Sun Dec 30, 2007 12:19 am

I finally had a chance to update some of my DVD spiders...

http://home.comcast.net/~twhite644/spid ... piders.zip

good luck,
tsw
Image
tswhite70
 
Posts: 319
Joined: Tue Jan 06, 2004 3:44 pm
Location: Houston, Tx

Re: Updated DVD Spiders

Postby Myth on Sat Jan 12, 2008 3:11 am

Is there a way to make the stills function work? It hasnt worked for quite a while (on my setup, anyway)
thanks
Myth
Myth
 
Posts: 83
Joined: Sun May 09, 2004 9:48 am

Re: Spiders

Postby tswhite70 on Mon Jan 14, 2008 6:53 pm

Myth - the answer is no, the stills download/spider is internal Xlobby code and since Steven is no longer providing support for the Pre-Calrad version of Xlobby we are out of luck.

On a happier note - I do have a Xscript script to pull stills for movies that I've been using for the past year or so. I'll be happy to upload it tonight, it pulls stills from movies.com and IMDB.com and is configurable for the # of stills you want.

I've also done a bunch of preliminary work on a Spider plugin to allow for some expanded functionality, most of the code is written I've just got to wrap it up as a plugin. I've got at least 3 things ahead of it though so it will definitely be a while.

I'll try and remember to upload the GetStills script tonight....

EDIT: I updloaded the script - see here
tsw
Image
tswhite70
 
Posts: 319
Joined: Tue Jan 06, 2004 3:44 pm
Location: Houston, Tx

Re: Spiders

Postby slaman on Tue Jan 29, 2008 5:34 pm

Thanks tswhite70...

I've been trying to tweak this spider to get it working for both single-line plot movies (Juno) and multi-line plot movies (American Gangster) but it doesn't seem to work!!! I get garbage characters for the multi-line plot movies...

Here's what I have... I can't combine single/multi-plot movies...


url=http://us.imdb.com/find?q=%searchstring%;tt=1
results=<a href="(?<url>/title/.*?/).*?">(?<display>.*?)</a>

//title & year
<div id="tn15title">.*?<h1>(?<title>.*?) <span>\(<a href="/Sections/Years/.*?">(?<year>.*?)</a>\)</span></h1>

//Directors
<h5>Director:</h5>(?<variable>.*?)<br/>
<a href="/name/.*?">(?<director>.*?)</a>

//Genres
<h5>Genre:</h5>(?<variable>.*?)</div>
<a href="/Sections/Genres/.*?/">(?<genre>.*?)</a>

//Rating
<h5><a href="/mpaa">MPAA</a>:</h5>.*?Rated (?<rating>.*?\.).*?</div>

//Runtime
<h5>Runtime:</h5>.*?(?<runtime>.*?)</div>

//Actors
<table class="cast">(?<variable>.*?)</tr></table>
<td class="hs"><a href="/name/.*?/">(?<actors>.*?)</a></td>

//Languages
<h5>Language:</h5>(?<variable>.*?)</div>
<a href="/Sections/Languages/.*?/">(?<language>.*?)</a>

//Full Plot
<h5>Plot Outline:</h5>.*?<a class="tn15more inline" href="(?<url>/title/tt.*?/plotsummary)" onClick=.*?more</a>
<p class="plotpar">(?<plot>.*?)<i>

//Coverart
<a name="poster" href="photogallery" title=".*?"><img border="0" alt=".*?" title=".*?" src="(?<coverart>.*?)".*?></a>

replace=coverart:m.jpg:f.jpg
slaman
 
Posts: 147
Joined: Sat Oct 14, 2006 10:30 pm

Re: Spiders

Postby tswhite70 on Wed Jan 30, 2008 10:44 pm

slaman - I took a look at my IMDB spider and there was a problem with American Gangster, although it turned out to be a result of the actors regex and not the plot. I've updated my zip file with new IMDB and IMDB_Direct spiders and also fixed the Ratings regex for Amazon.com. I tested them on the movies you mentioned plus a couple others and they seem to working.

The answer to your question about single vs. mutiline plots is that you have to have 2 separate plot regexes, singlel-line first then the multi-line second. Xlobby will always do the single-line plot, but it will be replaced by the multi-line if it is available - see the IMDB spider in the zip.

http://home.comcast.net/~twhite644/spid ... piders.zip

good luck,
tsw
Image
tswhite70
 
Posts: 319
Joined: Tue Jan 06, 2004 3:44 pm
Location: Houston, Tx

Re: Spiders

Postby slaman on Sun Feb 03, 2008 6:38 am

tswhite70 - thanks, but it didn't work... The actors regexp is still broke (get extra lines) and the "plot" variable doesn't get replaced, it gets appended to.

Perhaps it's because of Vista, because when I was running XP, this spider was fine?
slaman
 
Posts: 147
Joined: Sat Oct 14, 2006 10:30 pm

Re: Spiders

Postby tswhite70 on Tue Feb 05, 2008 7:44 pm

I don't have a Vista system to test on, but it's possible that your .Net version (3.0 or 3.5?) changed something in the regex functions that is causing the different result. You could try testing the regex with Expresso on your Vista machine and see if you can figure it out. Sorry I couldn't help.

good luck,
tsw
Image
tswhite70
 
Posts: 319
Joined: Tue Jan 06, 2004 3:44 pm
Location: Houston, Tx

Re: Spiders

Postby slaman on Wed Feb 06, 2008 7:16 pm

Will test it out right now - thanks for the link - was looking for a good test application. Re-starting XLobby after each spider change to "test" was getting annoying...
slaman
 
Posts: 147
Joined: Sat Oct 14, 2006 10:30 pm

Re: Spiders

Postby slaman on Mon Feb 11, 2008 12:32 am

Weird - doesn't replace the variable "plot"
tswhite, wnated to ask you, since you seem to be the guru of spiders here... how do I get the proper "fancy" characters, like french or german letters... I get weird characters when those things appear... an dI have to manually paste them in.
slaman
 
Posts: 147
Joined: Sat Oct 14, 2006 10:30 pm

Re: Spiders

Postby homepc on Mon Feb 11, 2008 3:27 pm

I have always used the F2 configuration to use spiders for my movies. This weekend I finally clicked on the spider button on my movieviewer overlay screen. Up pops this spider screen. (If I knew how to attach picture I would show you). At the top of the screen, I have three slections, Direct, WEB, ABC. When I click direct, nothing happens. When I click on WEB, it fires up the spider scripts and show coverart and info to choose for the movie. I looked at the event for these buttons and they are looking at "spider:spider search direct movies" for the direct, and "spider:spider search direct movies" for the WEB button. Can anyone tell me if these events are looking at the same spider scripts I use when I go directly to the F2 configuation screens and select spiders?

Thanks
homepc
 
Posts: 251
Joined: Tue Mar 20, 2007 1:27 pm
Location: Delaware

Re: Spiders

Postby tswhite70 on Mon Feb 11, 2008 6:38 pm

slaman - As for the non-english characters there's not a way around the problem that I am aware of. It would have to be coded in Xlobby to translate those characters into the correct syntax.

homepc - Are you using the pay version of Xlobby?

Seems like I've said this before, but I'm a free-lobby user. Any spiders I publish are designed to be used with the Pre-Calrad version of XL. They may work with the current pay-version, but I don't have a way to troubleshoot any issues. Sorry I can't be of more help.

good luck,
tsw
Image
tswhite70
 
Posts: 319
Joined: Tue Jan 06, 2004 3:44 pm
Location: Houston, Tx

Re: Spiders

Postby homepc on Mon Feb 11, 2008 7:17 pm

tswhite70, I am a free loader! haahaa I was just curious if the spider skin that was created, used a spider that was internal to XL or used the spiders that you see when you go directly in the configuration screens. does the event "spiders:spider search web movies" use an internal spider?

thanks
Ray
homepc
 
Posts: 251
Joined: Tue Mar 20, 2007 1:27 pm
Location: Delaware

PreviousNext

Return to Tips and Tricks

Who is online

Users browsing this forum: No registered users and 1 guest

cron