Media Server Setup

Speak your mind

Media Server Setup

Postby Groover on Sat Nov 27, 2004 1:37 am

Hey there,

Anyone running a media server utilising several Terabytes of space to run their DVD collection/TV recordings?

I was reading a post "Who Uses their PVR" when I spied a reply by kolemieux (who seems to run such a system). I was wondering (if you read this post) (or anyone else) if you could describe how your media server is set up and what components/software you use - hardware configs etc. Do you use a RAID (type?)

I have been researching media servers for some time but companies like Kaleidescape or Axonix MediaMax are way too expense for any sane person and I'm sure one could procure a reasonable setup for a lot less using a stand-alone PC. (Although I am sure it will still cost thousands of dollars)

One of my main desires is to have redundancy on the system, so if a HD fails it is easy to replace the lost info. (I've already had 6 DVD's die for no apparent reason and want to protect my investment).

What are your thoughts?

Thanks in advance for your replies.

Groover
Groover
 
Posts: 14
Joined: Sat Nov 27, 2004 1:09 am

Postby BaddaBing on Sat Nov 27, 2004 5:38 am

I'm currently running a RAID 5 array utilizing a HighPoint Technologies Rocket Raid 454 controller card and 8 WD Caviar RAID Edition Series 250GB 7200RPM EIDE Hard Drives under Windows XP Pro. Once you take parity into account that leaves around 1.8 terabytes for use.

Yes, it is significantly less expensive than the $25k+ that Kaleidescape asks for their product and no it is not an inexpensive option. Between the drives, the controller card, motherboard, cpu, case, power supply (500 watts+), memory, etc., you are looking at about $2500.00. Although it can get significantly higher if you want to eliminate any single point of failure and utilize redundant power supplies, UPS and/or add hot swap capability.

Also keep in mind where the array will live. Drives put out heat - so keeping it cool AND quite will also increase the cost - otherwise plan on sticking it in a closet or basement where the drive and fan noise won't drive you crazy. Did I mention that creating an array (or rebuilding one afer replacing a drive) can take 12 to 15 hours?

And FYI, RAID 5 is not totally redundant. It can survive having one drive 'die', but if two go - and its not unheard of - then all data is lost. The main advantage to having your media on hard drives is the ease of managment and the speed at which you can access it. (Oh yeah, and its great for impressing your geeky tech friends :lol: )

Another option to consider for movie storage is DVD changers. Sony sells a 400 disk DVD changer for about $300 here in the states. You can control it using Xlobby and a USB-UIRT. There is a little bit more time lag in launching a movie, but I have never had one of my changers 'eat' a DVD and it is a lot less expensive in the long run.

I use a combination of the two - RAID and CD changers. The CD changers have the bulk of my movie collection while the RAID array has mp3's and the movies that are watched most by my family (kid's movies, comedies and concerts)

Here is a shot of my media server and of my equipment rack with the changers and the media server in the lower right corner:

Image

Image
Last edited by BaddaBing on Mon Nov 29, 2004 9:43 pm, edited 1 time in total.
BaddaBing
 
Posts: 557
Joined: Fri Mar 26, 2004 2:39 pm
Location: DFW Texas

Postby AirPost on Sat Nov 27, 2004 9:31 am

Another option would be to use external drives. I have firewire drives daisychained to my HTPC but like all alternatives, there are pros and cons to this.

A few of the pros are that it is relatively idiot proof and easy to add drives as needed, inexpensive compared to others (you can also buy just the external case then you just add your harddrives that are laying around), fast access times and also, you can take it with you to your friends house, plug it in to his/her pc and watch/listen media.

And the cons: It is not redundant so you may lose your files unless you do it through software. There are only so much drive letters the OS can assign to the drives not unless you use a third party software. All in all though, I haven't had any problems with this setup except once in awhile, Microsh*t won't recognize the drives on some reboots. In this case, I just restart the pc or disable/enable the firewire card.

There are also different connection options you have with external storage like usb and rj45. I do have a scsi raid setup in my linux box which acts as a domain controller but for my media, I just stick with the external option. Good luck on whatever option you take. :wink:
AirPost
 
Posts: 160
Joined: Sat Mar 27, 2004 6:37 pm
Location: Sin City

Postby Groover on Sun Nov 28, 2004 1:07 am

Thanks for the replies -

BaddaBing - (impressive looking setup by BTW) Could you explain what Parity is? I've read this term numerous times, but don't fully understand the principle - especially as I've never had any expereince with any type of RAID setup. Also, why does it take so long to set a RAID up or replace a drive? How often would a drive fail and what is the likelihood of two drives failing at the same time? As you may have guessed, I'm no computer geek - maybe a wannabe one :D .

I have considered DVD changers but have dismissed them for several reasons. First, they may only cost $300 in the US, but here in NZ they are considerably more expensive. There is also the issue of region coding (I have about half and half R4 and R1) When I looked into A DVD changer last time, the best option would be to run one on R4 and the other on R1. Still an option I guess, but less so when my collection surrpasses 400 titles in one region. The other reason, apart from the reality of lag times in loading and the way certain DVD's are formatted, I'm getting increasingly annoyed at the way studios are presently their discs - especially with advertsing at the front of them which one can't ff through or jump straight to the menu, etc, etc, etc. To me, a HD based system seems more user-centiric and I can set it up in a way that appeals to me.

One major concern for me lately is the unexplained failure of DVD's without excessive use - as I've said, I've had 6 die - many of them have only been played once or twice. The retailers can't offer any explanations and I've heard conflicting stories from the local studio branches or distributors. With a growing collection that has cost $16000NZ it makes me a little nerous when I pull a title out as to whether it's going to fail or not. This DVD rot scenario is something that is NOT being addressed by the studios. Do I have to replace a title every third use? I worry about future DVD designs that utilise multiple layers, as the failure in mine has always been after the layer switch. This failure rate has made me want backup potential on whatever type of media server I decide to go with - but then it seems to reach a moot point when backup costs more than just replacing the discs that die - and backing up on to a blank DVD is seriously expensive here as well - may as well just buy another studio released one.

I did plan to setup the system in the basement/another room. Better for many reasons.

AirPost - I'm currently leaning in your direction - using external firewire drives. Seems easier to implement from my perspective and a lot easier to maintain from a non-geek POV. I was considering Hitachi's 400 Gb drive, but it costs a fortune here. The most cost effective unit I've found locally is LaCie's 1TB unit for $1800NZ. I figure 4 or 5 of those should set me up with enough space to grow and contain movies, music and TV. Is USB 2.0 any better/worse than firewire 800? Why would I need to hot swap drives and can't this be done already with a firewire drive? What is the story with drefrag on such systems?

I've have also been considering re-formatting all my DVD's to MPEG4 to save on space, but am uncertain if one can get as good a video as MPEG2. Do either of you have any expereine with this? How much of a saving would one achieve on say a 6Gb movie?

One more question before I fang - does backup software (like the one that comes with XP) compress the VOB files significantly to save heaps of space (are there any backup software suggestions), or are VOB's already heavily compressed?

Thanks again - Groover
Groover
 
Posts: 14
Joined: Sat Nov 27, 2004 1:09 am

Postby BaddaBing on Sun Nov 28, 2004 11:05 pm

Groover,
I'm not a RAID expert either, know just enought to get what I want working. Here is a great site for more info: http://commodore.ca/windows/raid5/raid5.htm It has the answers your questions.
BaddaBing
 
Posts: 557
Joined: Fri Mar 26, 2004 2:39 pm
Location: DFW Texas

Postby Groover on Mon Nov 29, 2004 5:16 am

BaddaBing wrote:Groover,
I'm not a RAID expert either, know just enought to get what I want working. Here is a great site for more info: http://commodore.ca/windows/raid5/raid5.htm It has the answers your questions.


Thanks for the link. Have sussed it out and it seems to make more sense now. RAID 5 seems like a good option - depending on how much storage I actually need. Will keep up the media server investigation.
Groover
 
Posts: 14
Joined: Sat Nov 27, 2004 1:09 am

Postby hjackson on Mon Nov 29, 2004 7:25 am

I have several external drives attached to a true dedicated 200 celeron server. I have just over 2 TB of storage, and no parity. My problem with using a RAID system is that you essentially cannot expand it once you've determined it's size (someone please correct me if this limitation no longer exists). As well, all the drives must be configured to hold the same storage capacity, so if you have drives of different sizes, they will all have to be configured to the size of the smallest drive. Also, you lose about one drive worth of storage for this parity. And last, after all is said and done, you can still crash a RAID server and lose all your stuff!!!
Currently, I would prefer several 200-300 GB drives to one massive 1 TB drive (whenever that becomes available) because I think the chance of all 5-6 drives crashing (as opposed to just the one jumbo drive) would be very slim.
Good Luck!

hjackson
hjackson
 
Posts: 371
Joined: Sat Nov 29, 2003 7:12 am
Location: Tampa, Florida

Postby chadwolcott on Mon Nov 29, 2004 4:08 pm

I'll weigh in on this as I have just built a media server myself.

My problem with using a RAID system is that you essentially cannot expand it once you've determined it's size (someone please correct me if this limitation no longer exists).


You can in fact expand arrays, but you need to have a relatively high-end card. LSI and RAIDCore cards support OCE (Online Capacity Expansion) and 3ware is soon to release a firmware upgrade to add this ability to their cards. I have a 3ware 9500S-12 which is a 12-port SATA RAID card. I have 10 250GB WD RAID edition drives connected with 7 in a RAID-5 (1 hot spare) and 3 more awaiting the OCE upgrade to add them to my 1.3TB array bringing it to 2TB. I have 2 Maxtor SATAII 300GB drives in a RAID-0 on the other 2 channels for recording TV from my 2 PVR-250s on the other 2 channels. Another 2 of the maxtors on my motherboards on-board SATA RAID in RAID-1 for the 120GB of CDs I have ripped.

It is possible to lose 2 drives in the array simultaneously, or at least lose the second drive before the controller has time to rebuild the array with the hot spare, but it is a small chance. If the server does crash the array will still live on, even if the controller goes, replacing the controller will allow rebuilding the array from the existing data.

Here is what it looks like, it's the server at the bottom, below the UPS. The external DVD Drive is connected to it for ripping the 250+ DVDs that I've done. 15 HDs total, with a P4 Prescott and 1GB RAM and dual gigabit NICs 3.5TB total drive capacity.

Image
chadwolcott
 
Posts: 107
Joined: Fri Sep 05, 2003 2:34 pm
Location: Boston, MA

Postby BaddaBing on Mon Nov 29, 2004 4:45 pm

Everything Chadwalcott posted is right, but let me offer a small clarification for RAID newbies:

even if the controller goes, replacing the controller will allow rebuilding the array from the existing data.


This works as long as it is the same make and model card. RAID configurations tend to be propriatry solutions, so if you if your LSI card dies - you need to replace it with an LSI card. So make sure you get the card with the features you want the first time. Otherwise you need some place to back up that Terabyte of data before you can create an array with that new RAID card.
Last edited by BaddaBing on Mon Nov 29, 2004 6:47 pm, edited 1 time in total.
BaddaBing
 
Posts: 557
Joined: Fri Mar 26, 2004 2:39 pm
Location: DFW Texas

Postby gamejester on Mon Nov 29, 2004 5:36 pm

I have the same problem here in the UK, half my discs are region 1 and half region 2.

I have considered DVD changers but have dismissed them for several reasons.........................There is also the issue of region coding (I have about half and half R4 and R1) When I looked into A DVD changer last time, the best option would be to run one on R4 and the other on R1


If you use a software utility called DVD region Free you can disable (by-pass) region checking both by the software player and the hardware firmware, works for RPC2 drives and RCE discs. Also, it has a function to disable annoying disc features like PCfrienly and you can skip all the trailers and security notes and go directly to the film or main menu.
gamejester
 
Posts: 131
Joined: Mon Jun 21, 2004 9:20 am

Postby rhinoman on Mon Nov 29, 2004 8:10 pm

My media server is an Asus P4P800 with a 2.4 celeron and 512mb or ram running xp pro. It uses a inexpensive 550w power supply that seems to cope fine.

The drives I have built up over time as capacity requirements have increased.

The boot drive is an old 4gb.

The first raid 5 array is a Highpoint 454 ide using 4*300gb 5400 2mb cache maxtor drives, ( I was warned against using having 2 drives per controller cable as it is supposed to be likly that if you lose one drive on the cable it will take out the other drive and you will lose your data).

This 4 drive array takes about 24 hours to initialise or rebuild following a lost drive. I've also completly moved an array from 1 machine to another with no problems.

In the same machine I have a second 454 and another 4*300gb drive array. The highpoint driver allows you to manage both of these arrays from windows.

For multiple file serving (even high def) these cheap cards work adequetly, they only struggle when doing simultaneous writes.

I have just added another highpoint card, this time a 8 way SATA jobbie ( cant remeber the part number :oops: ) with 8 Maxtor 300 gb 7200 16mb cache drives. This array initlalized in 6 hours and I copied 500gb to it in about 4 hours so it obviously performs much much better (unnecessary for pure file storage).

All in thats 17 drives running from a 550w psu crammed in a rackmount server case, most of the drives sit in front of 2* 120mm intake fans which keep them cool to touch even though they are crammed in, 2 80mm exhaust fans at the back and the machine runs case temps around 28C.

Other than the drives the individual bits are all reasonable priced.
rhinoman
 
Posts: 416
Joined: Wed Jan 14, 2004 8:58 pm
Location: Herne Bay, UK

Postby Groover on Tue Nov 30, 2004 7:57 am

Wow...lots of stuff to assimilate and I'm not even Borg!

Thanks for all the replies and advice. Seems like a RAID setup is more popular than just adding bulk HD's. I read on another site certain setup's only require an hour (hot swap ones) to rewrite from parity - otherwise it's approx 24 hours - eek! Still, I guess it would take some time to restore 1TB from a backup drive anyway. The laCie units I'm currently considering use several HD's setup in a RAID 0 config. I am looking into setting each one up as one part of a firewire RAID - if that's possible.

"I have 10 250GB WD RAID edition drives connected with 7 in a RAID-5 (1 hot spare) "


chadwolcott - what does 1 hot spare mean? (nice looking setup BTW - do you need a Phd to drive it?) Wow...3.5 TB. How much of it is spare?

How many of you are actually running your servers as movie servers - that is, having your DVD collections ripped on to the HD, or is it more for TV ? Do you run your ripped DVD's as VOB files, or as ISO's? Do you re-endode your DVD's to Divx or some other format to save space? How many of you just store the movie only? How many movies does your 2TB hold? Trying to figure how much space you need is a tough one - We have 460 titles currently, many of them boxsets - but I'm not sure I want to store all the extras as well - who watches it all anyway?

Always lots of questions I know. I'm trying to figure the best options so I don't have to take a mortgage out on the house :( to pay for it all.

I've been researching the old HTPC for many months now and I don't think I've even scratched the surface. Looks like it'll be a few more yet before I lay down some hard earned cash.
Groover
 
Posts: 14
Joined: Sat Nov 27, 2004 1:09 am

Postby rhinoman on Tue Nov 30, 2004 9:00 am

I believe that windows as a 2.1tb limit to an array (although that might be a limit of the cheaper controller cards, please feal free to correct me).

I rip my dvd's using dvd shrink, removing menus, credits, subtitles and unused soundtracks, I expect 2.1tb will hold about 400 dvd's in this manner.

My arrays are used for different stuff, The large SATA raid holds my dvd's (or will doi when I finish ripping), the other 2 smaller ones hold music and hidef respectivly.

Xlobby runs on a seperate machine and pulls the reqired media from this server.
rhinoman
 
Posts: 416
Joined: Wed Jan 14, 2004 8:58 pm
Location: Herne Bay, UK

Postby chadwolcott on Tue Nov 30, 2004 8:51 pm

Groover,

a hot spare is essentially an extra hard drive that is plugged in but no data is stored on it. If the RAID controller determines that a drive is failing or has failed it automatically begins to rebuild the array using the hot spare to replace the failing drive. So it's just a marketing term for a backup drive that is "hot" or online.

No Phd required, took me a day or so to build and is no more difficult to run than any windows PC. I didn't even put 2003 server on it, but I thought about it since I have a nice MSDN subscription :D

As I mentioned earlier, I use it for both DVDs and TV and music. I have 210 DVDs ripped to that array now with 150GB free. When I get my 3 other drives built into the array I expect to be able to hold another 700 GB worth so figuring about 5GB per movie I imagine about 350-390 total DVDs.

I rip all my DVDs with DVDShrink IFO/VOB and rip the movies only with AC3 sound and DTS if it's there. I would guess that with 460+ DVDs you are going to want 3TB or so worth of space unless you decide to re-encode to WMV/XViD/Divx. That process alone would take months, I can usually rip a movie in about 12 minutes. A re-encode would most likely take a couple hours. You definately need more CPU if you plan to re-encode than you would to just rip and serve.

Rhinoman,

You are correct, NTFS has a theoretical volume size limit of 16 exabytes but is really 256 terabytes in practice. You can only get there if you use dynamic volumes in windowsxp. The Master Boot Record's partition table only supports 2.2 TB. If you are using an Itanium, I believe you can get beyond the 2.2TB barrier. There has been some conjecture as to whether the 64-bit version of windows will get past the 2.2 barrier since the barrier is due in part to the 32-bit nature of XP. I haven't done any testing as of yet on this. I'm still trying to save up for a 64-bit processor to use my 64-bit version on. I suspect the limit will still be there since the 64-bit is mostly derived from the 32-bit code. Future versions of Longhorn may allow larger then 2.2 TB since they will eventually get to a completely new file system.
chadwolcott
 
Posts: 107
Joined: Fri Sep 05, 2003 2:34 pm
Location: Boston, MA

Postby rhinoman on Tue Nov 30, 2004 9:29 pm

Chad,

Did I possible read somewhere that some of the newest raid controllers allow this 2.1tb limit to be overcome without the need for 64bit processors?
rhinoman
 
Posts: 416
Joined: Wed Jan 14, 2004 8:58 pm
Location: Herne Bay, UK

Next