Newbie Question

[martink0646]

Hi Paul,

SUCCESS!!!! Thanks to your help I've got iso's playing & more importantly I understand why.

I had a bit of trouble with the ANT Movie Catalog. I run DVD Profiler & ANT just won't handle the size of the imported file (3500 records, 38mb XML file) on the PC I am using. Still, I think I have a handle on where I am. Due to my storge limitations I have only got about 200 DvD's ripped at the moment. My long term plan is to have them all stored digitally, but a quick calculation shows that in todays money it will cost me around £3K for all that HDD space. Not only would SWMBO kill me if I even thought about spending that much, I would have to have a dedicated room (or at least a sizeable cabinet & go to some form of central server or large NAS facility) to hold it all. What I am doing is choosing our favourite films & using the spiders to do the work. I quickly used the imported DvD Profiler info into xLobby but I had so few of the titles ripped that it didn't seem worth it for now.

All I need now is the bombshell that in 8 days when the licence runs out xLobby are gonna want £x00's & I'll have to jack it in or justify it to the wife. Must grab some kip.

Again, thanks.

Martin

[cooldog]

Martin,

GADS! You've got 3500 DVD's ?????

Did you find the info on importing from DVD-Profiler to AMC?

I've got about 500 movies online now, mostly HD stuff downloaded off Usenet. We're creeping up on 2000 CD's, all or which are ripped to MP3, then our still photos, and HDV and the older DV home videos. Add to that a whole lot of live multi-track digital audio recordings from our houseconcert series. All that occupies 6.14TB leaving me 1.82TB on my main storage server.

I can still add another 11 drives to that RAID controller, but when I do, I'll have to work out the cabinetry issues, because the controller has no external connectors, and there's no more room for drives in the box.

But I do have a 2.7TB array that I had transferred to the new one, so it's idle at the moment...

All in the basement, of course ...

[martink0646]

Hi Paul,

I haven't looked yet with regard to the import issues as I finished stupidly late last night. I actually think that it has more to do with the memory & processor limitations of the PC I'm using as a testbed for this project. I don't want to spec a really good PC before I know it works & xLobby have decided whether or not to go with this as a pro product & price out the individual. When they've sorted this out I will get a better PC.

Here is my DvD Profiler link; http://www.intervocative.com/dvdcollection.aspx/pest! (Actually this is a bit out of date so I probably nearer 4000 now)

Hypothetically, (because I can't afford it at the moment) if you were starting from scratch & putting together a system to house the 20TB I would need to digitize everything how would you go about setting it up. I have looked briefly into NAS with things like the Netgear or Buffalo systems but even so I would need 20+ of these with them holding 2 disks each. What would be the best solution for effectiveness & space utilization. I live in a converted chapel but we don't go in for cellars in the UK like you guys do in the states, so it would have to be space taken from somewhere else which isn't a major problem. I was thinking more about heat rejection issues, although it could double as a boost to the central heating & maybe go towards being cost neutral!!! (Yeah right. SWMBO will not fall for that!!) Your thoughts would be welcome.

[cmhardwick]

look at unRAID. http://www.lime-technology.com. You could setup 2 of these, each housing up to 16 drives (1 parity) for, with 1tb disks, 15tb per server.

[cooldog]

Martin,

That's quite a collection! If you can get DVD-Profiler info into XLobby with a hierarchical structure, I'd pursue that, since you've got a huge investment in time in that DB. Search the forum for info - the built-in import in XLobby is not the only way.

Also, the older free version of XLobby would do quite well for you if it turns out that the newer pay version is too pricey. You'd be fine with it, methinks.

With 20TB, you don't want to "F" around. If you lose it, you lose an enormous amount of time and work.


So, here's what I'd do:

Areca 24 port Raid controller
moderately powerful motherboard
CentOS 5 Linux OS
24x 750GB Seagate drives (or 1TB if building this in another six months)

As to chassis, the best solution would be Supermicro's new 24-drive monster, but it's a bit pricey. You could very cheaply make a totally DIY solution where the drives are simply hanging from metal rails. Also, you would probably be able to find a deal on an older, large, used server chassis which you could modify for the newer SATA drives.

Another possibility, if you want something fairly pretty, is to make use of Supermicro's 5-drive boxes. They are meant to slide into a three-bay (5.25" half-height) drive opening, and hold five SATA drives in hot-swap carriers. Five of them, built into a box, would be in between the price of a new single chassis with 24 hot-swap carriers and the total DIY solution.

The main difficulty is that there is no reasonable method for external connection of multiple SATA drives. The only standard is a 4-port cable based on Infiniband cabling and connectors, and they are ridiculously expensive ... and you'd need six of them ... and the expensive adapters at the drive box end to split the 4-port multi-lane connector back out to regular internal SATA again.

Some additional thoughts:

This beast *will* create a fair bit of noise, and will need to be in a ventilated closet.

Using a large array is much more space (both disk drive space and physical space) efficient, power efficient, and admin-time efficient than a huge stack of independent NAS boxes --- Linux will allow you to create huge arrays, and the Areca controller is FAST. Using RAID5, you only eat up one drive out of the array for redundancy info, and can tolerate a single drive failure ... with RAID6, you use two drives, but can tolerate a two drive failure.

Using Linux gets you the ability to really control how access to the files is allowed, and that allows you to protect against accidental deletion while still letting you work with the files when necessary. You can also set up Wizd as a UPnP server, so that you can drive networked media players.

That's about all I can think of at the moment.

So where in the UK are you? In November, we're going to be in Scotland for 10 days, then off to the London area for a week. My wife's cousin is getting married.

Paul

[martink0646]

Hi Paul,

Thanks for the detailed reply, thats given me a lot to think about. I have had a chat with my better half & we have some remodelling of the house to do early next year so I'm gonna put some plans together for then. It will be a good excuse to put in a gigabit ethernet network & really try & run everything from PC's although I will probably have to call in some help for that. Alternatively, I could try & centrally locate all my AV equipment & run everything over CAT5/5E/6. The technology seems to be developing fast & once I've seen a demo of it running Hi-Def I think that is a possibility. Time, money, complexity (set up) & ease of use will have to be juggled.

I am going to do some searching on the forum with regard to DvD Profiler importation tonight when the little one has gone to bed so I'll see what I can dig up.

When you're in the UK how are you getting about, train or car? I live in Wisbech which is the nearest port to the Wash at the top of East Anglia. It is a Georgian town & the capital of the Fens, an area of the UK that is below sea level, reclaimed marshland (totally flat). It reminds me somewhat of the area surrounding Minneapolis when I had to stopover there last year on a business trip. Truly flat as far as the eye can see. The reclamation work was done by the dutch for us in the 18th century. Because it is all flood plains it is excellent growing soil, just farmlamd everywhere.

If you are staying in the East of Scotland & are travelling by car you could come down the A1 which goes via Peterborough & I am about 20 miles from there (our distances are much smaller). If you are travelling by train from the East you will come down the East Coast Mainline which again goes through the main hub, Peterborough. Where are you staying in London? I commute into London most days. Check it out on a map & let me know, we could sort something out, maybe take a detour to the coast to break up your journey.

Martin

[cooldog]

Martin,

See http://xlobby.com/forum/viewtopic.php?f=14&t=3766&st=0&sk=t&sd=a&hilit=dxc_movie. I used to use the earlier version before switching from DVD-Profiler to AMC.

We'll be at Kilconquhar Castle, in Elie for the first week, then another 3 days somewhere in that area, then down to London. We will have a car.

I'm beat, so more later ....

Paul