|
Server upgrade path -
07-31-2006, 03:49 PM
Looking over the current disc space stats and server benchmarks I've come to the conclusion that it's time to begin the next upgrade. Right now the average byte unix benchmark "wht-variant" is 100.26 - that isn't too bad, and compares competitively with most servers in operation, and ran the benchmark on live production servers, at about 5:00PM one day last week. This is much higher than the average VPS. VPS's on the same benchmark are getting scores of between 10 and 50, and it shows. Which is why we don't like VPS's much.
Many of our servers are running on upgraded OS's originally installed up to 5 years ago,and could really use a fresh OS reinstall. Having two running side by side and flipping a switch is the much preferred way to transition to a clean OS install and that's what we plan to do. Software versions will stay the same so there will be no website functions breaking overnight. Downtime for the switch will not exist because the old and new will be up and running when the route get switched. All of our software is pretty much the latest version available, but built on previous upgrades and that is causing some problems.
We have been testing out some hardware alternatives and have whittled it down to dual core dual Opterons or the new dual core 5000 series Intel chip which we can't seem to get yet. We have an Intel based motherboard sitting here waiting on the cpu's.
The system I'm testing now is a dual AMD 64, dual core on a Tyan motherboard with 3ware 9550sx raid. Testing it without the raid returned a benchmark of 253, with raid, 320 but that will probably go up a little. All the servers will have over 1TB of user space available in a raid10 environment. Probably 4x750GB sata2 HD's striped & mirrored. Though not as fast as scsi, the tradeoff in capacity is impossible to pass up, and sata2 (@3Gb/s) is pretty fast, especially when striped across a few discs with a true hardware controller like the 3ware 9550sx. We will set up "redundant everything" in terms of hardware. If a power supply dies, the server will simply sound a alarm and switch to its backup power. A tech will then walk out and pull the bad module out and insert a good one. Same thing for hard drives. If a HD goes bad, we will simply insert a new drive while the server remains up and running, and it will rebuild on the fly from its working mirror drive. Under certain circumstances the servers could suffer multiple drive failures and remain up and serving all websites.
Backups will switch from the current method of compressing and backing up every website every night, to an rsync method where only the files that have changed since the last backup are moved to backup every night. This will change the time it takes to complete backups from roughly 15 hours/day to one hour, and free up about 30-60% of the cpu resources.
The bottom line for hosting customers is that servers will be an average of at least 3 times faster, the I/O bottleneck during backups and data redundancy problems will be solved by moving to an rsync based backup system and disc space quota on all hosting plans raised by a factor of 5 as well when these things are put into place. It will be about a $120,000 investment over the next year. The goal is to have them all replaced within that timeframe with the first one getting shipped to Atlanta within a month.
|
Linear Mode
