(Inexpensive) Highly Available Storage Systems

The internet has pampered our users by giving them 99.9% uptime. Now every corporation needs to have similar availability to 99.9%. One part of high availability solution is the storage. In the good old proven corner we have SAN and NAS storage solutions. They are not always highly available (you must ensure the system has more than one controller, in case one controller is broken, and RAID 5 is nowhere enough these days, choose RAID 6, and dont forget network rendundancy), but they are almost always expensive. In the era where IT Directors forced to think 'cost transformation', we always welcome cost-saving alternatives.
New hardware developments influenced our infrastructure options, lets write the factors down:
  • abundance of cheap gigabit ethernet card & router -> this allow us to forget expensive fiber connetivity, and allow us to leverage newly developed distributed systems.
  • cheap large SATA drives -> large commodity storage for the masses
Development of Infrastructure support software also influenced in giving more options :
  • OpenSCSI iSCSI target -> converting our SATA drives into shared storage
  • CLVM -> clustered LVM allows distributed volume management
  • OCFS2 -> Oracle Clustered Filesystem is an open sourced cluster file system developed by Oracle 
  • Ceph RADOS massive cluster storage technology
  • Amazon Elastic Block Storage -> we need to run our servers in AWS to use these
  • DRBD -> Distributed Replicated Block Device driver, allows us to choose between synchronous mirroring and asynchronous mirroring between disks attached to different hosts
A Question in Serverfault asks about mature high availability storage systems, summarizing the answers :
  • The surprising fact is that the oracle-developed OCFS 2 is not the tool of choice because needs downtime to add a capacity to the cluster filesystem
  • Ceph RADOS is promising newcomer in the arena, giving Google File System like characteristics with standard block interface (RBD). Old enterprises usually hesitate to use such new technology.
  • The ultimate choice of sys admins is HA NFS (highly available NFS).

HA NFS

Requirements for Highly Available NFS is as follows:
  • two host (PC or Server)
  • two set of similar sized storage (identical SATA II disks preferred)
  • cluster resource manager : Pacemaker
  • cluster messaging : Heartbeat (alternative: Corosync)
  • logical volume manager : LVM2
  • NFS daemon
  • DRBD (Distributed Replicated Block Device)
Distributed Replicated Block Device ( http://www.drbd.org/ )
Installation instruction is available at references below, in short :
  1. install DRBD devices (drbd0) on top of physical disks (ex: sda1) : fill in /etc/drbd.d/nfs. Do the same in both host (server)
  2. configure LVM to ignore physical disk used (ex: sda1). configure LVM to read volumes in DRBD (ex: drbd0), disable LVM cache (fill in /etc/lvm/lvm.conf)
  3. create LVM Physical Volume (pvcreate), LVM volume group (vgcreate), LVM logical volume (lvcreate). see your favorite LVM tutorial for details.
  4. install & configure Heartbeat
  5. install & configure Pacemaker. Pacemaker must be configured to have :
    1. drbd resource (ocf:linbit:drbd). automatically set DRBD master/slave mode according to situation at hand
    2. nfs daemon resource (lsb:nfs or lsb:nfs-kernel-server)
    3. lvm resource (ocf:heartbeat:LVM, ocf:heartbeat:Filesystem)
    4. nfs exports resource (ocf:heartbeat:exportfs)
    5. floating ip addr resource (ocf:heartbeat:IPaddr2)
Illustration of two host HA NFS system
Automatic failover mechanism could be activated to give seamless NFS operation during failover. The advantage of this HA NFS configuration is :
- clients will use old proven NFS interface
- realtime synchronous replication ensure no data loss

The disadvantage of shown configuration is
- no horizontal scalability
- standby capacity of second host is not leveraged

Linbit (www.linbit.com) provides enterprise support for DRBD.

Ceph RADOS

Ceph is a distributed object storage system. Ceph (http://ceph.com/docs/master/)  has these features :
  • massive clustering with thousands Object Storage Devices (OSD). Ceph could run with minimum 2 OSD.
  • automated data replication with per-pool replication settings (ex: metadata : 3x repl, data: 2x repl)
  • data striping to improve performance across cluster
  • has POSIX filesystem client (CephFS), Openstack Swift compatible interface, and even REST interface
  • block device interface (RBD) -> suitable for virtualization, OpenStack cloud support
  • horizontal scalability : add more OSD and/or disks for more storage or performance
Illustration of Ceph RADOS cluster
The disadvantage of Ceph RADOS cluster is :
- new technology, need training and hands on experience to operate 
- stability not yet industry-proven, but already seen large deployments
Inktank (http://www.inktank.com/) and Hastexo GmbH (www.hastexo.com) provides enterprise support for Ceph Cluster deployments.

Conclusion

New inexpensive storage systems technology now exists to be leveraged to provide high availability storage and/or performance for our legacy applications that still cluster-unaware. 

Comments

Post a Comment

Popular posts from this blog

Long running process in Linux using PHP

Background To do stuff, I usually create web-based applications written in PHP. Sometimes we need to run something that takes a long time, far longer than the 10 second psychological limit for web pages. A bit of googling in stack overflow found us this  http://stackoverflow.com/questions/2212635/best-way-to-manage-long-running-php-script , but I will tell the similar story with a different solution. One of the long running tasks that need to be run is a Pentaho data integration transformation. Difficulties in long running PHP scripts I encountered some problems when trying to make PHP do long running tasks : PHP script timeout. This could be solved by running set_time_limit(0); before the long running tasks. Memory leaks. The framework I normally use have a bit of memory issues, this can be solved either by patching the framework (ok, it is a bit difficult to do, but I did something similar in the past ) or splitting the data to process into several batches. And if you
Read more

Reverse Engineering Reptile Kernel module to Extract Authentication code

Image
 This time I will show how to (partially) reverse engineer a linux kernel module. The linux kernel module is larger than usual kernel module because it is actually a Reptile-based rootkit dropped by some hacker.  Part 1. Get basic module information First we gather basic module info by using modinfo : [root@lb lkrg-0.9.1]# modinfo falc0n filename:       /lib/modules/3.10.0-1127.10.1.el7.x86_64/kernel/sdc/falc0n.ko intree:         Y license:         GPL retpoline:       Y rhelversion:     7.8 srcversion:     81F508029A53F7490CCDB44 depends:         vermagic:       3.10.0-1127.10.1.el7.x86_64 SMP mod_unload modversions   We could also refer to the kernel module file not yet installed in the OS : [root@lb lkrg-0.9.1]# file falc0n.ko falc0n.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=1bde6abf4732eb620983032a47fbcf07689ece11, not stripped [root@lb lkrg-0.9.1]# modinfo falc0n.ko filename:       /root/lkrg-0.9.1/falc0n.ko intree:         Y license:         GPL retp
Read more

SAP System Copy Lessons Learned

Image
Background Earlier this year I was part of a team that does System Copy for a 20 terabyte plus SAP ERP RM-CA System. And just now I am involved in doing two system copy in just over one week, for much lesser amount of data. I think I would note some lessons learned from the experience in this blog. For the record, we are migrating from HP/UX and AIX to Linux x86 platform. Things that go wrong First, following the System Copy guide carefully is quite a lot of work - mainly because some important stuff are hidden in references in the guide. And reading a SAP note that are referenced in another SAP note, that are referenced in Installation Guide.. is a bit too much. Let me describe what thing goes wrong. VM Time drift The Oracle RAC Cluster have time drift problem, killing one instance when the other is shutting down. The cure for our VMWare-based Linux database server is hidden in SAP Note 989963 "Linux VMWARE Timing", which is basically add a tinker panic 0 in the
Read more