linux - storage - virtualization

vSphere Management App for iPAD

2010-09-11T12:44:00.001-07:00

An interesting application to manage your vmware vsphere datacenter … through an Applie iPAD. Pretty interesting .. The app is still in its early stages, but the demo shows great features of the app. Watch this video from vmworld TV.

Vsphere Tutorial & demo videos

2010-09-10T12:15:00.000-07:00

Mike Laverick has done a great job in sharing out the Vmware Vsphere video tutorials, that clearly explains number of vsphere concepts, howto and technologies in details. Worth watching!

Watch the vsphere tutorial demos at Mike Laverick’s page :

http://www.rtfm-ed.co.uk/?page_id=1442

Tuning Netapp snapmirror/snapvault speed

2010-09-09T12:14:00.000-07:00

In Netapp, there is a option to set/limit bandwidth of all snapmirror and snapvault transfers. The option can be either system-wide or for a particular transfer. We can tune either the transmit bandwidth on the source or the receive bandwidth on the destination or both together. For particular transfer, you can tune the throttle from snapmirror.conf.

When both per transfer and system-wide throttling are configured, throttling at system wide is applied only if the combined bandwidth used by all the relationships goes above the system-wide throttling value. System-wide throttling is enabled by using three new options using the options command.

To list the tunable replication throttle values:

destination-filer*> options replication replication.throttle.enable off replication.throttle.incoming.max_kbs unlimited replication.throttle.outgoing.max_kbs unlimited
destination-filer*>

Enable Throttling: This enables throttling of Snapmirror and Snapvault transfers.

destination-filer*> options replication.throttle.enable on

Set incoming bandwidth limit:

destination-filer*> options replication.throttle.incoming.max_kbs 1024

This netapp option has to be applied on the destination system. This option specifies the maximum total bandwidth used by all the incoming SnapMirror and SnapVault transfers, specified in kilobytes/sec. The default value is "unlimited" .

Set outgoing bandwidth limit:

source-filer*> options replication.throttle.outgoing.max_kbs 1024

This Netapp option has to be applied on the source system. This option specifies the maximum total bandwidth used by all the outgoing SnapMirror and SnapVault transfers specified in kilobytes/sec.

Netapp Deduplication – quick setup guide

2010-02-15T07:41:00.000-08:00

Deduplication refers to the elimination of redundant data in the storage. In the deduplication process, duplicate data is deleted, leaving only one copy of the data to be stored. However, indexing of all data is still retained should that data ever be required. Deduplication is able to reduce the required storage capacity since only the unique data is stored.

Netapp supports deduplication where only unique blocks in the flex volume is stored and it creates a small amount of additional metadata in the dedup process. The NetApp deduplication technology allows duplicate 4KB blocks anywhere in the flexible volume to be deleted and stores a unique one.

The core enabling technology of deduplication is fingerprints. These are unique digital signatures for every 4KB data block in the flexible volume.

When deduplication runs for the first time on a flexible volume with existing data, it scans the blocks in the flexible volume and creates a fingerprint database, which contains a sorted list of all fingerprints for used blocks in the flexible volume. After the fingerprint file is created, fingerprints are checked for duplicates and if found, first a byte-by-byte comparison of the blocks is done to make sure that the blocks are indeed identical. If they are found to be identical, the block’s pointer is updated to the already existing data block and the duplicate data block is released and inode is updated.

Netapp Deduplication commands:

Enable dedup (asis) license.

fractal-design> sis on /vol/demovol
If you have a new flex volume which was just created, follow this step to enable ASIS deduplication

fractal-design> sis on /vol/demovol
Deduplication for "/vol/demovol" is enabled.
Already existing data could be processed by running "sis start -s /vol/demovol”
If you have already existing flex volume with data in it, follow this step.

fractal-design> sis start -s /vol/demovol
Checking the status of deduplication.
fractal-design> vol status demovol
Volume          State   Status          Options
VolArchive      online raid_dp, flex   nosnap=on
                        sis
Containing aggregate: 'aggr0'
fractal-design>

fractal-design> sis status /vol/demovol
Path            State   Status      Progress
/vol/demovol    Enabled Idle        Idle for 00:02:12
fractal-design>
Check the storage space saved due to deduplication
fractal-design> df -s /vol/demovol
Filesystem used saved %saved
/vol/demovol/ 9316052 0 0%
fractal-design>
If you have to run deduplication at a later point of time on this volume, just do a “sis start /vol/demovol”.
The sis can be scheduled using “sis config” command.
Done.

More netapp blog posts at : http://unixfoo.blogspot.com/search/label/netapp

Solaris LDOMs virtualization : setup guide

2010-02-10T05:32:00.000-08:00

Sun Logical Domains or LDoms is a full virtual machine that runs an independent operating system instance and contains virtualized CPU, memory, storage, console, and cryptographic devices. This technology allows you to allocate a system resources into logical groupings and create multiple, discrete systems, each with their own operating system, resources, and identity within a single computer system. We can run a variety of applications software in different logical domains and keep them independent of performance and security purposes. The LDoms environment can help to achieve greater resource usage, better scaling, and increased security and isolation.

Logical Domains

View more documents from Jarod Wang.

Logical & Control domain : The control domain communicates with the hypervisor to create and manage all logical domain configurations within a server platform. The Logical Domains Manager is used to create and manage logical domains. The Logical Domains Manager maps logical domains to physical resources. Without access to the Logical Domains Manager all logical domain resource levels remain static. The initial domain created when installing Logical Domains software is a control domain and is named primary.

Image from : http://www.sun.com/blueprints/0207/820-0832.pdf

You can download Logical Domain manager from http://sun.com/ldoms . Please read the release notes for system firmware requirements and patch requirements. By default, Ldoms software gets installed to /opt/SUNWldm/. Make sure the below commands works - and that confirms Logical domain manager is running.

solfoo23# /opt/SUNWldm/bin/ldm list
Name State Flags Cons VCPU Memory Util Uptime
primary active -t-cv SP 32 16128M 49% 90mm

Creating default services : You need to create the default virtual services that the control domain uses to provide disk services, console access and networking. The below commands explains them.

Create Virtual Disk server(vds) : Virtual disk server helps importing virtual disks into a logical domain from the control domain.

solfoo23# ldm add-vds primary-vds0 primary
Create Virtual Console concentrator Server(vcc) : Virtual Console concentrator server provides terminal service to logical domain consoles.

solfoo23# ldm add-vcc port-range=5000-5100 primary-vcc0 primary
Create Virtual Switch server(vsw) : Virtual Switch server enables networking between virtual network devices in logical domains.

solfoo23# ldm add-vsw net-dev=e1000g0 primary-vsw0 primary
List the default services created

solfoo23# ldm list-services primary
VDS
NAME VOLUME OPTIONS DEVICE
primary-vds0

VCC
NAME PORT-RANGE
primary-vcc0 5000-5100

VSW
NAME MAC NET-DEV DEVICE MODE
primary-vsw0 00:11:5a:12:dc:fc e1000g1 switch@0 prog,promisc

Control Domain Creation : The next step is to perform the initial setup of the primary domain, which will act as the control domain. You should specify the resources that the primary domain will use and what will be released for use by other guest domains. In this document, we are creating the control domain with 2 cpu's and 1gb RAM.

solfoo23# ldm set-mau 0 primary
solfoo23# ldm set-vcpu 2 primary
solfoo23# ldm set-memory 1024M primary

Now, set these modified configuration permanent using list-spconfig option.

solfoo23# ldm list-spconfig
factory-default [current]

solfoo23# ldm add-spconfig initial

solfoo23# ldm list-spconfig
factory-default [current]
initial [next]

Reboot the server and it will come up with initial configuration.

Networking between domains : Networking between control, service and other domains is disabled by default. To enable this, the virtual switch device should be configured as a network device. On the server console and perform the following network configuration steps.

Plumb the virtual switch(vsw0)

solfoo23# ifconfig vsw0 plumb
Bring down the primary interface

solfoo23# ifconfig e1000g1 down unplumb
Configure Virtual switch with the primary interface details

solfoo23# ifconfig vsw0 <ip> netmask <netmask> broadcast + up
Modify the hostname file to make this configuration permanent

solfoo23# mv /etc/hostname.e1000g1 /etc/hostname.vsw0
Enable Virtual Network terminal server daemon

solfoo23# svcadm enable vntsd

Now the setup is done. Run "ldm list-bindings primary" and make sure they are ok.

Logical Domain Creation : Now that the system is ready, prepare and plan for the logical domain configuration. In this document, we are creating a logical domain with 2 CPUs and 1GB memory and "domfoo" is the name.

solfoo23# ldm add-domain domfoo
solfoo23# ldm add-vcpu 2 domfoo
solfoo23# ldm add-memory 1G domfoo
solfoo23# ldm add-vnet vnet1 primary-vsw0 domfoo
solfoo23# ldm add-vdsdev /dev/dsk/c1t2d0s2 vol1@primary-vds0
solfoo23# ldm add-vdisk vdisk1 vol1@primary-vds0 domfoo
solfoo23# ldm bind domfoo
solfoo23# ldm set-var auto-boot\?=false domfoo
solfoo23# ldm start-domain domfoo

You will be able see the domain using "ldm list-domain"

solfoo23# ldm list-domain
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv SP 2 2G 0.2% 3h 4m
domfoo inactive ----- 2 1G

Connect to the logical domain console by telneting to the virtual console port.

solfoo23# telnet localhost 5000
Trying 127.0.0.1...
Connected to localhost....
Escape character is ’^]’.
Connecting to console "domfoo" in group "domfoo" ....
Press ~? for control options ..
{0} ok

Your LDom is up!. You can install it using jumpstart. Your LDoms environment is ready!

Refer Solaris LDOM virtualization document links for more information :

Overview : http://www.sun.com/servers/coolthreads/ldoms/wp.pdf
LDOMs Beginners guide : http://www.sun.com/blueprints/0207/820-0832.pdf
LDOMs Demo : http://blogs.sun.com/ash/resource/flashdemos/64-ldoms-on-t2.html
LDOMs presentation : http://www.snpnet.com/customer_pub/sun/isv_LDOM/
Admin guide : http://docs-pdf.sun.com/819-6428-11/819-6428-11.pdf

Netapp performance monitoring : sysstat : Part II

2010-02-04T18:53:00.000-08:00

Here are some explanations on the columns of netapp sysstat command.

Cache age : The age in minutes of the oldest read-only blocks in the buffer cache. Data in this column indicates how fast read operations are cycling through system memory; when the filer is reading very large files, buffer cache age will be very low. Also if reads are random, the cache age will be low. If you have a performance problem, where the read performance is poor, this number may indicate you need a larger memory system or analyze the application to reduce the randomness of the workload.

Cache hit : This is the WAFL cache hit rate percentage. This is the percentage of times where WAFL tried to read a data block from disk that and the data was found already cached in memory. A dash in this column indicates that WAFL did not attempt to load any blocks during the measurement interval.

CP Ty : Consistency Point (CP) type is the reason that a CP started in that interval. The CP types are as follows:

- No CP started during sampling interval (no writes happened to disk at this point of time)
number Number of CPs started during sampling interval
B Back to back CPs (CP generated CP) (The filer is having a tough time keeping up with writes)
b Deferred back to back CPs (CP generated CP) (the back to back condition is getting worse)
F CP caused by full NVLog (one half of the nvram log was full, and so was flushed)
H CP caused by high water mark (rare to see this. The filer was at half way full on one side of the nvram logs, so decides to write on disk).
L CP caused by low water mark
S CP caused by snapshot operation
T CP caused by timer (every 10 seconds filer data is flushed to disk)
U CP caused by flush
: continuation of CP from previous interval (means, A cp is still going on, during 1 second intervals)

The type character is followed by a second character which indicates the phase of the CP at the end of the sampling interval. If the CP completed during the sampling interval, this second character will be blank. The phases are as follows:

0 Initializing
n Processing normal files
s Processing special files
f Flushing modified data to disk
v Flushing modified superblock to disk

CP util : The Consistency Point (CP) utilization, the % of time spent in a CP. 100% time in CP is a good thing. It means, the amount of time, used out of the cpu, that was dedicated to writing data, 100% of it was used. 75% means, that only 75% of the time allocated to writing data was utilized, which means we wasted 25% of that time. A good CP percentage has to be at or near 100%.

You can use Netapp SIO tool to benchmark netapp systems. SIO is a client-side workload generator that works with any target. It generates I/O load and does basic statistics to see how any type of storage performs under certain conditions.

Netapp performance monitoring : sysstat - Part I

2010-02-04T18:51:00.000-08:00

Netapp sysstat is like vmstat and iostat rolled into one command. It reports filer performance statistics like CPU utilization, the amount of disk traffic, and tape traffic. When run with out options, sysstat will prints a new line every 15 seconds, of just a basic amount of information. You have to use control-C (^c) or set the interval count (-c count ) to stop sysstat after time. For more detailed information, use the -u option. For specific information to one particular protocol, you can use other options. I'll list them here.

-f FCP statistics
-i iSCSI statistics
-b SAN (blocks) extended statistics
-u extended utilization statistics
-x extended output format. This includes all available output fields. Be aware that this produces output that is longer than 80 columns and is generally intended for "off-line" types of analysis and not for "real-time" viewing.
-m Displays multi-processor CPU utilization statistics. In addition to the percentage of the time that one or more CPUs were busy (ANY), the average (AVG) is displayed, as well as, the individual utilization of each processor. This is only handy on multi proc systems. Won't work on single processor machines.

Netapp Simulator – Installation steps

2010-02-01T08:10:00.000-08:00

Those who are newly learning Netapp can use Netapp Data OnTap Simulator to get comfortable with Netapp commands. This tool gives you the experience of administering and using a NetApp storage system with all the features of Data ONTAP. The Simulator can be downloaded from http://now.netapp.com/NOW/cgi-bin/simulator ( you need NOW access ). The simulator has fully functional license keys for all Netapp functionalities.

The simulator can be loaded onto a Red Hat or SuSE Linux box and looks and feels exactly like Data ONTAP. Almost anything you can do with Data ONTAP can be done with the simulator. Without purchasing new hardware or impacting your production environment, you can test functionality, export NFS and CIFS shares etc.

Steps to install Simulator:

Download the simulator iso from netapp and mount it on a linux machine.

[root@fedora01 ~]# mount -o loop 7.3.2-sim-cdrom-image-v22.iso /mnt
[root@fedora01 ~]# cd /mnt
[root@fedora01 mnt]# ls
disks2.tgz disks.tgz doc license.htm readme.htm runsim.sh setup.sh sim.tgz Vmware, Linux and Simulator installation.doc

Install the simulator software, using the setup script as below

[root@fedora01 mnt]# ./setup.sh

Script version 22 (18/Sep/2007)
Where to install to? [/sim]: /data/
/data/ already exists. This may overwrite files.
Are you sure? [no]: yes
Would you like to install as a cluster? [no]: no
Would you like full HTML/PDF FilerView documentation to be installed [yes]: yes
Continue with installation? [no]: yes
Creating /data/
Unpacking sim.tgz to /data/
Configured the simulators mac address to be [00:50:56:14:1c:d9]
Please ensure the simulator is not running.
Your simulator has 3 disk(s). How many more would you like to add? [0]: 21

The following disk types are available in MB:
        Real (Usable)
a -   43   ( 14)
b -   62   ( 30)
c -   78   ( 45)
d - 129   ( 90)
e - 535   (450)
f - 1024   (900)

If you are unsure choose the default option a
What disk size would you like to use? [a]: e
Disk adapter to put disks on? [0]:
Use DHCP on first boot? [yes]:
Ask for floppy boot? [no]:
Checking the default route...
You have a single network interface called eth0 (default route) .
Which network interface should the simulator use? [default]:
The recommended memory is 512MB.
How much memory would you like the simulator to use? [512]:
Create a new log for each session? [no]: yes
Adding 21 additional disk(s).
Complete. Run /data/runsim.sh to start the simulator.
[root@fedora01 mnt]#

Run /data/runsim.sh and provide the details during the first boot and halt the simulator.
Run /data/runsim.sh to start the Netapp simulator again. You can launch the FilerView with url http://ip-address-of-simulator/na_admin.

[root@fedora01 data]# /data/runsim.sh
runsim.sh script version Script version 22 (18/Sep/2007)
This session is logged in /data/sessionlogs/log-1265126584

NetApp Release 7.3.2: Thu Oct 15 03:13:41 PDT 2009
Copyright (c) 1992-2009 NetApp.
Starting boot on Tue Feb 2 16:03:05 GMT 2010
Tue Feb 2 16:03:35 GMT [fmmb.current.lock.disk:info]: Disk v4.16 is a local HA mailbox disk.
Tue Feb 2 16:03:35 GMT [fmmb.current.lock.disk:info]: Disk v4.17 is a local HA mailbox disk.
Tue Feb 2 16:03:35 GMT [fmmb.instStat.change:info]: normal mailbox instance on local side.
Tue Feb 2 16:03:37 GMT [raid.cksum.replay.summary:info]: Replayed 0 checksum blocks.
Tue Feb 2 16:03:37 GMT [raid.stripe.replay.summary:info]: Replayed 0 stripes.
sparse volume upgrade done. num vol 0.
Vdisk Snap Table for host:0 is initialized
Tue Feb 2 16:03:40 GMT [vol.language.unspecified:info]: Language not set on volume vol0. Using language config "C". Use vol lang to set language.
Tue Feb 2 16:03:40 GMT [rc:notice]: The system was down for 39 seconds
Tue Feb 2 16:03:56 GMT [dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Tue Feb 2 16:03:56 GMT [sfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk shelves.
Tue Feb 2 16:03:56 GMT [netif.linkUp:info]: Ethernet ns0: Link up.
Tue Feb 2 16:03:57 GMT [perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 195 counters.
add net default: gateway 192.168.0.1
Tue Feb 2 16:04:03 GMT [mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Tue Feb 2 16:03:01 GMT 2010
Tue Feb 2 16:04:03 GMT [mgr.boot.reason_ok:notice]: System rebooted after a halt command.
CIFS local server is running.

Password:
unixfoo-simulator> Tue Feb 2 16:04:13 GMT [console_login_mgr:info]: root logged in from console

Documents related to Netapp Simulator :

Linux Enterprise Cryptographic Filesystem : ecryptfs

2010-01-27T07:16:00.001-08:00

eCryptfs is a stacked cryptographic filesystem embedded within the Linux kernel. Being a stacked filesystem, it can easily encrypt and decrypt the files on your Linux server as they are written to or read from the hard disk. The greatest advantage of eCryptfs is that all encryption is made at the file level. This means that you don’t have to create a fixed size container to hold your files.

Here are the steps how to use ecryptfs:

Install ecryptfs using yum

[root@fedora01 ~]# yum install ecryptfs-utils
Load the ecryptfs module into the kernel, using modprobe

[root@fedora01 ~]# modprobe ecryptfs
Now, I have to encrypt all my files that I read & write on /home/secretdata. So, mount /home/secretdata on a encrypted mount-point.

[root@fedora01 ~]# mount -t ecryptfs /home/secretdata/ /encrypted
Select key type to use for newly created files:
1) tspi
2) openssl
3) passphrase
Selection: 3
Passphrase:
Select cipher:
1) aes: blocksize = 16; min keysize = 16; max keysize = 32 (not loaded)
2) blowfish: blocksize = 16; min keysize = 16; max keysize = 32 (not loaded)
3) des3_ede: blocksize = 8; min keysize = 24; max keysize = 24 (not loaded)
4) twofish: blocksize = 16; min keysize = 16; max keysize = 32 (not loaded)
5) cast6: blocksize = 16; min keysize = 16; max keysize = 32 (not loaded)
6) cast5: blocksize = 8; min keysize = 5; max keysize = 16 (not loaded)
Selection [aes]:
Select key bytes:
1) 16
2) 32
3) 24
Selection [16]:
Enable plaintext passthrough (y/n) [n]:
Attempting to mount with the following options:
ecryptfs_unlink_sigs
ecryptfs_key_bytes=16
ecryptfs_cipher=aes
ecryptfs_sig=35954df565c12fac
Mounted eCryptfs
The directory /encrypted is an “encrypted one” now. What ever you write to this directory will be encrypted here. See the demo in below steps. Here I copied /etc/passwd to the encrypted filesystem. I am able to read the file till the encrypted filesystem is mounted.

[root@fedora01 encrypted]# cp /etc/passwd /encrypted/

[root@fedora01 encrypted]# head -1 /encrypted/passwd
root:x:0:0:root:/root:/bin/bash
[root@fedora01 encrypted]#

Once the encrypted filesystem is out of action, we cannot read the file – as it is encrypted.

[root@fedora01 ~]# umount /encrypted/

[root@fedora01 ~]# file /home/secretdata/passwd
/home/secretdata/passwd: data
[root@fedora01 ~]#
Cool, huh? . Read more at https://launchpad.net/ecryptfs

Dynamically detecting new disks in Linux

2010-01-26T06:48:00.000-08:00

When you have new LUNs created on the SAN fabric, zoned & mapped it to the server; how can you detect the luns on the linux server online, without rebooting it?.

When you dynamically add new disks to a Linux VM running on ESX server, how do you detect that disks on the Linux virtual machine?.

Here are the steps to do that :

Install sg3_utils and lsscsi package.

[root@fedora01 ~]# # yum install –y sg3_utils lsscsi
The “lsscsi” command will list the disks attached to it. If you have just attached a disk, you will not be able to see it. You can also list this using “fdisk –l”

[root@fedora01 ~]# lsscsi
[0:0:0:0] disk VMware Virtual disk 1.0 /dev/sda
[root@fedora01 ~]#

As you can see above, I currently have one disk connected to the system. To scan for a new device I just added, we should run rescan-scsi-bus.sh from the host.
Run the command “/usr/bin/rescan-scsi-bus.sh” , to dynamically detect and activate the new disk.

[root@fedora01 ~]# /usr/bin/rescan-scsi-bus.sh -l
Host adapter 0 (mptspi) found.
Scanning SCSI subsystem for new devices
Scanning host 0 for SCSI target IDs 0 1 2 3 4 5 6 7, LUNs 0 1 2 3 4 5 6 7
Scanning for device 0 0 0 0 ...
OLD: Host: scsi0 Channel: 00 Id: 00 Lun: 00
      Vendor: VMware   Model: Virtual disk   Rev: 1.0
      Type:   Direct-Access                  ANSI SCSI revision: 02
Scanning for device 0 0 1 0 ...
NEW: Host: scsi0 Channel: 00 Id: 01 Lun: 00
      Vendor: VMware   Model: Virtual disk   Rev: 1.0
      Type:   Direct-Access                  ANSI SCSI revision: 02
1 new device(s) found.
0 device(s) removed.
[root@fedora01 ~]#

[root@fedora01 ~]# lsscsi
[0:0:0:0]    disk    VMware   Virtual disk     1.0   /dev/sda
[0:0:1:0]    disk    VMware   Virtual disk     1.0   /dev/sdb
[root@fedora01 ~]#

You see the new disk is visible. Now you can create a partition or filesystem on it.

After running those commands, check dmesg and /var/log/messages to see if there are any device detections. You can also do "fdisk -l" or "cat /proc/scsi/scsi" to see the attached LUNs. This works fine in RHEL5, SuSE 10, CentOS5, OEL5.

OCFS2 cluster – quick setup guide

2010-01-06T08:42:00.000-08:00

OCFS2 is a POSIX-compliant shared-disk cluster file system for Linux capable of providing both high performance and high availability. Cluster-aware applications can make use of parallel I/O for higher performance. OCFS2 is mostly used to host Oracle Real application clusters (RAC) database on Linux clusters.

The below steps shows how to create ocfs2 filesystem on top a multipath'd SAN lun and mount it on Linux clusters.

Identify the nodes that will be part of your cluster.
Export/Zone the LUNs on the SAN end and check whether they are accessible on all the hosts of the cluster. (fdisk -l or multipath -ll)
If you need multipathing, configure multipath and the multipathing policy based on your requirement. For Linux multipath setup, refer Redhat’s multipath guide.
Create OCFS2 configuration file (/etc/ocfs2/cluster.conf) on all the cluster nodes.
The example presents you a sample cluster.conf for a 3 node pool. If you have heartbeat IP configured on these cluster nodes, use the heartbeat IP for ocfs2 cluster communication and specify the hostname (without FQDN). Copy the same file to all the hosts in the cluster.

[root@oracle-cluster-1 ~]# cat /etc/ocfs2/cluster.conf
node:
        ip_port = 7777
        ip_address = 203.21.2.101
        number = 0
        name = oracle-cluster-1
        cluster = ocfs2

node:
        ip_port = 7777
        ip_address = 203.21.2.102
        number = 1
        name = oracle-cluster-2
        cluster = ocfs2

node:
        ip_port = 7777
        ip_address = 203.21.2.103
        number = 2
        name = oracle-cluster-3
        cluster = ocfs2

cluster:
        node_count = 3
        name = ocfs2

[root@oracle-cluster-1 ~]#
On each node check the status of OCFS2 cluster service and stop "o2cb" if the service is already running.

# service o2cb status
# service o2cb stop
On each node, load the OCFS2 module.

# service o2cb load
Make the OCFS2 service online on all the nodes.

# service o2cb online
Now your OCFS2 cluster is ready.
Format the SAN lun device from any one of the cluster node.

# mkfs.ocfs2 -b 4k -C 32k -L oraclerac /dev/mapper/mpath0

-b : Block size (values are 512, 1K, 2K and 4K bytes per block)
-C : Cluster size (values are 4K, 8K, 16K, 32K, 64K, 128K, 256K, 512K and 1M)
-L : Label

Note : Replace /dev/mapper/mpath0 with your device name.
Update /etc/fstab on all the nodes in the cluster with the mount point.

Like : /dev/mapper/mpath0 /u01 ocfs2 _netdev 0 0
Mount the /u01 volume using mount command

# mount /u01
Enable ocfs and o2b service at runlevel 3.

# chkconfig --level 345 o2cb on ; chkconfig --level 345 ocfs2 on
The /u01 repository setup on a SAN Lun is done.
You can now configure Oracle RAC database on this filesystem.

rpm package command tips

2009-12-01T17:57:00.001-08:00

If you want to find which rpm contains a specific file in your linux system, you can use "rpm -qf" to get the package information.

opensuse:~ # rpm -qf /etc/vmware-tools/tools.conf
open-vm-tools-2008.09.03-5.49
opensuse:~ #

Find out what all dependencies on an package?

opensuse:~ # rpm -qR libgcrypt11
rpmlib(VersionedDependencies) <= 3.0.3-1
/bin/sh
/bin/sh
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(CompressedFileNames) <= 3.0.4-1
libc.so.6
libc.so.6(GLIBC_2.0)
libc.so.6(GLIBC_2.1)
libc.so.6(GLIBC_2.1.3)
libc.so.6(GLIBC_2.3)
libc.so.6(GLIBC_2.3.4)
libc.so.6(GLIBC_2.4)
libgpg-error.so.0
rpmlib(PayloadIsLzma) <= 4.4.2-1
opensuse:~ #

How to list what all files are part of a package?

opensuse:~ # rpm -q --filesbypkg libgcrypt11-1.4.1-4.1
libgcrypt11               /lib/libgcrypt.so.11
libgcrypt11               /lib/libgcrypt.so.11.4.4
libgcrypt11               /usr/share/doc/packages/libgcrypt11
libgcrypt11               /usr/share/doc/packages/libgcrypt11/AUTHORS
libgcrypt11               /usr/share/doc/packages/libgcrypt11/COPYING
libgcrypt11               /usr/share/doc/packages/libgcrypt11/COPYING.LIB
libgcrypt11               /usr/share/doc/packages/libgcrypt11/ChangeLog
libgcrypt11               /usr/share/doc/packages/libgcrypt11/NEWS
libgcrypt11               /usr/share/doc/packages/libgcrypt11/README
libgcrypt11               /usr/share/doc/packages/libgcrypt11/THANKS
libgcrypt11               /usr/share/doc/packages/libgcrypt11/TODO
opensuse:~ #

Fileop : Filesystem IO benchmarking tool

2009-11-30T20:34:00.000-08:00

Iozone Linux filesystem benchmarking tool has another cool utility called "fileop". This utility does file operation on the filesystem and gives the ops/sec for each operation. This tool will be useful to test your filesystem performance as well as storage performance. Or, if you are going to select a filesystem or storage that is going to host huge number of small files (like source codes), then fileop can be used for benchmarking.

Fileop can create files of different size ( with -s option ) and to different directory depth ( with -f option ).

[root@bryant foo367]# /opt/iozone/bin/fileop -f 10 -s 1k

Fileop: File size is 1024, Output is in Ops/sec. (A=Avg, B=Best, W=Worst)
.      mkdir rmdir create   read write close   stat access chmod readdir link   unlink delete Total_files
A   10    795    771   1381 22915   1147   2282   1128 312914   1697    742    579    825    809         1000
[root@bryant foo367]#

[root@bryant foo367]# /opt/iozone/bin/fileop -f 40 -s 4k -t

Fileop: File size is 4096, Output is in Ops/sec. (A=Avg, B=Best, W=Worst)
mkdir:   Dirs =      1640 Total Time = 2.069957018 seconds
         Avg mkdir(s)/sec     =       792.29 ( 0.001262169 seconds/op)
         Best mkdir(s)/sec    =      1039.74 ( 0.000961781 seconds/op)
         Worst mkdir(s)/sec   =       355.87 ( 0.002810001 seconds/op)

rmdir:   Dirs =      1640 Total Time = 2.155941963 seconds
         Avg rmdir(s)/sec     =       760.69 ( 0.001314599 seconds/op)
         Best rmdir(s)/sec    =      1273.70 ( 0.000785112 seconds/op)
         Worst rmdir(s)/sec   =        52.90 ( 0.018905163 seconds/op)

create: Files =     64000 Total Time = 47.258722544 seconds
         Avg create(s)/sec    =      1354.25 ( 0.000738418 seconds/op)
         Best create(s)/sec   =      5440.08 ( 0.000183821 seconds/op)
         Worst create(s)/sec =         7.50 ( 0.133283138 seconds/op)

write:   Files =     64000 Total Time = 58.136307001 seconds
         Avg write(s)/sec     =      1100.86 ( 0.000908380 seconds/op)
         Best write(s)/sec    =      3155.98 ( 0.000316858 seconds/op)
         Worst write(s)/sec   =        11.95 ( 0.083698034 seconds/op)

...

The fileop does perform mkdir, rmdir, create, write, close , stat, write, access, link, unlink, chmod, readdir and gives you performance stats in ops/sec and secs/op.

Lustre – cluster filesystem : quick setup guide

2009-11-08T09:09:00.000-08:00

The Lustre file system is a distributed high performance cluster filesytem that redefines I/O performance and scalability for large and complex computing environments. This is ideally suited for data-intensive applications which requires the high IO performance.

Lustre components:

MDS – Metadata Server: The MDS server makes metadata stored in the metadata targets available to Lustre clients.

MDT – Metadata Target: This stores metadata, such as filenames, directories, permissions, and file layout, on the metadata server.

OSS – Object Storage Server: The OSS provides file I/O service, and network request handling for the OSTs. The MDT, OSTs and Lustre clients can run concurrently on a single node. However, a typical configuration is an MDT on a dedicated node, two or more OSTs on each OSS node, and a client on each of a large number of compute nodes.

OST – Object Storage Target: The OST stores data as data objects on one or more OSSs. A single Lustre file system can have multiple OSTs, each serving a subset of file data.

Client: The systems that mount the Lustre filesystem.

Steps to create Lustre FS

Configure Lustre Management Server (lustre-mgs.unixfoo.biz – Server 1)

Add the disk to volume manager

[root@lustre-mgs mnt]# pvcreate /dev/sdb
Physical volume "/dev/sdb" successfully created

[root@lustre-mgs mnt]# pvs
PV VG Fmt Attr PSize Pfree
/dev/sdb lvm2 -- 136.73G 136.73G
Create lustre volume group

[root@lustre-mgs mnt]# vgcreate lustre /dev/sdb
Volume group "lustre" successfully created
[root@lustre-mgs mnt]# vgs
VG #PV #LV #SN Attr VSize Vfree
lustre 1 0 0 wz--n- 136.73G 136.73G
Create logical volume "MGS" (the management server)

[root@lustre-mgs ~]# lvcreate -L 25G -n MGS lustre
Create Lustre Management filesystem.

[root@lustre-mgs ~]# mkfs.lustre --mgs /dev/lustre/MGS

   Permanent disk data:
Target:     MGS
Index:      unassigned
Lustre FS: lustre
Mount type: ldiskfs
Flags:      0x74
              (MGS needs_index first_time update )
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters:

checking for existing Lustre data: not found
device size = 10240MB
2 6 18
formatting backing filesystem ldiskfs on /dev/lustre/MGS
        target name MGS
        4k blocks     0
        options        -J size=400 -q -O dir_index,uninit_groups –F

mkfs_cmd = mkfs.ext2 -j -b 4096 -L MGS -J size=400 -q -O dir_index,uninit_groups -F /dev/lustre/MGS
Writing CONFIGS/mountdata
[root@lustre-mgs ~]#
Activate the lustre management filesystem using mount command.

[root@lustre-mgs ~]# mount -t lustre /dev/lustre/MGS /lustre/MGS/
[root@lustre-mgs ~]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda2             54558908   5276572 46466144 11% /
/dev/sda1               497829     29006    443121   7% /boot
tmpfs                  8216000         0   8216000   0% /dev/shm
/dev/lustre/MGS       10321208    433052   9363868   5% /lustre/MGS
[root@lustre-mgs ~]#

Configure Lustre Metadata Server (In this guide, both the management & metadata server runs on the same host)

Create logical volume "MDT_unixfoo_cloud"

[root@lustre-mgs ~]# lvcreate -L 25G -n MDT_unixfoo_cloud lustre
Create Lustre Metdata filesystem for the filesystem “unixfoo_cloud”.

[root@lustre-mgs ~]# mkfs.lustre --fsname=unixfoo_cloud --mdt --reformat --mgsnode=lustre-mgs@tcp0 /dev/lustre/MDT_unixfoo_cloud
   Permanent disk data:
Target:     unixfoo_cloud-MDTffff
Index:      unassigned
Lustre FS: unixfoo_cloud
Mount type: ldiskfs
Flags:      0x71
              (MDT needs_index first_time update )
Persistent mount opts: errors=remount-ro,iopen_nopriv,user_xattr
Parameters: mgsnode=10.217.0.237@tcp mdt.group_upcall=/usr/sbin/l_getgroups
device size = 20480MB
2 6 18

formatting backing filesystem ldiskfs on /dev/lustre/MDT_unixfoo_cloud
        target name unixfoo_cloud-MDTffff
        4k blocks     0
        options        -J size=400 -i 4096 -I 512 -q -O dir_index,uninit_groups –F
mkfs_cmd = mkfs.ext2 -j -b 4096 -L unixfoo_cloud-MDTffff -J size=400 -i 4096 -I 512 -q -O dir_index,uninit_groups -F /dev/lustre/MDT_unixfoo_cloud
Writing CONFIGS/mountdata
[root@lustre-mgs ~]#
Activate the metdata filesystem using mount command.

[root@lustre-mgs ~]# mkdir /lustre/MDT_unixfoo_cloud
[root@lustre-mgs ~]# mount -t lustre /dev/lustre/MDT_unixfoo_cloud /lustre/MDT_unixfoo_cloud

Configure OSTs ( servers: oss1, oss2 .. )

Add /dev/md0 to volume manager

[root@oss1 ~]# pvcreate /dev/md0
Create volume group "lustre"

[root@oss1 ~]# vgcreate lustre /dev/md0
Create logical volume (ost) for unixfoo_cloud

[root@oss1 ~]# lvcreate -n OST_unixfoo_cloud_1 -L 100G lustre
Create OST lustre filesystem

[root@oss1 ~]# mkfs.lustre --fsname=unixfoo_cloud --ost --mgsnode=lustre-mgs@tcp0 /dev/lustre/OST_unixfoo_cloud_1

mkfs_cmd = mkfs.ext2 -j -b 4096 -L unixfoo_cloud-OSTffff -J size=400 -i 16384 -I 256 -q -O dir_index,uninit_groups -F /dev/lustre/OST_unixfoo_cloud_1
Writing CONFIGS/mountdata
[root@oss1 ~]#
Activate the OST by using mount command

[root@oss1 ~]# mkdir -p /lustre/unixfoo_cloud_oss1
[root@oss1 ~]# mount -t lustre /dev/lustre/OST_unixfoo_cloud_1 /lustre/unixfoo_cloud_oss1

Mount on the client:

Mount the lustre filesystem unixfoo_cloud

[root@lustreclient1 ~]# mount -t lustre lustre-mgs@tcp0:/unixfoo_cloud /mnt

[root@lustreclient1 ~]# df –h
Filesystem            Size Used Avail Use% Mounted on
/dev/sda2              52G 5.1G   44G 11% /
/dev/sda1             487M   29M 433M   7% /boot
tmpfs                 7.9G     0 7.9G   0% /dev/shm
lustre-mgs@tcp0:/unixfoo_cloud
                       99G 461M   93G   1% /mnt
[root@lustreclient1 ~]#
Done.

HTTP Compression & Web Caching

2009-11-06T03:49:00.000-08:00

After setting up my squid proxy for web caching, I was trying to see if it supports http compression. But the version which I had installed doesn't support webcache compression. Google gave me ViCompress .

Here is what www.visolve.com says about ViCompress ..

“ViCompress HTTP proxy server reduces download time by gzip compressing text pages before sending them on the client. This results in faster download times, especially over slow network connections. ViCompress works with existing web servers and web browsers. All modern browsers (Internet Explorer, Firefox, Safari, Chrome) recognize gzip-encoded data, and will automatically decompress the data on the fly. Also, ViCompress can compress both static and dynamic web pages, including Apache/PHP, Microsoft ASP pages, Perl CGI scripts, and others”

Steps to setup ViCompress:

Download ViCompress rpm from visolve.com

# wget http://www.visolve.com/vicompress/ViCompress-1.1.0-1.i386.rpm
Install the package (rpm -ivH ViCompress-1.1.0-1.i386.rpm).
This installs in /usr/local/vicompress/
Edit the configuration file for the proxy port number and other values.

opensuse:~ # cat /usr/local/vicompress/etc/vicompress.conf
# Vicompress configuration file

listen 0.0.0.0 8081           # IP address and port to listen on
outgoingip 0.0.0.0          # IP address to bind to for outgoing connections
enable_sessions yes         # Enable sticky sessions with multiple webservers
enable_compression yes      # Enable compression of html/text pages
enable_caching yes          # Enable caching of web pages/images
cache_memory 200            # Amount of memory used for cache, in Megabytes
max_cacheditem_size 512     # Max size of cached items, in kilobytes
cache_expires 240           # How long a cached page is valid (in hours)
enable_dns_caching yes      # Enable caching of dns lookups
dns_expires 48              # How long cached dns lookups are valid (in hours)
user nobody                 # User to run server as
rotatesize 100              # Size (in megabytes) at which to rotate log files
logformat squid             # Format of access log. Either apache or squid
enable_debug no             # Enable debug messages in the errorlog
accesslog /var/log/vicompress/accesslog      # Access log file path
errorlog   /var/log/vicompress/errorlog       # Error log file path
errorpage /usr/local/vicompress/etc/errorpage.html # HTML page for errors
logstats   /usr/local/vicompress/logstats           # Log statistics directory
opensuse:~ #
Tune the network parameters of the kernel by using the script given by vicompress. This tunes the network parameters like rmem_max, wmem_max, tcp_rmem, tcp_wmem .

opensuse:~ # /usr/local/vicompress/bin/tune_kernel.sh
opensuse:~ #
Start the vicompress compression enabled web cache proxy server.

opensuse:~ # /usr/local/vicompress/bin/vicompress.sh start
/usr/local/vicompress/bin/vicompress.sh start: vicompress started
opensuse:~ #
On the client browsers, change the proxy and check the speed of compression enabled browsing.
vicompress also provides reporting collected by the command
/usr/local/vicompress/bin/update_log_stats.sh.

Squid Cache proxy – a quick setup guide

2009-11-01T23:27:00.000-08:00

This post gives the steps to quickly setup squid proxy and make it up and caching in minutes. There are also some tips & tricks on squid. My squid proxy server runs on OpenSuSE 11.1.

opensuse:~ # cat /etc/SuSE-release
openSUSE 11.1 (i586)
VERSION = 11.1
opensuse:~ #

To download “squid” packages on opensuse, use zypper. The Zypper is a package/patch management tool in Suse, which works similar to yum.

opensuse:~ # zypper install squid3 squidGuard yast2-squid
Loading repository data...
Reading installed packages...
Resolving package dependencies...

The following NEW packages are going to be installed:
libboost_regex1_36_0 squid3 squidGuard yast2-squid

Check the installed Squid rpms.

opensuse:~ # rpm -qa | grep squid
squidGuard-1.3-57.63
squid3-3.0.STABLE10-2.11
yast2-squid-2.17.5-1.11
opensuse:~ #

The default squid configuration file is /etc/squid/squid.conf. The below configuration shows my setup, where I have changed the proxy port number & the cache directory. Make sure you have the cache directory created with the owner as squid.

opensuse:~ # grep -v ^# /etc/squid/squid.conf | grep -v ^$
acl manager proto cache_object
acl localhost src 127.0.0.1/32
<snip>
http_port 8080
hierarchy_stoplist cgi-bin ?
cache_dir ufs /home/cache/squid 200 16 256
access_log /var/log/squid/access.log squid
coredump_dir /var/cache/squid
opensuse:~ #

Start Squid service using the below command.

opensuse:~ # service squid start
Starting WWW-proxy squid done
opensuse:~ #

Now set your client browsers to access internet thorough this proxy. (opensuse:8080)

To check the squid detailed cache information, you can use the command “squidclient”, as described below

opensuse:~ # squidclient -p 8080 cache_object://localhost/info
Squid Object Cache: Version 3.0.STABLE10
Connection information for squid:
        Number of clients accessing cache:      1
        Number of HTTP requests received:       0
        Number of ICP messages received:        0
        Number of ICP messages sent:    0
        Number of queued ICP replies:   0
        Number of HTCP messages received:       0
        Number of HTCP messages sent:   0
        Request failure ratio:   0.00
        Average HTTP requests per minute since start:   -0.0
        Average ICP messages per minute since start:    -0.0
        Select loop called: 23738 times, -143987.578 ms avg
<snip>

opensuse:~ # squidclient -p 8080 cache_object://localhost/info | grep -A10 ^Cache
Cache information for squid:
        Hits as % of all requests:      5min: 0.0%, 60min: 0.0%
        Hits as % of bytes sent:        5min: -0.0%, 60min: -0.0%
        Memory hits as % of hit requests:       5min: 0.0%, 60min: 0.0%
        Disk hits as % of hit requests: 5min: 0.0%, 60min: 0.0%
        Storage Swap size:      6324 KB
        Storage Swap capacity:   3.1% used, 96.9% free
        Storage Mem size:       116 KB
        Storage Mem capacity:    1.4% used, 98.6% free
        Mean Object Size:       14.85 KB
        Requests given to unlinkd:      0
opensuse:~ #

Some more very useful squid tips at : http://sial.org/howto/squid/

Netapp Snapmirror Setup Guide

2009-10-23T22:41:00.000-07:00

Snapmirror is an licensed utility in Netapp to do data transfer across filers. Snapmirror works at Volume level or Qtree level. Snapmirror is mainly used for disaster recovery and replication.

Snapmirrror needs a source and destination filer. (When source and destination are the same filer, the snapmirror happens on local filer itself. This is when you have to replicate volumes inside a filer. If you need DR capabilities of a volume inside a filer, you have to try syncmirror ).

Synchronous SnapMirror is a SnapMirror feature in which the data on one system is replicated on another system at, or near, the same time it is written to the first system. Synchronous SnapMirror synchronously replicates data between single or clustered storage systems situated at remote sites using either an IP or a Fibre Channel connection. Before Data ONTAP saves data to disk, it collects written data in NVRAM. Then, at a point in time called a consistency point, it sends the data to disk.

When the Synchronous SnapMirror feature is enabled, the source system forwards data to the destination system as it is written in NVRAM. Then, at the consistency point, the source system sends its data to disk and tells the destination system to also send its data to disk.

This guides you quickly through the Snapmirror setup and commands.

1) Enable Snapmirror on source and destination filer

source-filer> options snapmirror.enable
snapmirror.enable on
source-filer>
source-filer> options snapmirror.access
snapmirror.access legacy
source-filer>

2) Snapmirror Access

Make sure destination filer has snapmirror access to the source filer. The snapmirror filer's name or IP address should be in /etc/snapmirror.allow. Use wrfile to add entries to /etc/snapmirror.allow.

source-filer> rdfile /etc/snapmirror.allow
destination-filer
destination-filer2
source-filer>

3) Initializing a Snapmirror relation

Volume snapmirror : Create a destination volume on destination netapp filer, of same size as source volume or greater size. For volume snapmirror, the destination volume should be in restricted mode. For example, let us consider we are snapmirroring a 100G volume - we create the destination volume and make it restricted.

destination-filer> vol create demo_destination aggr01 100G
destination-filer> vol restrict demo_destination

Volume SnapMirror creates a Snapshot copy before performing the initial transfer. This copy is referred to as the baseline Snapshot copy. After performing an initial transfer of all data in the volume, VSM (Volume SnapMirror) sends to the destination only the blocks that have changed since the last successful replication. When SnapMirror performs an update transfer, it creates another new Snapshot copy and compares the changed blocks. These changed blocks are sent as part of the update transfer.

Snapmirror is always destination filer driven. So the snapmirror initialize has to be done on destination filer. The below command starts the baseline transfer.

destination-filer> snapmirror initialize -S source-filer:demo_source destination-filer:demo_destination
Transfer started.
Monitor progress with 'snapmirror status' or the snapmirror log.
destination-filer>

Qtree Snapmirror : For qtree snapmirror, you should not create the destination qtree. The snapmirror command automatically creates the destination qtree. So just volume creation of required size is good enough.

Qtree SnapMirror determines changed data by first looking through the inode file for inodes that have changed and changed inodes of the interesting qtree for changed data blocks. The SnapMirror software then transfers only the new or changed data blocks from this Snapshot copy that is associated with the designated qtree. On the destination volume, a new Snapshot copy is then created that contains a complete point-in-time copy of the entire destination volume, but that is associated specifically with the particular qtree that has been replicated.

destination-filer> snapmirror initialize -S source-filer:/vol/demo1/qtree destination-filer:/vol/demo1/qtree
Transfer started.
Monitor progress with 'snapmirror status' or the snapmirror log.

4) Monitoring the status : Snapmirror data transfer status can be monitored either from source or destination filer. Use "snapmirror status" to check the status.

destination-filer> snapmirror status
Snapmirror is on.
Source                          Destination                          State          Lag Status
source-filer:demo_source        destination-filer:demo_destination   Uninitialized -   Transferring (1690 MB done)
source-filer:/vol/demo1/qtree   destination-filer:/vol/demo1/qtree   Uninitialized -   Transferring (32 MB done)
destination-filer>

5) Snapmirror schedule : This is the schedule used by the destination filer for updating the mirror. It informs the SnapMirror scheduler when transfers will be initiated. The schedule field can either contain the word sync to specify synchronous mirroring or a cron-style specification of when to update the mirror. The cronstyle schedule contains four space-separated fields.

If you want to sync the data on a scheduled frequency, you can set that in destination filer's /etc/snapmirror.conf . The time settings are similar to Unix cron. You can set a synchronous snapmirror schedule in /etc/snapmirror.conf by adding “sync” instead of the cron style frequency.

destination-filer> rdfile /etc/snapmirror.conf
source-filer:demo_source destination-filer:demo_destination - 0 * * * # This syncs every hour
source-filer:/vol/demo1/qtree destination-filer:/vol/demo1/qtree - 0 21 * * # This syncs every 9:00 pm
destination-filer>

6) Other Snapmirror commands

To break snapmirror relation - do snapmirror quiesce and snapmirror break.
To update snapmirror data - do snapmirror update
To resync a broken relation - do snapmirror resync.
To abort a relation - do snapmirror abort

Snapmirror do provide multipath support. More than one physical path between a source and a destination system might be desired for a mirror relationship. Multipath support allows SnapMirror traffic to be load balanced between these paths and provides for failover in the event of a network outage.

To read how to tune the performance & speed of the netapp snapmirror or snapvault replication transfers and adjust the transfer bandwidth , go to Tuning Snapmirror & Snapvault replication data transfer speed

ZFS : Basic administration guide

2009-10-23T07:33:00.000-07:00

ZFS is a combined file system and logical volume manager designed by Sun Microsystems. The features of ZFS include support for high storage capacities, integration of the concepts of filesystem and volume management, snapshots and copy-on-write clones, continuous integrity checking and automatic repair, RAID-Z and native NFSv4 ACLs.

Watch this video about ZFS overview and demo. A good one..

This ZFS guide provides an overview of ZFS and its administration commands that will be helpful for beginners.

ZFS Pool:

ZFS organizes physical devices into logical pools called storage pools. Both individual disks and array logical unit numbers (LUNs) that are visible to the operating system can be included in a ZFS pools. These pools can be created as disks striped together with no redundancy (RAID 0), mirrored disks (RAID 1), striped mirror sets (RAID 1 + 0), or striped with parity (RAID Z). Additional disks can be added to pools at any time but they must be added with the same RAID level.

ZFS Filesystem :

ZFS offers a POSIX-compliant file system interface to the Solaris/OpenSolaris operating system. ZFS file systems must be built in one and only one storage pool, but a storage pool may have more than one defined file system. ZFS file systems are managed & mounted through /etc/vfstab file. The common way to mount a ZFS file system is to simply define it against a pool. All defined ZFS file systems automatically mount at boot time unless otherwise configured.

Here are the basic commands for getting started with ZFS.

Creating Storage pool using "zpool create" :

bash-3.00# zpool create demovol raidz c2t1d0 c2t2d0
bash-3.00# zpool status
pool: demovol
state: ONLINE
scrub: none requested
config:

        NAME         STATE     READ WRITE CKSUM
        demovol      ONLINE       0     0     0
          raidz1     ONLINE       0     0     0
            c2t1d0   ONLINE       0     0     0
            c2t2d0   ONLINE       0     0     0

errors: No known data errors
bash-3.00#

"zfs list" will give the details of the pool and other zfs filesytems.

bash-3.00# zfs list
NAME                   USED AVAIL REFER MOUNTPOINT
demovol               1.00G 900G 38.1K /demovol
bash-3.00#

Creating File Systems : "zfs create" is used to create zfs filesytem.

bash-3.00# zfs create demovol/testing
bash-3.00# zfs list
NAME                   USED AVAIL REFER MOUNTPOINT
demovol               1.00G 900G 38.1K /demovol
demovol/testing       32.6K 900G 32.6K /demovol/testing
bash-3.00#

bash-3.00# ls /dev/zvol/dsk/demovol -- This should show you the disk file.

Setting Quota for the filesytem : Until Quota is set, the filesytem shows the total available space of the containter zfs pool.

bash-3.00# zfs set quota=10G emspool3/testing
bash-3.00# zfs list
NAME                     USED AVAIL REFER MOUNTPOINT
demovol                 1.00G 900G   39.9K /demovol
demovol/testing         32.6K 10.0G 32.6K /demovol/testing

Creating a snapshot :

bash-3.00# zfs snapshot demovol/testing@snap21
bash-3.00# zfs list
NAME                     USED AVAIL REFER MOUNTPOINT
demovol                 1.00G 900G   39.9K /demovol
demovol/testing         32.6K 10.0G 32.6K /demovol/testing
demovol/testing@snap21      0      - 32.6K -
bash-3.00#

Get all properties of a ZFS filesytem :

bash-3.00# zfs get all demovol/testing
NAME             PROPERTY         VALUE                  SOURCE
demovol/testing type             filesystem             -
demovol/testing creation         Mon Feb 9 9:05 2009 -
demovol/testing used             32.6K                  -
demovol/testing available        10.0G                  -
demovol/testing referenced       32.6K                  -
demovol/testing compressratio    1.00x                  -
demovol/testing mounted          yes                    -
demovol/testing quota            10G                    local
demovol/testing reservation      none                   default
demovol/testing recordsize       128K                   default
demovol/testing mountpoint       /demovol/testing       default
..

Cloning a ZFS filesystem from a snapshot :

bash-3.00# zfs clone demovol/testing@snap21 demovol/clone22
bash-3.00# zfs list
NAME                     USED AVAIL REFER MOUNTPOINT
demovol                 1.00G 900G   39.9K /demovol
demovol/clone22             0 900G   32.6K /demovol/clone22
demovol/testing         32.6K 10.0G 32.6K /demovol/testing
demovol/testing@snap21      0      - 32.6K -
bash-3.00#

Performance IO Monitoring the ZFS storage pool:

bash-3.00# zpool iostat 1
               capacity     operations    bandwidth
pool         used avail   read write   read write
---------- ----- ----- ----- ----- ----- -----
demovol     4.95M 900G      0      0      0     35
demovol     4.95M 900G      0      0      0      0
demovol     4.95M 900G      0      0      0      0
demovol     4.95M 900G      0      0      0      0

Please refer to the man pages, zfs and zpool, for more detailed information. Additional documentation may be found at docs.sun.com and OpenSolaris ZFS Community.

Dstat : Linux Performance analysis tool

2009-10-20T10:25:00.000-07:00

Dstat is a handy utility for monitoring systems during performance tuning tests, benchmarks or troubleshooting. It combines vmstat, iostat, ifstat, netstat information and more.

Dstat overcomes some of their limitations and adds some extra features, more counters and flexibility. One more great feature of Dstat is that it is written in python, modular & easy to extend, add your own counters as plugins. It also allows to export CSV output, which can be imported in Gnumeric and Excel to make graphs.

Combines vmstat, iostat, ifstat, netstat information and more
Shows stats in exactly the same timeframe
Enable/order counters as they make most sense during analysis/troubleshooting
Modular design
Written in python so easily extendable for the task at hand
Easy to extend, add your own counters (please contribute those)
Includes about 10 external plugins to show how easy it is to add counters
Can summarize grouped block/network devices and give total numbers
Can show interrupts per device
Very accurate timeframes, no timeshifts when system is stressed
Shows exact units and limits conversion mistakes
Indicate different units with different colors
Show intermediate results when delay > 1

You can download Dstat at : http://dag.wieers.com/home-made/dstat

Linux Network bonding – setup guide

2009-10-13T04:34:00.000-07:00

Linux network Bonding is creation of a single bonded interface by combining 2 or more Ethernet interfaces. This helps in high availability of your network interface and offers performance improvement. Bonding is same as port trunking or teaming.

Bonding allows you to aggregate multiple ports into a single group, effectively combining the bandwidth into a single connection. Bonding also allows you to create multi-gigabit pipes to transport traffic through the highest traffic areas of your network. For example, you can aggregate three megabits ports into a three-megabits trunk port. That is equivalent with having one interface with three megabytes speed

Steps for bonding in Oracle Enterprise Linux and Redhat Enterprise Linux are as follows..

Step 1.

Create the file ifcfg-bond0 with the IP address, netmask and gateway. Shown below is my test bonding config file.

$ cat /etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE=bond0
IPADDR=192.168.1.12
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
USERCTL=no
BOOTPROTO=none
ONBOOT=yes

Step 2.

Modify eth0, eth1 and eth2 configuration as shown below. Comment out, or remove the ip address, netmask, gateway and hardware address from each one of these files, since settings should only come from the ifcfg-bond0 file above. Make sure you add the MASTER and SLAVE configuration in these files.

$ cat /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
# Settings for Bond
MASTER=bond0
SLAVE=yes

$ cat /etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
# Settings for bonding
MASTER=bond0
SLAVE=yes

$ cat /etc/sysconfig/network-scripts/ifcfg-eth2

DEVICE=eth2
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes

Step 3.

Set the parameters for bond0 bonding kernel module. Select the network bonding mode based on you need, documented at http://unixfoo.blogspot.com/2008/02/network-bonding-part-ii-modes-of.html. The modes are

mode=0 (Balance Round Robin)
mode=1 (Active backup)
mode=2 (Balance XOR)
mode=3 (Broadcast)
mode=4 (802.3ad)
mode=5 (Balance TLB)
mode=6 (Balance ALB)

Add the following lines to /etc/modprobe.conf

# bonding commands
alias bond0 bonding
options bond0 mode=1 miimon=100

Step 4.

Load the bond driver module from the command prompt.

$ modprobe bonding

Step 5.

Restart the network, or restart the computer.

$ service network restart # Or restart computer

When the machine boots up check the proc settings.

$ cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.0.2 (March 23, 2006)

Bonding Mode: adaptive load balancing
Primary Slave: None
Currently Active Slave: eth2
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth2
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:13:72:80: 62:f0

Look at ifconfig -a and check that your bond0 interface is active. You are done!. For more details on the different modes of bonding, please refer to unixfoo’s modes of bonding.

To verify whether the failover bonding works..

Do an ifdown eth0 and check /proc/net/bonding/bond0 and check the “Current Active slave”.
Do a continuous ping to the bond0 ipaddress from a different machine and do a ifdown the active interface. The ping should not break.

Pictorial description of DAS NAS & SAN

2009-10-01T09:53:00.000-07:00

DAS:

Direct-attached storage, or DAS, is the most basic level of storage which most people are familiar with, in which storage devices are part of the host computer, as with drives, or directly connected to a single server, as with RAID arrays or tape libraries. Network workstations must therefore access the server in order to connect to the storage device. DAS is ideal for localized file sharing in environments with a single server or a few servers - for example, small businesses or departments and workgroups that do not need to share information over long distances or across an enterprise. From an economical perspective, the initial investment in direct-attached storage is cheaper.

NAS:

Networked storage was developed to address the challenges inherent in a server- based infrastructure such as direct-attached storage. Network-attached storage, or NAS, is a special purpose device, comprised of both hard disks and management software, which is 100% dedicated to serving files over a network.

SAN :

A storage area network (SAN) is an architecture to attach remote computer storage devices (such as disk arrays, tape libraries, and optical jukeboxes) to servers in such a way that the devices appear as locally attached to the operating system. Although the cost and complexity of SANs are dropping, they are uncommon outside larger enterprises. With their high degree of sophistication, management complexity and cost, SANs are traditionally implemented for mission-critical applications in the enterprise space. In a SAN infrastructure, storage devices such as NAS, DAS, RAID arrays or tape libraries are connected to servers using Fibre Channel. Fibre Channel is a highly reliable, gigabit interconnect technology that enables simultaneous communication among workstations, mainframes, servers, data storage systems and other peripherals. Without the distance and bandwidth limitations of SCSI, Fibre Channel is ideal for moving large volumes of data across long distances quickly and reliably.

Xen : Bridging on bonded & trunked interfaces

2009-09-30T10:35:00.000-07:00

When you deploy some critical stuffs like Oracle DB, Oracle RAC DB etc on the Xen or Oracle virtual machine, you'll have to use bonded network interfaces and the vlan trunks. By default, Xen and Oracle VM doesn't support the xen bridges created on bonded-vlaned-interfaces. The below document gives you a clear idea on how to create and configure xen bridges on bonded and vlan trunked interfaces.

Xen wiki document on bonded-vlaned xen network configuration
Oracle VM server configuration for bonded & trunked interfaces

Netapp Snapvault guide

2009-08-26T21:23:00.000-07:00

Netapp SnapVault is a heterogeneous disk-to-disk backup solution for Netapp filers and heterogeneous OS systems (Windows, Linux , Solaris, HPUX and AIX). Basically, Snapvault uses netapp snapshot technology to take point-in-time snapshot and store them as online backups. In event of data loss or corruption on a filer, the backup data can be restored from the SnapVault filer with less downtime. It has significant advantages over traditional tape backups, like

Reduce backup windows versus traditional tape-based backup
Media cost savings
No backup/recovery failures due to media errors
Simple and Fast recovery of corrupted or destroyed data

Snapvault consists of major two entities – snapvault clients and a snapvault storage server. A snapvault client (Netapp filers and unix/windows servers) is the system whose data should be backed-up. The SnapVault server is a Netapp filer – which gets the data from clients and backs up data. For Server to Netapp Snapvault, we need to install Open System Snapvault client software provided by Netapp, on the servers. Using the snapvault agent software, the Snapvault server can pull and backup data on to the backup qtrees. SnapVault protects data on a client system by maintaining a number of read-only versions (snapshots) of that data on a SnapVault filer. The replicated data on the snapvault server system can be accessed via NFS or CIFS. The client systems can restore entire directories or single files directly from the snapvault filer. Snapvault requires primary and secondary license.

How snapvault works?

When snapvault is setup, initially a complete copy of the data set is pulled across the network to the SnapVault filer. This initial or baseline, transfer may take some time to complete, because it is duplicating the entire source data set on the server – much like a level-zero backup to tape. Each subsequent backup transfers only the data blocks that has changed since the previous backup. When the initial full backup is performed, the SnapVault filer stores the data on a qtree and creates a snapshot image of the volume for the data that is to be backed up. SnapVault creates a new Snapshot copy with every transfer, and allows retention of a large number of copies according to a schedule configured by the backup administrator. Each copy consumes an amount of disk space proportional to the differences between it and the previous copy.

Snapvault commands :

Initial step to setup Snapvault backup between filers is to install snapvault license and enable snapvault on all the source and destination filers.

Source filer – filer1

    filer1> license add XXXXX
    filer1> options snapvault.enable on
    filer1> options snapvault.access host=filer2

Destination filer – filer2

    filer2> license add XXXXX
    filer2> options snapvault.enable on
    filer2> options snapvault.access host=filer1

Consider filer2:/vol/snapvault_volume as the snapvault destination volume, where all backups are done. The source data is filer1:/vol/datasource/qtree1. As we have to manage all the backups on the destination filer (filer2) using snapvault – manually disable scheduled snapshots on the destination volumes. The snapshots will be managed by Snapvault. Disabling Netapp scheduled snapshots, with below command.

    filer2> snap sched snapvault_volume 0 0 0

Creating Initial backup: Initiate the initial baseline data transfer (the first full backup) of the data from source to destination before scheduling snapvault backups. On the destination filer execute the below commands to initiate the base-line transfer. The time taken to complete depends upon the size of data on the source qtree and the network bandwidth. Check “snapvault status” on source/destination filers for monitoring the base-line transfer progress.

    filer2> snapvault start -S filer1:/vol/datasource/qtree1 filer2:/vol/snapvault_volume/qtree1

Creating backup schedules: Once the initial base-line transfer is completed, snapvault schedules have to be created for incremental backups. The retention period of the backup depends on the schedule created. The snapshot name should be prefixed with “sv_”. The schedule is in the form of “[@][@]”.

On source filer:

For example, let us create the schedules on source as below - 2 hourly, 2 daily and 2 weekly snapvault . These snapshot copies on the source enables administrators to recover directly from source filer without accessing any copies on the destination. This enables more rapid restores. However, it is not necessary to retain a large number of copies on the primary; higher retention levels are configured on the secondary. The commands below shows how to create hourly, daily & weekly snapvault snapshots.

    filer1> snapvault snap sched datasource sv_hourly 2@0-22
    filer1> snapvault snap sched datasource sv_daily 2@23
    filer1> snapvault snap sched datasource sv_weekly 2@21@sun

On snapvault filer:

Based on the retention period of the backups you need, the snapvault schedules on the destination should be done. Here, the sv_hourly schedule checks all source qtrees once per hour for a new snapshot copy called sv_hourly.0. If it finds such a copy, it updates the SnapVault qtrees with new data from the primary and then takes a Snapshot copy on the destination volume, called sv_hourly.0. If you don’t use the -x option, the secondary does not contact the primary and transfer the Snapshot copy. It just creates a snapshot copy of the destination volume.

    filer2> snapvault snap sched -x snapvault_volume sv_hourly 6@0-22
    filer2> snapvault snap sched -x snapvault_volume sv_daily 14@23@sun-fri
    filer2> snapvault snap sched -x snapvault_volume sv_weekly 6@23@sun

To check the snapvault status, use the command "snapvault status" either on source or destination filer. And to see the backups, do a "snap list" on the destination volume - that will give you all the backup copies, time of creation etc.

Restoring data : Restoring data is as simple as that, you have to mount the snapvault destination volume through NFS or CIFS and copy the required data from the backup snapshot.

You can also try Netapp Protection manager to manage the snapvault backups either from OSSV or from Netapp primary storage. Protection manager is based on Netapp Operations manager (aka Netapp DFM). It is a client based UI, with which you connect to the Ops Manager and protect your storages.

Vmware - Netapp cloning : Rapid VM provisioning

2009-07-10T05:59:00.000-07:00

Checkout this demo of provisioning 5000 Vmware VMs using Netapp File cloning and volume flexclone technologies. Good one ..

Pocket guide for netapp commands

2009-07-05T00:09:00.001-07:00

This post contains the list of commands that will be most used and will come handy when managing or monitoring or troubleshooting a Netapp filer.

sysconfig -a : shows hardware configuration with more verbose information
sysconfig -d : shows information of the disk attached to the filer
version : shows the netapp Ontap OS version.
uptime : shows the filer uptime
dns info : this shows the dns resolvers, the no of hits and misses and other info
nis info : this shows the nis domain name, yp servers etc.
rdfile : Like "cat" in Linux, used to read contents of text files/
wrfile : Creates/Overwrites a file. Similar to "cat > filename" in Linux
aggr status : Shows the aggregate status
aggr status -r : Shows the raid configuration, reconstruction information of the disks in filer
aggr show_space : Shows the disk usage of the aggreate, WAFL reserve, overheads etc.
vol status : Shows the volume information
vol status -s : Displays the spare disks on the filer
vol status -f : Displays the failed disks on the filer
vol status -r : Shows the raid configuration, reconstruction information of the disks
df -h : Displays volume disk usage
df -i : Shows the inode counts of all the volumes
df -Ah : Shows "df" information of the aggregate
license : Displays/add/removes license on a netapp filer
maxfiles : Displays and adds more inodes to a volume
aggr create : Creates aggregate
vol create <volname> <aggrname> <size> : Creates volume in an aggregate
vol offline <volname> : Offlines a volume
vol online <volname> : Onlines a volume
vol destroy <volname> : Destroys and removes an volume
vol size <volname> [+|-]<size> : Resize a volume in netapp filer
vol options : Displays/Changes volume options in a netapp filer
qtree create <qtree-path> : Creates qtree
qtree status : Displays the status of qtrees
quota on : Enables quota on a netapp filer
quota off : Disables quota
quota resize : Resizes quota
quota report : Reports the quota and usage
snap list : Displays all snapshots on a volume
snap create <volname> <snapname> : Create snapshot
snap sched <volname> <schedule> : Schedule snapshot creation
snap reserve <volname> <percentage> : Display/set snapshot reserve space in volume
/etc/exports : File that manages the NFS exports
rdfile /etc/exports : Read the NFS exports file
wrfile /etc/exports : Write to NFS exports file
exportfs -a : Exports all the filesystems listed in /etc/exports
cifs setup : Setup cifs
cifs shares : Create/displays cifs shares
cifs access : Changes access of cifs shares
lun create : Creates iscsi or fcp luns on a netapp filer
lun map : Maps lun to an igroup
lun show : Show all the luns on a filer
igroup create : Creates netapp igroup
lun stats : Show lun I/O statistics
disk show : Shows all the disk on the filer
disk zero spares : Zeros the spare disks
disk_fw_update : Upgrades the disk firmware on all disks
options : Display/Set options on netapp filer
options nfs : Display/Set NFS options
options timed : Display/Set NTP options on netapp.
options autosupport : Display/Set autosupport options
options cifs : Display/Set cifs options
options tcp : Display/Set TCP options
options net : Display/Set network options
ndmpcopy <src-path> <dst-path> : Initiates ndmpcopy
ndmpd status : Displays status of ndmpd
ndmpd killall : Terminates all the ndmpd processes.
ifconfig : Displays/Sets IP address on a network/vif interface
vif create : Creates a VIF (bonding/trunking/teaming)
vif status : Displays status of a vif
netstat : Displays network statistics
sysstat -us 1 : begins a 1 second sample of the filer's current utilization (crtl - c to end)
nfsstat : Shows nfs statistics
nfsstat -l : Displays nfs stats per client
nfs_hist : Displays nfs historgram
statit : beings/ends a performance workload sampling [-b starts / -e ends]
stats : Displays stats for every counter on netapp. Read stats man page for more info
ifstat : Displays Network interface stats
qtree stats : displays I/O stats of qtree
environment : display environment status on shelves and chassis of the filer
storage show <disk|shelf|adapter> : Shows storage component details
snapmirror intialize : Initialize a snapmirror relation
snapmirror update : Manually Update snapmirror relation
snapmirror resync : Resyns a broken snapmirror
snapmirror quiesce : Quiesces a snapmirror bond
snapmirror break : Breakes a snapmirror relation
snapmirror abort : Abort a running snapmirror
snapmirror status : Shows snapmirror status
lock status -h : Displays locks held by filer
sm_mon : Manage the locks
storage download shelf : Installs the shelf firmware
software get : Download the Netapp OS software
software install : Installs OS
download : Updates the installed OS
cf status : Displays cluster status
cf takeover : Takes over the cluster partner
cf giveback : Gives back control to the cluster partner
reboot : Reboots a filer

If you are not aware of the complete details of these commands and need more information on these commands, refer the Netapp Data Ontap administration manual from now site.

More netapp blog posts at : http://unixfoo.blogspot.com/search/label/netapp