If you are there, come track me down. I’ll be hanging about with the NetApp guys a little, teasing the other storage vendors, and I’ll definitely be at the NetApp Communities event on Tuesday night – http://communities.netapp.com/thread/10582.

Here is a quick list of the sessions I think I’ll attend. I’m still undecided about a few and have narrowed it down to 2. I’ll be sure to watch a lot more sessions than this after the event as there is a lot of good stuff I’m missing out on! I look forward to seeing everyone there though!

Monday
9am : ALT2002 – vCenter VSheild
10am : SS1055 – Partner Super Session
10.30am : TA6720 – Troubleshooting with ESXTOP for Advanced Users
12pm : BC7773 – SRM Misconceptions and Misconfigurations
12pm : GD01 – vSphere Security
1.30pm : BC7729 – Intelligent HA, Application awareness with HA
3pm : TA7994 – vStorage Update for vSphere 4.1

Tuesday
11am : MA7140 – vCloud Architecture Design Strategies
12.30pm : TA8218 – VMware Storage Vision
2pm : SE8206 – Security Hardening Guidelines for vSphere
2pm : ALT3001 – SRM Extended Configuration
3.30pm : DV7706 – View Composer, Technical Deep Dive
3.30pm : SE8389 – Architectural Overview of Security for the Private Cloud
5pm : ALT3003 – Performance Tuning for vSphere

Wednesday
9am : MA8030 – Saving Time with vCenter Orchestrator
9am : MA6580 – Bridging the Management gap with vMA
10.30am : EA6705 – Best Practices for virtualising Active Directory
12pm : EA7849 – Design, Deploy and Optimise Exchange 2010
1.30pm : TA8233 – Storage I/O Control
1.30pm : TA6862 – vDS Deep Dive
3pm : MA7528 – vCenter Operational Best Practices
3pm : TA8065 – Storage Best Practices (presented by XIV)
4.30pm : TA7743 – ESX iSCSI News
4.30pm : TA8245 – ESXi Internals

Thursday
10.30am : TA6944 – PowerCLI for Administrators
10.30am : ALT3004 – vSphere Troubleshooting
12pm : MA9800 – vSphere Cluster to Cloud
1.30pm : TA8101 – Virtual Storage Best Practices (presented by EMC)
1.30pm : TA7171 – Performance Best Practices
3pm : BC8372 – SRM Futures

Related articles by Zemanta

• Looking forward to VMworld (zdnet.com)
• What to Expect at VMworld 2010 (customerthink.com)
• My VMworld North America Session (flyingpenguin.com)

You’ll need pre-shared keys setup (as ever), but the rest is prompted for. There’s no data validation (no surprise). The script will output 4 text files, one for creating/restricting all the volumes, one for the “snapmirror.conf”, on to be used in place of “snapmirror.conf” while you are doing the baseline initialization, and finally one to actually initialize the snapmirrors. There’s no intelligence around concurrent streams, so initialization is still a bit of a juggling act and waiting game.

Please let me know if you find this useful.

#!/bin/bash

echo “Please enter the name of the PRIMARY filer: ”
read PRI_FILER

echo “Please enter the name of the DR filer: ”
read DR_FILER

ConnectString=”ssh -c 3des”
#ConnectString=”rsh”

SnapMirrorHour=22
SnapMirrorMinute=0
SnapMirrorStagger=5

echo “” > ${DR_FILER}_filer_volumes.txt
echo “” > ${DR_FILER}_snapmirror.conf
echo “” > ${DR_FILER}_sm_initialize.txt
echo “” > ${DR_FILER}_snapmirror_init.conf

for AGGR in `${ConnectString} $PRI_FILER “aggr status” | awk ‘$2!~/State/{print $1}’`
do
 SIZE=`${ConnectString} $PRI_FILER “df -Ah $AGGR” | sed ’s/\([0-9][KMGT]\)B/\1/g’ | awk ‘$1!~/.snapshot|Aggregate/{print $2}’`
 for VOL in `${ConnectString} $PRI_FILER “aggr show_space $AGGR” | awk ‘$1!~/Space/{print $0}’ | awk ‘$4~/volume|file|none/{print $1}’`
 do
  echo “vol create ${VOL} -s none ${AGGR} ${SIZE} ” >> ${DR_FILER}_filer_volumes.txt
  echo “vol restrict ${VOL}” >> ${DR_FILER}_filer_volumes.txt
  echo “snapmirror initialize -S ${PRI_FILER}:${VOL} ${DR_FILER}:${VOL}” >> ${DR_FILER}_sm_initialize.txt
  echo “${PRI_FILER}:${VOL} ${DR_FILER}:${VOL} – ${SnapMirrorMinute} ${SnapMirrorHour} * *” >> ${DR_FILER}_snapmirror.conf
  echo “${PRI_FILER}:${VOL} ${DR_FILER}:${VOL} – - – - -” >> ${DR_FILER}_snapmirror_init.conf

  SnapMirrorMinute=`expr $SnapMirrorMinute + $SnapMirrorStagger`

  if [ $SnapMirrorMinute -gt 58 ]
  then
   SnapMirrorMinute=0
   SnapMirrorHour=`expr $SnapMirrorHour + 1`
  fi
  if [ $SnapMirrorHour -gt 23 ]
  then
   SnapMirrorHour=0
  fi
 done
done

This isn’t something I see very often, but when I do, it’s interesting to see the stats speak for themselves. I’m with a customer who had a scripted deployment of their NetApp estate a few years ago, and it wasn’t designed or delivered with too much care or attention (something I want to discuss another day). They have a VMware estate with SQL, Exchange and other things. It all runs across a total of over 100 15k FC spindles. It’s not a huge estate in comparison with other sites, so I’m intrigued into why they have such performance issues.

Now when you run through “sysstat –u”, you can see that the filer itself is doing very little, quite happily getting on with what it should do. But the disk is hitting 100% quite often. Immediately this shows a disk problem. They need more spindles, obviously?

Firstly there is an imbalance of spindles. They have a second aggregate on the partner controller that only has test volumes. I get permission to remove this and hot, I re-allocate these to the other controller and expand the existing aggregate. This doubles the spindle count, but I know it’s not going to do anything for existing performance (in that the data won’t automatically redistribute itself!).

If I run through “stats show disk:*:disk_busy” I can see something pretty obvious. There is a single disk in the entire system that is hitting 100%, the rest are not. There are a bunch of other disks (about 10), that are running 50-60%, and then the remaining disks ticking away at around 20-30%. So what has happened here? NetApp technology should prevent any form of hot spindle in the system.

My theory is this. The filer was racked and stacked out of the box, but the aggregate wasn’t grown (3 disk aggregate, 1 data, 2 parity). Some storage was provisioned and data migrated. They ran out of space, so grew the aggregate (a little), then copied a bunch more data onto the disks. After all this, they then added the rest of the disks. Now because data won’t automatically re-allocate on the fly, any data that remains unchanged (as will happen with VM System Disks, old Exchange emails, and old Data Warehousing data), then they are still sat on the original spindles or even spindle as when they were first installed.

So I’m now looking forward to the weekend. We’ll be upgrading them to Data ONTAP 7.3.2 and I can then run some reallocation scans across the system without impacting snapshot space usage (huge bonus, thank you NetApp!). I’m hoping that this will remove the hot spindle issue. I have some before stats, and I’ll pull out some after stats next week. I’ll update this post accordingly.

Lesson from the story? Setup your storage system COMPLETELY and thoroughly before you start throwing data at it. Don’t get excited about using your new storage toy and throw data on it immediately. I have seen the above scenario on several occasions now, and prior to ONTAP 7.3, it was a pain to fix.

Quick snapshot of the stats output. Keep in mind that across a cluster this will show all disks, so all disk stats are entirely relevant. The busy disks here just don’t add up to the actual number of disks in the system, and you can clearly see the one busy disk.

> sysstat -u 1
 CPU   Total    Net kB/s    Disk kB/s    Tape kB/s Cache Cache  CP  CP Disk
       ops/s    in   out   read  write  read write   age   hit time ty util
 11%    3220  6942  3270   4232      0     0     0    12   95%   0%  -  60%
 11%    2898  7385  4030   4892      0     0     0    11   94%   0%  -  69%
  9%    3547  1820  3496   3920     24     0     0    11   93%   0%  -  89%
  7%    2329  1160  3048   3892      0     0     0    11   93%   0%  -  81%
 10%    3173  2055  4851   4644      8     0     0    11   93%   0%  -  67%
  9%    2491  1860  4547   4568     24     0     0    11   91%   0%  -  98%
  9%    2523  2960  4404   5372      0     0     0    11   90%   0%  -  89%
 14%    5136  8173  4465   3352      0     0     0    11   95%   0%  -  81%

> stats show disk:*:disk_busy
… snip …

disk:88922F61:C2026AF9:E5D68A17:B49415B1:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:51%
disk:B677DDF1:8588AAF1:D71B2CFD:3C2BB6D4:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:34%
disk:08F03F36:067E5BD0:749AF2F3:429BBEC2:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:41%
disk:54DA4952:17856AC1:22FB0FCE:A283EF92:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:57%
disk:7CD76662:56D914F1:EC90930B:EC01EC2C:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:51%
disk:2777F697:BC2DE4B1:A6967344:3FF4F607:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:46%
disk:3BC9BA7A:D0085F2E:06DF43C8:5498E687:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%
disk:07EBF5F5:904AEAD8:03BAC8B5:CAF1F896:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:55%
disk:03AE8BA0:8D269AF9:4FF4F01B:B5BA8B7A:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:40%
disk:326B6A85:20FA7613:86E60929:2C2559B9:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:90%
disk:EFEDE851:60FD51E3:A7D780C4:0D4E1CC0:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:37%
disk:6A89A9E3:575C15F3:E369B326:CD51EBFF:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%
disk:A6840DBD:8D81796E:A2E9CF67:663F9940:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:0%
disk:6D37E864:7426A291:2C946571:F96E0F84:00000000:00000000:00000000:00000000:00000000:00000000:disk_busy:54%

… snip …

This basically entails growing a stately ‘tash for the month of November in order to raise money for Prostate Cancer charity.

Please sponsor me by heading over to my Mo-Space page – http://uk.movember.com/mospace/13540

My Mo-tash after 2 weeks

Related articles by Zemanta

• Hair Today…Cancer Gone Tomorrow? Make Movember Matter! (sassholes.blogspot.com)
• Movember: Sport A ‘Stache To Support Men’s Health (huffingtonpost.com)

It’s an interesting topic, and something that I’d like to share with you at this early stage in my own project as it makes you look at the storage aspects in a different light.

What affects a piece of data’s risk class?

1.  Who has access to it?
2. How confidential is it?
3. Does it have a tangible value?
4. How portable is it?
5. Could it potentially damage the business reputation?
6. Is it protected?
7.   … probably a lot more!

Some of these are all questions we already have asked about the data sets as we need to define snapshot, replication and tape policies, but data protection goes a lot further than just this. Interestingly the Zemanta plugin for my blog has linked “data protection” with “Information Privacy”, which is a key point!

Who has access to it?

Not just from a front-end authorised point of view, although you do need to know this. Payroll for instance, generally it would just be HR and Accounting that have access to this, but is there a mechanism for anyone else to gain access to it? If so, is there any audit control to check who has been granted access, or who has gained access? The audit control is almost more important than the security in the first place. Security can and will always be broken, but if you can prove it was broken, then you can fix it!

How confidential is it?

Most of us have a fair grasp on what is confidential and what is not. Employee data, Customer data, Payroll, Accounting, are all obvious candidates for highly confidential data. But other things are still confidential, even if they are not classified as highly confidential. External IP schema’s, low level system passwords, although they may be freely accessible by the technical teams, they are not available to the secretaries for instance, so this makes them confidential in some way. Are they marked as confidential? Other than applying common sense, have you ever told anyone that they shouldn’t email a domain administrator password around for instance?

Does it have a tangible value?

Very difficult one to play and the analysts will love this! But some things have a real immediate tangible value, purchase order or a signed contract maybe. There is scope for defining a cost scoring system against data, but it is very difficult to calculate. Something will cost money in very indirect ways, for instance if something damages the company’s reputation, it could cause loss of revenue. This should really be assessed in other areas and not necessarily spend time putting a tangible value on every piece of data (I want to finish my project this year!!!).

How portable is it?

With the age of Virtual Machines, portability is very important, and very dangerous. Someone can literally just walk off with an entire database system now on a portable hard drive! How do you protect against this? Is there any way to bind key systems, or police raw access to them? As much as technology and WAN speeds have come along, it’s still fairly unreasonable to assume you could email an entire system. However it is very easy to email spreadsheets and documents around. Preventing this from happening can be restrictive on day-to-day running of a business, so we fall back to auditing and monitoring. There are a lot of bases to cover, portable media, email, file shares, etc.

Could it damage the business reputation?

This is a good one, and not something you might immediately think of. Not just necessarily “dirt” on the business, but perhaps the business has a key technology or system that means they are unique in the market. If this is leaked to another company, it could damage the reputation as others could then start doing the same. Could the business reputation be damaged if the data were absent? If a key system was offline for a period of time, how would the business reputation be damaged (take a look at some start-up Cloud companies!). A damaged reputation could sink a company. Naturally company ethics and business practices are a good way of destroying a reputation. I have many friends that still won’t buy Nestle!

Is it protected?

And this is an amalgamation of all of the above really. The questions above help you to define the business value on a particular data asset. So if it has a high value, how protected is it? How protected should it be? How long can the business survive while it is being recovered? How much would data loss actually cost the company?

Depending on the risk class and business value will greatly affect the protection and auditing you deploy around it.

Putting this into action

I’d love to hear from people about how you put the above into action. We use NetApp ourselves (definitely practice what we preach), and this gives me a great level of control over my data sets and the protection we employ. Protection goes further than just snapshots and tape backup however; we need to protect it from more than just data loss. While NetApp have some great tools for protecting against data loss, there is a requirement to help with the other areas of data protection, and I’d love to see NetApp build on this space.

I have some experience using tools like Varonis, Acopia, Northern Storage Suite, NTP QFS, TekTools, to name a few, and these have all helped us in the past in deploying a complete solution. I have said on several occasions, and it’s something I really believe in so I’ll say it again; a complete solution is a combination of many different technologies that complement each other.

I’d like to revisit this topic again in a few months when I have progressed my project further, but I’d like to hear from the field to see what other people are doing to gain complete data protection.

New YouTube video showing some of the new features and functionality of NetApp SMVI 2.0. Not sure on a release date just yet, but looks promising!

I’m going to try produce more of these for various guides I do, they are much easier to understand quickly when you’re in a hurry.

The SnapDrive one here is a little dated, so please don’t hold that against the content.

What’s NetApp’s next play? NetApp were very excited a month ago about this potential aqcuisition, so it must fall as quite a disappointment. I heard someone (not from NetApp) voicing the possibility of a Quantum purchase. That would be interesting and very comical as EMC have a lot of that technology in their products! But other than annoying EMC, what benefits would it bring them? Quantum are in a lot of trouble at the moment, so it’d probably be a cheap purchase all the same.

I guess this can only be good news for the consumer though. NetApp will be driven to improve and excel with their de-dupe technology, and maybe give VTL a much-needed make-over.

Related articles by Zemanta

• NetApp Bows Out, Clearing Way for EMC-Data Domain Nuptials (xconomy.com)
• EMC Snagged Data Domain, So What’s Next for NetApp? (gigaom.com)
• EMC Wins Bidding War for Data Domain (nytimes.com)

Management Group

Chargeback

Generate Pre-Built Report

Volume Growth

User usage

Build a Custom Report

Scheduled Reports

Management Group

Creating custom reports from within Ops Mgr is relatively simple. The first step is to create some Management Groups so that we can run reports on certain areas of storage rather than across the entire system.

Next to Groups, click “Edit Groups”

Now give the new group a name. If some groups already exist, you can place the new group inside another if needed. Here we are going to add a new group at the global level. Then click “Add”.

The new group appears at the bottom, but you’ll notice the “Group Membership” reads “Empty”. Click on “Empty” so we can add some filesystem members to this group.

From the drop down on the next page, choose the volumes (or other areas if you wish) that you want to add to this reporting group (use shift or ctrl to select multiples) and use the “>>” button to add them. The page will refresh each time you click this, so it’s easier to select and add multiples. Here we have added all the VMware volumes.

There is no commit button or save button. When you click “>>” these are then added to the group. Over the left hand side we can see this new group, and any related warnings are passed down to it.

Chargeback

When you have management groups setup, you can then setup charging mechanisms per group. This is incredibly useful to help work out the TCO of a NetApp system and also obviously how to charge departments back for their usage (or even just show them how much their usage is costing).

How to calculate your cost per GB should be is outside of the scope of this doc, but should take into account several aspects of the filer management, user admin costs and also the basic costs of storage. To get a full costing all aspects need to be reviewed and adjusted for (power, cooling, space, admin costs, management, and so on). For this example we are simply using 5 per GB (although 50 may be closer to reality on a large system). 5 can be $, £, or whatever you want. Because the £ symbol is not a standard ASCII symbol, we have to enter the HTML character code for it. This is “£”. The currency can be set in “Setup”, “Options”, “Chargeback”. You can also set the defaults here.

Got back into “Edit Groups”

And choose the group we have just setup (here we setup a new group for the User Home Directories).

At the bottom of the page we have an “Annual Charge Rate (per GB)” and you’ll see the default we’ve now set. You can alter this for different areas, so VMware may have a different cost to users home directories. Or FC disk may have a different cost to SATA disk.

Chargeback works on averages, so if this is a fresh system or you have just initialised the quota’s, then this may not give any useful information just yet.

Generate Pre-Built Report

Volume Growth

Now we can create some reports based on this. Click on “Reports” and “All” and we can generate a pre-built report based on our new management group.

We want to create a growth report here, so on the left hand side, under “Logical Objects” choose “Volumes”.

Then choose “Volume Growth”

At the bottom under “Using Resource”, click “Browse”.

Then under “Resources” and “Groups” drill down to our newly created group. Just select the top level group and not the individual members. OK this.

The selection a the bottom changes and click on “Show” and we’ll get our volume growth for this management group.

The below example had only recently been setup, the growth rates have not had chance to be registered.

User usage

For this to work successfully, you need to have some quota’s setup on the users home directories. These don’t have to be hard quota’s, in this example we will be setting up some soft quota’s on the filer, and then reporting against this.

From FilerView, goto “Volumes”, “Quotas”, “Add”.

Here we have a single home directory volume and we are going to add a default User quota based on this.

In this particular volume we have a qtree setup for “normal” users. In this example this is actually just one qtree for all users, but this can be setup to report on multiple qtrees if needed.

For this example here we don’t actually want to use quota’s, we simply want to report against our users. So we are going to use a soft quota of 5 GB. I have also put a soft quota on 50,000 (50k) files. All other entries are left blank to prevent any hard limits or other limits being enforced.

Remember to enable this quota we need to go back to the manage page, check it, and click “on” at the bottom. If it has already been enabled, click “resize” to reload the quota details.

The filer will now scan the volume, if it is large it may take some time…

Once this has complete, verify that it is working as expected by looking at “Quotas” and “Report”. We can see the soft limits working in place for our example here.

I also have setup a “Tree” quota, again with soft limits, on a departmental share with separate Qtrees for each department. We can see the results of that quota below.

Back in Operations Manager. By default the monitoring period for Qtree’s and Quota’s is 8 hours and 1 day respectively. If we are demo’ing this, or setting this up for a customer, we may want to reduce this to allow us to show the immediate effects.

Goto “Setup”, “Discovery”.

Then goto “Monitoring”

And update the Qtree Monitoring and User Quota Monitoring intervals. Here I have changed them to 5 minutes and had a quick coffee break.

We will need a new management group this, so I have created a new group called “User Areas” and I added in the Qtree “/home/normal” into the list of members.

We can check that this has successfully picked up our new quotas and is recognising the users by looking at the “Group Status” and then “Quotas” tab.

We now see a list of all the users and the status of their level. Users with green status will be under quota, those with yellow will be over quota.

Now we can finally report on this usage! Under “Reports”, “All”, then under “Monitoring” choose “User Quotas”.

Choose “User Quotas, All” and make sure you are “Using Resource” for the management group we just setup.

This report is almost identical to the one shown within FilerView, but it is a little easier to understand and is sortable.

Users that no longer exist on the domain will appear with their SID rather than their username.

The limitation here is that Quota’s are entirely driven by ownership. So if a user does not own their home directory, then the above process will not work, you will find that “Administrator” owns a large proportion of user files. This should be addressed as it is best practice to make sure all users own their own files anyway.

Build a Custom Report

If we want to build a custom report, maybe comparing 2 data fields that don’t normally exist on the same report, we can do so. Under “Reports” click “Custom”.

You’ll be taken through to the “Create a Report” page by default. There is a veritable banquet of options for custom reports, so have a careful think before simply diving in to create or show this feature off. It may be best to base what you require on an existing report. Most things are available in the pre-built reports, so it may be that you just want to customise this slightly.

For this example, I want a single report to display the username, disk space used, files used, chargeback, daily growth rate % and days to full.

Whatever report you create must have at least one field taken from the “Base Catalog”, so choose the “Base Catalog” that best suits what you are trying to report on. For this example it is obviously going to be “UserQuota”.

The display tab is quite important as this shows where in Operations Manager this report will be visible from a drop down menu. When you are different pages, Operations Manager gives you different lists that you can choose from. Again, on this occasion it makes sense to include this report in the “Quotas” tab.

Next we need to start defining what information we want to include in this report. So from what I defined at the start, I will start filling out my report. Remember you need at least 1 field from the top level.

Daily Growth Rate %
Days to Full
Disk Space Used
Files Used
Username (note I have to drill down a level to get this info)
Chargeback

Then I need to order this in the way I defined for my report.

And finally click “Create” and we see our new Custom report at the bottom.

To see this report in action, go back “Home”, then click on “User Home Directories” (my custom group), “Quotas” and chose this from the drop down “Report” at the top right.

As mentioned earlier, there are some blanks in this as the report hasn’t been collecting data for long, so there is no data for the growth rate or the chargeback amounts.

If you want to tweak this report, you can go back into “Reports”, “Custom” and select “Edit” from the list at the bottom of the page.

Scheduled Reports

Once you know which reports are useful, or perhaps pulled certain aspects out of different reports to create your own custom one, then you may want to schedule these reports and get them emailed to you. Goto “Reports” and “Schedule”

First we want to create a schedule, so click on the “Schedule” tab and click “Add New Schedule”

Create a schedule based on what your requirements are. Here I want a report generated every Sunday at 8pm.

Now back to “Report Schedules” and we can “Add New Report Schedule”.

From here I can now create my Scheduled Report and the email recipients. I am going to get my new custom report (User Custom Usage) to run against my new management group (User Home Directories) to get generated per my new schedule (Weekly Sunday Evening) and emailed to myself. I’m going to keep the standard HTML formatting and all the other standard settings.

Make sure that a valid mail server has been configured before generating any reports. Goto “Setup”, “Options”, “Events and Alerts”. Here I have also updated the purge interval as I don’t want alerts older than 1 week.

To test this schedule, I want to run it now. Check the checkbox of my new schedule and click “Run Selected”.

The report will be attached in a zip file.