[This was first posted to Technorati on February 6th, 2012. You can read the original here.]

I keep hearing the term “Big Data”. My old-school IT instincts always makes me think of inefficiently storing two-letter state abbreviation codes inside of an nvarchar(50) field because that’s the default data type you get when creating a database table from inside Visual Studio. But it turns out that Big Data means something much more than that.

Big Data means big profits. It means predictive analysis. It means a shift in how we process data. It means that as database professionals our skills are going to be needed for years to come.

It means everything.

Check out this article from Popular Science magazine that discusses how the city of Santa Cruz is using data in order to better prevent crime from happening (yes, almost like “Minority Report”). This is not your ordinary “business intelligence”, this is the next stage of development. This is “Business Analytics“, and people everywhere are going to want to have trusted experts that can help them reliably mine their data.

Want to know when and where the next crime will be committed? How about how the spread of a disease from city to city? Or finding a way to help traffic flow? Don’t believe that Big Data can help us predict such things? Then check out this book: Linked: How Everything Is Connected to Everything Else and What It Means. Or take a few minutes to learn about the “3 Vs” of Big Data (Volume, Velocity, Variety) from Edd Dumbill and how more companies are using these concepts to gain a competitive edge.

Turns out the concepts behind Big Data is not so new. Researchers have been trying to find ways to connect data for decades. Even the idea for hyperlinks goes back to 1945, a full ten years before Tim Berners-Lee was even born.

Think about all this for a moment. We have been trying to find ways to link data together for over 60 years. Big Data is not new. What is new are the shiny tools we are being handed each year to mine the data. And why are companies investing in building us shiny new tools? Because they know the market for business analytics, and Big Data, is quite vast.

And there will also be a market for folks who know how to use the tools. Start investing in some training now, it will pay off for you later. Those folks who shun such concepts as Big Data, or the “cloud”, are going to be left far behind. And at the rate changes happen these days, you may find yourself left behind sooner than you think.

You could be here!

Do you enjoy working with SQL Server? Do you enjoy relaxing in wine country? Would you like to work with SQL server AND relax in wine country at the same time?

I know I would. And so would Denny Cherry. In fact, Denny liked the idea so much he decided to create SQL Excursions, a series of events where people can get together to learn about SQL in a relaxing manner. And I was honored that Denny asked me to be the first guest presenter with him for the very first SQL Excursion coming up in Napa, CA from May 17-19.

Check out the agenda for the event. If you want I can summarize it for you here: #SQLWINNING

Performance Tuning Made Easy - This will be similar to the half-day session offered at PASS that had 250 people in the room and another 100 that were turned away. If you were one of those 100, you will want to attend as I plan on taking a deeper dive into how to help you be more effective in your current role, something I cannot do when speaking to hundreds of people at once.

Storage Configuration and Tuning - Denny Cherry is one of the top experts in the world when it comes to storage configuration and tuning and a Microsoft Certified Master in SQL Server. Come listen to him teach you some tips and tricks with regards to storage, which will lead nicely into…

Virtualization and SQL Server - …Denny and I sharing our experiences with SQL Server running in a virtualized environment. We expect to not only show you some examples of VMWare and Hyper-V, but we are also planning on having you put your hands on the products as well. We want you to leave Napa feeling comfortable navigating your way through a virtual environment.

SQL Server Indexing Top Down - Denny will walk you through indexing and SQL Server, discussing strategies for how you can tune queries using proper indexing and how to conduct proper index maintenance.

How Table Partitioning Works to Save You Money - Denny has been helping clients for years save money with some very simple partitioning techniques. Denny will help you understand how to get this done for your shop as well.

Hands On Troubleshooting - This will be similar to the series of “Choose Your Own Adventure” talks that I have been participating in recently. We will walk you through real scenarios where you will need to troubleshoot using native tools only. This shared learning experience will provide you a solid base for how to troubleshoot the vast majority of performance issues in your own shop.

OK, now comes the important part: go here to register. Don’t forget we will have wine tastings, wonderful weather, great food, and an opportunity to network and build relationships with your peers that will serve you well going forward.

I can’t wait to see you there!

Never eat alone!

Today I did a guest post over at the SQL Server team blog and you can click here to go read it for yourself. Go ahead, I can wait.

Done?

Good.

I listed out three things that you can do right now to jumpstart your career. Of course there’s more ways than just those. Some of my favorite ones include: writing, speaking, and never eat alone.

If you are not writing (or speaking), get started. The simple effort of improving your communication skills will serve you well and I always finds leads to greater job satisfaction. No, not for that lousy job you have now, but for the next job. You know, the one that will be better because they need someone with good communication skills. Yeah, that one.

And never, ever eat alone. Check out this email I sent back in 2004:

—–Original Message—–
From: LaRock, Thomas W
Sent: Thursday, October 21, 2004 2:11 PM
To: Kinsel, Allen; Wright, Patrick
Subject: hermits

OK, I’ll be the first one to send off a quick email to the others. No reason for all of us to be anti-social.

It was nice to meet both of you while at PASS this year. I hope that we can stay in touch from time to time.

I received an email requesting some info about myself for volunteering for PASS. I decided to walk the walk and sent back some info so I should be getting involved with a committee or two of some sort. Hopefully we will be able to stay in touch through PASS, if not through regular email.

Best,

Thomas LaRock
Database Administration Manager

Had I chosen to eat alone at the 2004 PASS Summit I am certain my path would have been very, very different.

Next Monday is the first Monday of the month, which means it is time for your writing assignments.

The topic this time around will be: working with deadlines.

Think about any tips or tricks you have with regards to working with (or around) deadlines. Your stories and experience with getting stuff done with only minutes to spare would be great to share with others.

Most of my best stories about getting things done with only minutes to spare have to do with coaching and I really couldn’t put into words on a blog so you’ll need to stop me sometime and ask me about them in person. As far as being a database professional, the tip I would give you is this: work backwards.

If you have a deadline, you likely have a project of some type. If you have a project of some type you likely have details and deliverables that make up the project. If you know the details and deliverables you likely will know the things you need in order to get those details done and delivered. If you know the things you need to do, then you likely know how long they will take. And all of that means you should be able to figure out when certain things need to get started, and finished.

Working backwards has always helped me stay on top of such things, and I would encourage you to try that as well.

This tiny guy runs everything you do!

Ever try to troubleshoot something that doesn’t really exist?

I think this virtualization thing is here to stay, I really do, but the trouble I see people having with virtualization is that they don’t know where the bottleneck truly lies. There are so many layers involved that most standard troubleshooting techniques fall short. Just think for a minute about being a SQL Server DBA in a virtual world: you administer a piece of software, running on a guest O/S, that is running on a host, that is connected to a datastore that may or may not be shared with “noisy neighbors”.

Knowing all that, when an end-user comes to you and says “this query is running slow”, where do you begin to look?

I always start with the virtual layers. I’d rather spend 5 minutes ruling out the virtual layer as a bottleneck before I try to tune a query. It makes no sense to spend time trying to make a query run faster if the bottleneck is at the host level, for example. That’s wasted time, and I *hate* wasting time.

One example I see often these days has to do with CPU pressure. In my experience most database servers are not CPU bound. (I said “most” and “my experience”.) Usually a database server has a lot of extra CPU cycles to spare. But that is not always the case in the virtual world. So when I come across a database instance that is showing CPU pressure I like to ask a very general question: Is it internal pressure, or external?

One of my favorite metrics to examine when researching potential CPU pressure is called signal waits. You can read more about signal waits at this blog post by a member of the SQLCAT team. The default threshold I like to set is 25%, meaning once I see I have about 25% of my sessions simply waiting for an available CPU I am likely to believe that the CPU pressure is internal to the instance. Of course more factors come into play before I would define 25% to be a problem, such as volume of sessions, length of time for each session to execute, etc.

But what about a guest O/S that is also waiting for an available CPU from the host? Imagine that your sessions are waiting for the SQLOS to give you an available CPU, and the SQLOS itself is waiting for an available CPU from the host? Wouldn’t it be nice if there was a metric for you to know if the issue is at the guest layer, and not internal to SQL?

Well as luck would have it such a metric does exist: CPU ready time. This is simply a metric that tells you the percentage of time that your guest was ready to run, but could not get scheduled.

Together the signal waits and CPU ready time help to give you a more complete picture of what is happening to your database instance. Unfortunately, you can’t get at all of this information easily from within vSphere. Here’s a quick overview of what you would need to do in vSphere to research CPU pressure:

1. Open vSphere, then go to the correct host, then to the guest that you are interested in monitoring.
2. On the Performance tab add in the counters you want (but no more than two!)
3. See real-time information (too bad if your issue was last week and you don’t have details any longer)
4. Ascertain if the issue is local to guest by going back to the host and looking at Performance tab
5. Add in the counters for the host Performance tab view (but no more than two!)
6. See real-time information (too bad if the issue was last week and you don’t have details any longer)
7. Throw up your hands when you realize that you still don’t know if the pressure is internal or external to SQL running on that guest, on that host
8. Take your hands and slap yourself when you realize you can’t tie anything back to specific SQL statements

Sounds rather time consuming, right? Especially if you still won’t have any idea about a root cause or what specific actions to take after spending all that time investigating the issue. If you want to see it for yourself I made a video to show you some of the challenges that vSphere has when it comes to displaying all of the metrics necessary for you to make an informed decision.

The short of it is that vSphere is never going to give you a complete view (including SQL statements or even number of sessions running) in one screen. But IgniteVM will do that for you, saving you time, and ultimately money as well, money that can then be used to buy more bacon related products for your home or office (don’t forget Valentine’s Day is right around the corner).

Not only is virtualization here to stay, but tools like IgniteVM will be here as well to help administrators quickly find the root cause for performance bottlenecks.

Want to chat more about monitoring your virtual database servers? Then contact me and I would be happy to set up some time to talk. Also, Confio has regular product demos: go register for the next one here.

Since I wrote just last week about interview questions for a DBA, I came across this article and wanted to share it here also. I don’t believe that an interview should be a trivia contest. These questions aren’t about knowing a trivial piece of information. They are (probably) designed to illicit a response that helps everyone to understand your thought process.

http://mashable.com/2011/12/27/glassdoor-interview-questions-2011/

When it comes to a job interview my advice is this: don’t panic. Be yourself, have a conversation, and let people know who you are and how you think. If you pretend to be somebody else and get hired the end result is often a poor fit.

In Seattle this week, so naturally coffee is on the menu and I felt the need to share the following information:

• Caffeine is “technically” lethal, but you would have to drink an insane amount to have a 50% chance of death
• There is no reason to ever stop drinking coffee because it is pure awesome
• Caffeine helps improve your memory
• Caffeine can help fight cardiovascular disease, allow me to consume more bacon

Also, apparently the drink of choice used to be beer and gin, but then people started to consume more coffee and tea. So what will be the next drink of choice? I’m hoping it’s wine.

Because the answers might surprise you.

An interview should be a conversation, not a trivia contest. I’m not looking to stump someone during an interview. I don’t need for my DBA to know obscure facts such as database mirroring is only supported starting with SQL 2005 SP1.

Memorizing such facts does not mean that you will handle the pressure of being a production DBA. The role does not require you to know the answer as much as being able to find the answer. And in order to truly be successful as a DBA you will need good communication skills. A good DBA will be able to teach, and train, others.

So how do you find such a person? The answer is easy: you talk with them. I always describe the interview process as being a lot like dating. You want to have a conversation, get to know someone, without assuming anything beforehand. Since people tend to embellish on their resumes, if you aren’t familiar with a person or their previous work then you are going to want to set a baseline. For me, that baseline are the following five questions that I have used over the years. I find that they help guide me through having a good conversation and by the end I know whether or not I want to call them again in the morning.

1. What is a database?

I always like to start with the basics. While it may seem silly to ask this question keep one thing in mind: there are no silly interview questions. No matter how ridiculous a question may seem to both the person asking it and the person responding, they will always serve a purpose. In this case you may find the candidate talking about MS Access, or Filemaker, or whatever they have been exposed to previously. Let their response be an indication of their fundamental knowledge of some RDBMS system. See if they can explain the difference between a data and a log file, for example.

Remember, people tend to embellish on resumes. If you have never met the person before, why not ask the most basic question you can, right from the start? The answer may surprise you, but the conversation is likely to reveal the depth of their experience right from the start.

2. Who is the most important user of a database?

This question helps you get an understanding if they candidate knows who the “stars” are. And who are the stars? Well, everyone. That’s right. Every connection to a database server is as important as any other. What’s that? You say it is only a development server, so it is not that important? Well, a development server is considered a production server to a developer. In other words, every person and every connection is important, no matter if they sit in the corner office or not.

I once had a candidate tell me that the CEO was the most important user, and they were adamant that they were right. Sadly, most CEO’s aren’t doing the day-to-day data entry that others are doing, so I found it hard to agree that such an absolute answer was correct.

If the candidate lists out only a handful of people or groups as important and fails to understand that everyone is important, consider that a red flag.

3. Which is faster: Inserting one million rows of data, or updating one million rows of data?

This could be one of the more technical questions you could ask a candidate. You are not looking for the correct answer (which is “it depends”). If they do answer correctly then examine the scenarios they lay out before you. It should be very insightful, you should get a real sense for how they think. Chances are they will guess at one or the other, which gives you the chance to talk with them. And when you do so, take the opportunity to see how the discussion goes. If they ask you lots of questions, that is a very good sign. If they just sit there, listen to your every word, accept everything you say as the truth, then consider that another red flag. No one is perfect, not even you, and you do not want to surround yourself with someone that just follows your every word. You need someone that is insightful enough to ask you more questions, you want someone that is always looking to learn.

I recall asking this question once and was told that it was a trick question because you couldn’t update (or insert) that many rows at once, you could only do them one at a time. Yes, that’s right, they told me that everything was done through cursors, and didn’t understand the concept of set-based updates.

4. If I asked you to learn how to make a query faster, what would you do?

Does the candidate look like a person that shows some ambition? Do they read any web sites with regularity? Have they ever opened a user manual, or the BOL? Who do they consider to be an expert? Do they have their own library of books, or their own toolbox of scripts? How do they typically attack a query with regards to performance tuning?

If you need a DBA that can help troubleshoot poorly performing code, and to do so quickly, then you want to make sure they aren’t the type to panic when the pressure is on. In short, find out how would they start to look for answers. Which also happens to be a great segue into the final question…

5. How do you troubleshoot problems in your current role?

The ideal candidate will be able to clearly explain their thought process in how they troubleshoot something. Developers will most likely discuss with you some piece of code that worked well on one server but not on another. A Windows admin might talk about a service that stopped and would not restart no matter how many times they clicked on that little arrow. In either case, you want to identify someone who is articulate to a certain degree, can explain themselves, stand up for their decisions, and also be open to the possibility that there was more than one way to solve the problem at hand.

You need to know more about their thought process when it comes to solving problems. If their troubleshooting skills consist of “call the vendor for help”, that won’t suffice. You don’t want someone with all the answers (and such persons don’t exist anyway), you want someone that has a clear and defined thought process.

In addition to the questions, and the conversation that ensues, you are also going to want to evaluate the candidate in other areas. For example, you can look for certain things that would have the candidate stand out among the others, or where they fail to measure up.

And there you have it, my five questions to ask of any DBA in an interview. You can alter these to fit your needs for whatever role you are trying to fill, just keep in mind that you want to frame everything in the form of a conversation. After these five questions you can move into more targeted questions with any candidate that has done well.

If you can follow this outline and have a good conversation then you are going to be able to identify the right candidate.

I’m normally not one to advocate the use of the Database Engine Tuning Advisor (DETA), a tool available inside of Microsoft SQL Server. (To be fair, I’m not a fan of the SQL Tuning Advisor from Oracle, either.) But I recently came across a little nugget of information that I wanted to share with you regarding an improvement in the DETA for SQL2012. You may not know it from clicking on that link, so I’ll spell it out for you here:

“To tune a database by using the plan cache”

And suddenly I am interested in using the DETA again. This makes things so much easier for me as an end-user. I no longer need to configure the DETA to run at a specific time. I can simply point-and-click and instantly have a handful of recommendations to evaluate.

Of course taking action on the recommendations without doing any analysis is not a recommended course of action. But the idea that I can use the plan cache is something I am eager to try.

Here’s hoping I won’t be disappointed…again.

Trust me, I'm a DBA

“Signs point to yes.” – Magic 8-Ball prophecy

You have probably heard this line at one point, the one that says “the best DBAs are the ones you never see or hear?” I used to think that was a result of some sketchy personal hygiene choices and later on understood it to mean that if everything is running smoothly then there isn’t a need to see your DBA. It’s kinda like how you never see your plumber. Oh sure, you could call him (or her) and chat, have them over for drinks or dinner one night, but you never bother talking to them until your toilet backs up and there’s crap everywhere. That’s how most DBAs are treated as well, we don’t get called until after the crap is on the floor.

Unlike your plumber most DBAs are performing regular work on your databases. Much of this work will go unnoticed. The reason for that is simple: you’ll freak out if you knew what we were doing. And if you ask us about something we are likely to tell you half-truths because (1) we don’t want you to freak out and (2) you never understand us when we talk to you anyway.

Here’s my list of things that I am willing to bet your DBA hasn’t always been truthful about.

1. Your database server is still on physical hardware

Probably the number one misconception that end users have these days centers around the use of virtualization. People fear change, and when they are told that their servers are about to be virtualized they freak out. And for the handful of users that don’t freak out immediately their reaction is just delayed until the moment that something goes wrong and then they blame the fact that the server is on virtual hardware. What’s that? The bank didn’t send the files last night, so the import never started? It must be due to the fact that we were virtualized, because before then nothing like this ever happened.

Wrong. Bad things happened before you were virtual, too. It’s just that you don’t remember. So we don’t bother telling you about the switch. No, we wait until it leaks out months later, just so we can say things like “look, it’s been on a VM for six months and you never once complained about performance, so don’t try blaming virtualization for your issues now.”

2. Everything is fine, nothing to see here

Chances are everything is not fine. In fact, chances are the data center is on fire and your server melted last night (good thing they were virtualized so we could failover immediately to a different data center without you knowing) but we aren’t going to say a word to you about it. And why not?

Is it getting warm in here?

Because the minute most users hear about something having gone wrong they like to blame EVERYTHING on whatever went wrong. Take this example:

Me: “Hi Craig, just want you to know that last night at 11PM the differential backup failed. We’re not sure why yet, but we re-ran them at 11:30PM and those worked. So, if you ever need something from Monday night and ask for the 11PM backup and you’re told that we only have one for 11:30, you don’t need to panic.”

Craig: “The backups failed?”

Me: “Yes, but we re-ran them right away, so we have a good one, just from 11:30PM, and not 11PM”

Craig: “OK then, that explains why those reports at midnight didn’t work.”

Me: “No, it doesn’t, the backups have no bearing on those reports. It’s two different servers. Your report server melted in the fire. That’s why they didn’t run. This is a different server, and a different time. There is no relation between the two.”

Craig: “But the reports use data from that server. Chances are the data they retrieved caused the fire.”

Me: “Is it too early to start drinking? I guess not, because clearly you have already.”

That’s why we don’t tell you anything. It’s because no matter what happens you will latch onto it and start blaming it for anything else no matter how unrelated or crazy. You’ll send emails to managers stating “facts” that are just not true, creating more chaos and confusion than if we had said nothing at all.

So we don’t say anything. No matter how big or small of an event.

3. It’s the network

Your DBA is likely the best looking and smartest person in your IT department. But that doesn’t mean that they are omnipotent. From time to time they are not going to have an immediate answer for you no matter how long you stand in their cube tapping your feet and reminding them how much money is being lost for each passing minute that they haven’t solved whatever imaginary problem you are having.

You should have seen it BEFORE we consolidated.

It’s during times like these that the most seasoned of DBAs will reach into for the modern-day equivalent of a “get out of jail free” card by turning their heads, looking deep into your eyes and saying those magical words:

“Looks like it could be a problem with the network.”

This one line is pure gold. It buys any DBA the time they need to investigate the real problem as most folks will walk out of the cube once they believe the DBA is on the trail and closer to finding a solution. And once those people leave you alone to focus on the issue you are likely to be able to find a solution.

And don’t worry about blaming the network. The network admins don’t mind, because they just blame management for not buying the right hardware the first time around. And then management tells the end users they will need to live with the bad performance until the network issues are worked out. It’s the circle of life in IT, and it works.

4. Of course your server is dedicated

Not likely, no it is not. Not only does your database server also run all sorts of other things like anti-virus software and software for taking server tape backups, but chances are it is running other services like SSAS or SSIS. And let’s not forget all those vendors that insist their application be installed on the same server as the database “for performance reasons” which I always found amusing every time we found the performance bottleneck to be the application itself. Good times.

And let’s not forget the price of all that hardware and software. As much as I enjoy having my overall CPU utilization to be less than ten percent if the guys who write the checks find out that we are wasting ninety percent of our capacity then there is a good chance we won’t ever be able to buy anything new and then you’ll have to hear your DBA say things like “this is why we can’t have nice things” each time you ask if your server is dedicated. So, just don’t bother. Your server is a resource and is likely being used by others. Get used to the idea.

5. Yes I gave you the permissions you asked for

Not only do vendors like to insist that their applications are installed on the same server as the database, but they also insist on having full administrative rights. Why are such rights necessary? Quite often, they aren’t, but vendors ask for it because they can. Any good DBA will ask for a list of actions that are needed so that they can grant the permissions necessary for those actions. I once had a vendor tell me they needed full system administrative rights for the database instance AND the O/S because they “had to create tables and stuff”.

DBAs, especially the good ones, know this isn’t true. And so we give you the permissions that you need. Quite often that is different from the permissions that you asked for. And honestly we’re too tired to explain to you the difference because you usually don’t care to understand it anyway.

There you go, five things your DBA has probably been less than truthful with you about. That doesn’t make us bad people. In fact, it makes us more like an adult…say a parent…someone who knows the difference between “want” and “need” and has to make the hard choices that others don’t want to make.