Showing posts with label DBA Task. Show all posts
Showing posts with label DBA Task. Show all posts

Wednesday, September 10, 2008

The DBA Routine

By Andy Warren, 2007/11/29

As a trainer that focuses on beginning to intermediate students it's always interesting to see which questions and concerns come up again and again across classes. One of the items at the top of the list has to do with the fact that most DBA's work alone, that is, with no peers. That often leads them to worry that they aren't doing everything they should be, especially if the nature of the business precludes using all of the published 'best practices' we hear so much about. For those that ask the question the training is a much a 'health check' as it is a learning experience! An article won't replace the value of having a peer or attending a class (I hope, but I'm biased!), but here's my list of what I think should be top on the list for DBA's.

#1 - Backup. Job one is to back up the data and be able to restore it when needed. That means a full backup at least once a week and if possible every day, and log backups in between at a decent interval, probably 5 to 30 minutes. Whether you use the maintenance plans or write your own, there is nothing more important than guaranteeing you can provide a copy of the data if the worst happens. The hard part is that important as this task is, most of us set up the job and at most monitor it. Entirely human and pragmatic, but it's worth checking up on manually once in a while. It's also important to do the occasional point in time restore so that you're ready to go if the need arises. Not many things will get a DBA fired but not having a backup is one that will.



#2 - House Keeping. I'll talk more about time management in a moment, but the key to sanity as a DBA is to set aside 30-60 minutes per day for house keeping. For me it's being at the office 30 minutes before the bulk of the employees/users starts work to make sure the servers are up (alerts are good, checking is better), then I check all my nightly jobs, email alerts and notifications, event logs, and free space. During this time I also review my calendar for the day and remainder of the week, and then a few minutes on the daily version of my professional development plan reading various newsletters and blogs, and sometimes listening to the Voice of the DBA. Don't let anyone deny you this time. You may have to reschedule it on rare occasion, but just force it to be a routine and usually the business will accept it. Sometimes during big projects and periods of stress you'll be tempted to skip house keeping - don't! It's the path to the dark side. I would argue that this item is the primary reason you collect your paycheck. Better to work an extra hour to get it done than to skip it.

#3 - Best Practices. Best practices aren't one size fits all. If you're in an environment where best practices don't seem to fit, find a peer to discuss it with. See if you can convince them that the alternate practice makes sense in the context of the business. For example, recently I had a student working at a very specialized company where there was a good sized group of non DBA's that had the ability to create tables within certain databases. Definitely not a recommended practice but based on the situation it was a valid solution. Note that I'm not recommending you disregard best practices. Just understand that the realities of the world don't always make it possible to implement them all.

#4 - Security. This is a little bit about best practices, but it's also about common sense. Only DBA's should have sysadmin access, and no one should be using the actual SA account. Assign permissions to roles and put users in roles. If at all possible grant permissions on stored procedures only. Avoid use of the Public role.

#5 - Gatekeeper. This is the main time that DBA's are visible to developers and the rest of the business, when a change needs to be deployed. Change absolutely has to be controlled and tracked. As a DBA it's our job to make those changes within a reasonable amount of time and to let the requester know when the change will be made. The 'reasonable time' portion of that is one place where good communication and setting expectations is important. We don't want non DBA's making changes because if something goes wrong, we need to know very clearly what recent changes have occurred. Logging can be as complex as checking deployment packages into source control or as simple as saving the change script in an email folder. A DBA should always be able to answer the question "what changes have been made recently?".

#6 - Baselineing. The least fun time in the life of a DBA is when a major performance event occurs. You start Profiling and Perfmoning with the boss anxiously looking over your shoulder, and then you see a statement that takes 10,000 reads. Boss asks "does it always take that many reads" and your not so exciting answer is "I don't know". Capturing a weekly or monthly baseline using Profiler and Perfmon gives you a way to research that question. Not just to give the boss an answer, but to decide if that's the right item to spend your time on right now. It's also useful for assessing how your environment has changed, answering questions like "how many connections did we average from the application a year ago?".

#7 - Not Just a Gatekeeper. It's not always possible for us to look at stored procedure and tell a user that it won't return the correct answer. It is possible for us to make sure it conforms to our coding standards (requires tables to be schema qualified, no goto, no cursors, comments required, etc) and to just apply the "is it stupid" test. Beyond that, the biggest mistake I see DBA's make is not also assessing the performance implications of the change. If it's a change to an existing stored procedure I want to see what the current one costs versus the new one. A change of more than 10% might indicate it needs tuning, or that I need to work on the query plan and indexes to maintain performance at current levels. If you don't look at every single change this way you'll eventually suffer performance issues due to the overall entropy that results. It's a little more work, but it's absolutely worth it.

#8 - Customer Service & Consulting. Think of yourself as a service provider. Businesses pay you to keep the database servers running and the data backed up and secure. They want you to do that without impeding their ability to do things that will lead to revenue generation. This sets up a natural tension between your job to do the above and their need to move fast. Sometimes it means you have to modify or abandon a best practice. If the business doesn't make money you don't have a job. If you lose the data you don't have a job. Try to avoid extremes and understand the needs of the business and you'll go further (and make more money) than those that see the world in black and white. Get involved with your developers and remember that their job is at least as hard as ours, and they are just trying to do what they were hired to do (solve problems) as best as they know how. Take the time to understand their challenges and to help them understand yours, and you'll have the beginning of a dialog that can lead to some very good results. As a DBA I try to become the internal data consultant, not just the guy who does the backups.

#9 - Learn Beyond. I meet a lot of DBA's that are very successful in their current job but I think would struggle to get the next job. Why? They are working in a specialized environment and aren't maintaining all the general skills they need. For example, I see DBA's that express no interest in connection pooling and how it affects them because they only use off the shelf software that can't be altered. Or that don't want to learn replication because their employer doesn't use it. It's definitely hard to learn some things without a project to drive them, or having the right hardware in some cases, but don't become a one trick DBA.

#10 - Whack a Mole. This is my secret to success, I put it at #10 so only the most curious will learn it! Once a week run a couple Profiler sessions, one to capture queries with more than x reads, and another to capture queries over x duration. I start at 10k reads and 10 seconds. Some of these will be jobs and similar one off occurrences, but the rest represent tuning opportunities. Add them to your list and set aside an hour on Friday afternoon to try to get one of them to run faster. Eventually you'll get all queries below the threshold. Then it's time to reset the bar at 9k reads and 9 seconds and go again. This forces you to once a week look at queries that perhaps used to perform ok but as the data has grown things have slowly declined, and fixing them helps maintain a steady load on the server. Combine this with #7 and you'll have a lot fewer performance issues.

It's not as easy as just doing my 10 steps of course, but if you're doing all that stuff you're not in bad shape. Build yourself a formula, stick to it, and reevaluate it once a year to see what you need to change. I look forward to your comments.

DBA Morning Check List

By Bill Richards, 2008/04/14 Database Administrators can sometimes have one of the most stressful jobs in the company. If you have been a DBA for long, you know the scenario. You have just sat in your chair with your cup of coffee, and your phone starts ringing off the hook. The voice on the other end states that they can't pull up their data or they are getting timeouts, or the system is running slow. Okay, time to dig in; it's going to be one of those days! Is it Friday yet?

In this article, I will present ways to minimize those stressful days by having a pre-defined DBA morning checklist. A morning DBA checklist is a document of pre-defined administrative checks that are performed every morning to ensure that your server is at optimal performance. By having a standard list of items to check, you are more likely to catch and fix issues before there is a real problem.



The end result of the morning DBA checklist should have three sections. Section one contains the list of items that need checked. Section one should include checks from the following categories: performance, job failures, disk space, backups, connectivity, and anything specific to your environment, such as replication, mirroring, clustering, etc. Section two contains a place to write down issues and how they were resolved. The third section is a confirmation section where it is signed and dated. The third section is very important. Without this section, it is difficult to enforce and guarantee that these checks were performed.

The first step to create an effective morning checklist is to meet with all the DBAs and ask them these questions:
1. What do you check in the morning?
2. How do you check it?
3. What do you do when there is a problem?
4. Is there anyone you notify in the event of a failure?

In my experience, every DBA has his own mental checklist and different ways that he / she fix issues. It is important to get a list of the items written down in a document. By combining the ideas of every DBA, you will come up with a more thorough checklist, a standardized way to fix issues, and problems are less likely to fall through the cracks.

After the DBA morning checklist is created, completed checklists should be archived in a notebook to ensure that each check was performed every day. This also serves as a history of fixes for past issues, and an audit trail for the DBA.

Since every database environment is different, and every IS shop has its own tools, every DBA's checklist will be different. The end goal is to create a checklist that is customized to your environment, in which issues can be found and fixed quickly, so that you can avoid having one of those difficult days.

With this in mind, listed below is a sample checklist. Your checklist should be unique to your environment and should help find and fix issues as quickly as possible.

Section 1: DBA Morning Checklist

Backups

- Verify that the Network Backups are good by checking the backup emails. If a backup did not complete, contact _____ in the networking group, and send an email to the DBA group.

- Check the SQL Server backups. If a backup failed, research the cause of the failure and ensure that it is scheduled to run tonight.

- Check the database backup run duration of all production servers. Verify that the average time is within the normal range. Any significant increases in backup duration times need to be emailed to the networking group, requesting an explanation. The reason for this is that networking starts placing databases backups to tape at certain times, and if they put it to tape before the DBAs are done backing up, the tape copy will be bad.

- Verify that all databases were backed up. If any new databases were not backed up, create a backup maintenance plan for them and check the current schedule to determine a backup time.

Disk Space
- Verify the free space on each drive of the servers. If there is significant variance in free space from the day before, research the cause of the free space fluctuation and resolve if necessary. Often times, log files will grow because of monthly jobs.

Job Failures
- Check for failed jobs, by connecting to each SQL Server, selecting "job activity" and filtering on failed jobs. If a job failed, resolve the issue by contacting the owner of the job if necessary.

System Checks

- Check SQL logs on each server. In the event of a critical error, notify the DBA group and come to an agreement on how to resolve the problem.

- Check Application log on each server. In the event of a critical or unusual error, notify the DBA group and the networking group to determine what needs to be done to fix the error.

Performance

- Check Performance statistics for All Servers using the monitoring tool and research and resolve any issues.

- Check Performance Monitor on ALL production servers and verify that all counters are within the normal range.

Connectivity
- Log into the Customer application and verify that it can connect to the database and pull up data. Verify that it is performing at an acceptable speed. In the event of a failure, email the Customer Support Group, DBA group, and the DBA manager, before proceeding to resolve the issue.

- Log into the Billing application and verify that it can connect to the database and pull up data. Verify that it is performing at an acceptable speed. In the event of a failure, email the Billing Support Group, DBA group, and the DBA manager, before proceeding to resolve the issue.

Replication

- Check replication on each server by checking each publication to make sure the distributor is running for each subscription.

- When replication is stopped, or changes to replication are made, send an email to the DBA group. For example, if the DBA stops the distributor, let the other DBAs know when it is stopped and then when it is restarted again.

- Check for any emails for the SQL Jobs that monitor row counts on major tables on the publisher and subscriber. If a wide variance occurs, send an email message to the DBAs and any appropriate IS personnel.

Section 2: Write down any issues and how they were resolved

This space is reserved for writing down issues and how they were fixed.

Section 3 - Confirmation

Completed By __________________________ Date: ___________________

Conclusion

Creating a morning DBA checklist has helped me many times in the past. Often times, I found CPU usage up near 100%, broken replication, connectivity problems, and space issues that I have been able to resolve before the majority of the work force was present and the issue could escalate. By having a standard DBA checklist document, it ensures that nothing is forgotten, which could result in a problem. It also minimizes down time of a company or department, provides a archive of past issues and how they were fixed, and helps ensure that the DBA will have a less stressful day!

Resources:

DBA CheckList Article.doc

By Bill Richards, 2008/04/14