Finding Installed Event Sessions

As a DBA, it is not uncommon to feel like you are marooned off on some tiny little island trying to figure things out for yourself. It may even be very common to feel the icy pain of the development cold shoulder (intentional or not) as you go about doing your duties to protect the data and secure high octane performance out of the database.

Not only may it be very commonplace to experience this due to the nature of the job, but it can also get to feel that way when a new product is released. Suddenly, that icy island is further north into the Arctic Ocean and you are struggling to gasp for a bit of air in the frigid cold.

I am certain this may be some of what you feel as you try to take a look at Extended Events. It is a fancy, new technology and there is not a ton of time to get up to speed while trying to warm yourself against the development chill. As you stew about the situation, you begin to ponder, “How am I even supposed to know what an XE is, let alone what XEs are running on my server?” Never mind the confusion on what an XE is and that it is actually running or deployed sessions on the server. Don’t worry about the terminology just yet. What can we do to figure out what might be out there?

Well, that is where the trusty Google has come to help you. Thanks to a better than adequate ability to search the Internet, you landed on this page that has a script to help you explore what might be running as far as Extended Events on your server. You may be surprised to learn that you have more Extended Events Sessions running than you fathomed. Yes, SQL Server comes with some default sessions. Yes, some of these default sessions are more public than others. And yes, some of these

Come on in out of the cold. Get off that arctic island and warm up with a little XE fun. This script is quite simple but will show you what you may have out there on your servers. Once you see what is there, hopefully your curiosity will get a little piqued and you will want to learn a little more (there is more of that here on this site too).

 

sessions are “hidden” from you. That is, until now.

And now, for that script:

Quick and simple, and full of that warming XE sensation. This script just shows you what is a private session versus public. What is a private session, you ask? Well, that happens to be all of those database audits you have deployed to your server. It also includes those little things like the “sp_server_diagnostics session” session (yes it sounds redundant but that is because session is also part of the name for that session).

This script will also show you what is deployed to the server versus what is running on the server (not all sessions have to be running). And to top it all off, I set it up so you can query for a specific session or for all sessions.

Try it out on one or more of your servers and let me know how it goes.

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Interested in seeing the power of XE over Profiler? Check this one out!

This has been the seventh article in the 2018 “12 Days of Christmas” series. For a full listing of the articles, visit this page.

SQLs Black Box Recorder – sp_server_diagnostics

Yesterday i shared the first article in a three part series about the SQL Black Box Recorder, that you can read here, with a promise of at least two more posts. The first article in the series laid the groundwork about the black box recorder in SQL Server. Furthermore, there was a brief introduction into the three components of the black box recorder. The three components as laid out in that article are:

  • Default Trace
  • system_health Extended Event Session
  • sp_server_diagnostics procedure

The first article went into deeper detail about the first leg of this three-legged black box recorder – the default trace. In this article, the focus will be directed to the next leg of the black box recorder – or the system_health Extended Event Session. If by some measure you are in the dark about what Extended Events is, then I recommend you read my personal blog series on the subject. There are numerous articles stepping through the subject in easy-to-digest fashion. You can find many of the articles in the following table of contents – here.

The focus of this article will take on the topic of the third piece of the black box recorder, sp_server_diagnostics. I will show how this procedure is critical in gathering pertinent information that is crucial to gathering an overall picture of the system—from a default black box recorder perspective.

SQL Black Box Recorder

What is the sp_server_diagnostics process?

Beyond being a component of the black box for SQL Server, what exactly is this diagnostics process? The sp_server_diagnostics is much as the name implies—it is a “diagnostics” service that attempts to gather information about various events that may affect the overall health of the SQL Server instance.

The diagnostics process will trap various server related health (diagnostics) information related to the SQL Server instance in an effort to try and detect potential failures and errors. This diagnostics session/process traps information for five different categories by default. There is a sixth category of information for those special servers that happen to be running an Availability Group.

The Six

  • system: overall health of the server including information about spinlocks, severe processing conditions, non-yielding tasks, page faults, and CPU usage.
  • resource: health of resources, specifically including physical and virtual memory, buffer pools, pages, cache and other memory objects
  • query_processing: captures query processing data about worker threads, tasks, wait types, CPU intensive sessions, and blocking tasks.
  • io_subsystem: health related data to IO.
  • events: errors and events of interest recorded by the server, including details about ring buffer exceptions, ring buffer events about memory broker, out of memory, scheduler monitor, buffer pool, spinlocks, security, and connectivity.
  • AvailabilityGroup: basic information about the AvailabilityGroup if there is an AvailabilityGroup enabled on the server.

This is fantastic introductory information. As luck would have it, you can also read more about it from my blog – here.

How to use this process?

What good is any level of information if we don’t quite know how to use or consume that data? Do we consume it via trace? Do we execute the stored procedure? Do we capture the information from some other means? The answer is yes! Wait, yes to what? Yes to all of it! Let’s explore how to consume this data.

Option 1

The first step to learning how to consume this data is to start at probably the most basic of levels. As you have already noted, this is a stored procedure. It was probably the sp_ in the name that gave it away, right? Since it comes in the form of a stored procedure, it makes it pretty easy to figure out the first means to get the data from it—just execute it. Actually, let me clarify that just a bit. sp_server_diagnostics is a CLR stored procedure. Nonetheless, all that needs be done is execute it at this level to get to the data.

Look at that! Pretty slick! I execute the stored procedure and I can see a point in time result of the “health check” from this component of the black box recorder. Notice here that I only have five of the six categories that I had mentioned. This is due to the lack of a configured availability group (AlwaysOn) from this particular instance. Another point of interest is that the data column is not in a very friendly format for DBA consumption. This is flat text that just goes on and on. It is painful to consume in this manner.

Tidy that Code

Looking at the code, you can see quickly the difference. I have changed the data column to be XML-based to better match the data being output from the stored procedure. Let’s see that in action:

That is much better. Now I can simply click one of the data results and see a nicely formatted easy to read XML output of that component. Here is an example from the query_processing component:

Option 2

Cool! I have just shown one means of exploring this facet of the black box recorder. Now, let’s take a look at other means to get at this particular data, beyond just executing the stored procedure. Did you know that this procedure is very closely related to Extended Events? (I am sure that you picked up on that nugget due to reading the previous article in the series or possibly the other article I mentioned here). With such a close relationship to Extended Events, it makes sense that sp_server_diagnostics also has a couple of events and is consumed by multiple sessions.

SQL Server comes with various default Extended Event sessions, of those sp_server_diagnostics is consumed by the system_health session as well as the hidden session for SQLDiag that is used by Windows Clustering and Availability Groups to determine server health. You already heard about the system_health (from the previous article), but few have heard about the hidden session. I will take a look at the events as well as the sessions that consume the sp_server_diagnostics data.

Events

First, I will share the details for each of the events related to the sp_server_diagnostics process.

As I have shown here, you can see that these events are a part of the debug channel. In the next couple of images, I want to show some of the payload attached to these events.

These events show what I have shown previously with regards to output of the stored procedure. There is a consistency in the output of the diagnostics process. What I have yet to show in this data is one more piece of information that could be helpful down the road and that pertains specifically to the extended events data. That piece of information I need to share is the keyword associated with each of these events. When looking for related events, I need you to understand that the sp_server_diagnostics events are a part of the scheduling classification. There, I have said it. I just gave you a big clue to help you in your troubleshooting endeavors and the use of the diagnostics processes/events.

Which Sessions then?

Now that I have shown which events are related to diagnostics, I can finally show which sessions are dependent upon the diagnostics data. To show that kind of information, I have this next query to help:

When I execute the statements in the preceding query, I will receive a single result—system_health. That said, do not believe everything I just showed you with that query. SQL Server does a decent job at hiding some of the details. When I run the next query, I will see a little bit more information.

When I run the preceding query, I will see the following two sessions (given that I have not change the default settings):

Take note of the names of those two sessions. I showed that the system_health session is reliant upon the diagnostics events. Now, I am showing that there is another event session called “sp_server_diagnostics session“. Wait, there is more! SQL Server continues to hide stuff from us.

Hidden Sessions

When you have AlwaysOn configured, there is an additional hidden session that is not visible within the sessions lists inside of SQL Server. To view this hidden session, you must follow the directions shown in this article on msdn. Once you have navigated to the folder containing the hidden XEL files, you can open them in Management Studio and view the data. It is while viewing the data that you will be able to see the relationship between sp_server_diagnostics and this newly discovered trace file.

The payload from this image demonstrates the relationship between sp_server_diagnostics and the hidden trace file. I challenge you to take some time to explore this hidden trace file on one of your servers where AlwaysOn is configured. I know you will be thankful for the investment at some point down the road.

Conclusion

I have just shown a ton of info about the third component of the black box recorder—sp_server_diagnostics. This procedure may be lesser known than the first two components, but it is highly integral into the overall monitoring and health of your SQL Server instance. In this article, I have not only shown how to use sp_server_diagnostics (albeit quickly), but I have also shown that the diagnostics are heavily relied upon by other components of the black box recorder. In addition, I have shown how this component is integrated into other hidden traces that are essential in the monitoring and troubleshooting of your instance.

The black box recorder is real. You need to invest time in getting to know the black box recorder components and how they can be of help to you in becoming a superhero DBA.

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Interested in seeing the power of XE over Profiler? Check this one out!

This has been the sixth article in the 2018 “12 Days of Christmas” series. For a full listing of the articles, visit this page.

SQL Servers Black Box Recorder – system_health

Yesterday i shared the first article in a three part series about the SQL Black Box Recorder, that you can read here, with a promise of at least two more posts. The first article in the series laid the groundwork about the black box recorder in SQL Server. Furthermore, there was a brief introduction into the three components of the black box recorder. The three components as laid out in that article are:

  • Default Trace
  • system_health Extended Event Session
  • sp_server_diagnostics procedure

The first article went into deeper detail about the first leg of this three-legged black box recorder – the default trace. In this article, the focus will be directed to the next leg of the black box recorder – or the system_health Extended Event Session. If by some measure you are in the dark about what Extended Events is, then I recommend you read my personal blog series on the subject. There are numerous articles stepping through the subject in easy-to-digest fashion. You can find many of the articles in the following table of contents – here.

The focus of this article will not be to introduce the topic of Extended Events. Rather, it will be to introduce the system_health session and dive into it a fair bit.

SQL Black Box Recorder

What is the system_health session?

Beyond being a component of the black box for SQL Server, what exactly is this event session? The system_health is much as the name implies – it is a “trace” that attempts to gather information about various events that may affect the overall health of the SQL Server instance.

The event session will trap various events related to deadlocks, waits, clr, memory, schedulers, and reported errors. To get a better grasp of this, let’s take a look at the event session makeup based on the available metadata in the dmvs and catalog views.

In addition to the types of events I mentioned, there are also a few more interesting things to discover from the results of this particular query. For instance, the actions that are being employed help to gather even more information such as the callstack, tsql_callstack, database id, and sql text of the query that was executing at the moment of the event that is trapped.

One very peculiar event that is trapped is the sp_server_diagnostics_component_result event. Remember from earlier in this article that I noted the third leg of the black box record happens to be a very similarly named session – “sp_server_diagnostics procedure”. When I cover this third leg, it will make a little more sense. Understand, for now, that your suspicion of it being a bit more integral to the black box (than merely 1/3 of the overall black box) is confirmed.

This is a good set of information that can be trapped. Furthermore, this is a good set of information which can give a fairly good diagnosis of several different potential problems within your SQL Server Instance! (Yes it may seem repetitive, but I wanted to underscore the importance of the data that can be trapped.) That said, compare the types of events to those of the default trace. Take note of how many events actually overlap between the two components of the black box recorder.

For ease of comparison, let’s simplify that previous query to get a set of events to read without the duplication due to the actions on each event.

And since I am feeling particularly generous, I have lumped the events from the two traces together in the following query:

That should help to compare the 17 events from the system_health session and the 34 from the default trace with a little more ease and speed. The events from the two sessions are rather complimentary to each as they help to construct the SQL Server Black Box recorder.

How to use this session?

With the events trapped by this session, there are a few ways to use the trapped data that should stand out pretty easily. To me, the quick application of the data from this session can come from the review of deadlocks, reported errors, or even waits analysis. Using the deadlock as probably the most common problem people will want to troubleshoot, this information is suddenly much more valuable and easy to access (no more need for that trace flag). Since the system_health is now trapping the deadlock by default, one could quickly access the deadlock graph from the event data and proceed to troubleshooting. More information on getting that deadlock graph can be found here and here.

Learn to use the data made available through this member of the black box recorder. Becoming familiar with this tool (and additionally with Extended Events) will make you a rock-star in the office. The information needed to troubleshoot most problems is right at your fingertips through the use of the different pieces of the black box recorder. If it is not available there, then more than likely it can be made available through a bit of tinkering with Extended Events. Get to know the tools and be that rock-star DBA.

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Interested in seeing the power of XE over Profiler? Check this one out!

This has been the fifth article in the 2018 “12 Days of Christmas” series. For a full listing of the articles, visit this page.

SQL Servers Black Box Recorder – Def Trace

Across many professions and industries there is often a need to have some sort of device that “audits” everything that happens with a device or process. We are probably all quite familiar with the infamous black box recorders used by the passenger airline and train industries. It is also quite possibly fairly common knowledge that many pace makers perform the same sort of functionality – on a smaller scale. Various metrics and data points are registered and recorded somewhere. Whether to a local storage device or whether the device phones home to some other remote location, it is recorded.

We often pontificate about the what-ifs for a black box recorder within SQL Server, right? We wish and dream and go about creating our own little recorder to suit the requirements we deem necessary for successful monitoring of the server for the just-in-case scenario. Well, the truth of the matter is that we really don’t need to go to such great lengths to create a “black box recorder” because Microsoft has already done all of that for us.

Wait, what? Yes, that’s right! Truth be told this shouldn’t be much news for most Senior Level data professionals working with SQL Server for the past few years. But if you are new to the product, this might be good news for you. Then again, it might just be bad news depending on your take.

Very much like what you may find with the airline industry, the black box recorder in SQL Server is not just a single method (device) implemented to capture all of the desired data points. On a passenger jet, you may find that there are three or more such devices that contribute to the capture and recording of the in-flight data. In SQL Server, there are three major processes that help capture our in-flight data. Over the next few articles I will discuss each of these processes. These processes include:

  • Default Trace
  • system_health Extended Event Session
  • sp_server_diagnostics procedure

This multi-tiered approach does provide a sort of fail-safe system. If one should be disabled, there still remain up to two more processes that may be running. That said, each of these can be disabled but it does take a bit of an overt effort. And since it does require somebody to put out effort to try and disable each of the black box components, you could potentially capture the culprit via audits from the other components or via an explicit audit that you create for these types of purposes.

Since this will be something to discuss over a few articles, I will break out each process into an individual article. For today, I will discuss the default trace.

Default Trace

The default trace by itself is something that can be turned off via configuration option. There may be good reason to disable the default trace. Before disabling the default trace, please consider the following that can be captured via the default trace. I will use a query to demonstrate the events and categories that are configured for capture in the default trace.

In this query, I have requested a few more data points than necessary to illustrate the point. That is to help illustrate an additional point that the default trace isn’t filtering out any data for these events. If the event fires (in this trace), it is recorded. Let’s divert right back to the events and categories for now. The results of that query will produce the following sample list of events for me on SQL Server 2014:

That is quite a range of events covered by this particular recording device. From changes to objects down to security related events and even errors and warnings. If somebody drops an object, the default trace can catch it. If one of the various DBCC statements is executed, it will be trapped in this trace.

There is one thing that is captured by this trace that is not overly obvious. In fact, it is rather annoying in how it is recorded in my opinion. Server configuration settings such as “cost threshold of parallelism” or the lesser utilized “user options” are not registered as an “Object:Alter” event but rather as an ErrorLog event. I wouldn’t call this type of “change” an error and don’t necessarily like seeing these registered with other legitimate errors. That said, it would be nice to see these logged differently (they are currently logged the same way with Extended Events). So, if somebody is dorking around with server configurations, it becomes a bit more convoluted to figure it out, but we can get there. Let’s see how that works:

Looking at my system, I get the following sample results:

You can see there are plenty of settings that I have been playing with captured by the default trace. If you happen to have a junior DBA or maybe a cowboy DBA that loves to shoot from the hip and make changes, this will help you find all the necessary details to bring to that DBA for discussion.

The default trace comprises one third of the black box recorder. I have shown a quick use for the default trace and have also shown some of the various events that are captured from this trace. I recommend getting to know your default trace just a little bit better. You never know when you may need to resort to the flight recorder data held in the default trace. Being familiar with the default trace before you need to use it will help improve your comfort level when under the stress of trying to figure out what happened just before the server went belly up!

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Interested in seeing the power of XE over Profiler? Check this one out!

This has been the fourth article in the 2018 “12 Days of Christmas” series. For a full listing of the articles, visit this page.

How to: File Target use in Extended Events

I have been doing a lot of posts of late about extended events but have failed to share some of the essential building blocks. I am not looking to do a series of definitions, but just share a couple of quick tips and examples here and there that will make the use of extended events a little easier to undertake.

The first one to tackle is the use of the file target. Using a file target is not difficult from a creation point of view (event session creation). But, some basics are critical or you could end up with fewer follicles.

Let’s take a quick peek at some of these tips.

File System Prep

This first little tip comes from a painful experience. It is common sense to only try and create files in a directory that exists, but sometimes that directory has to be different on different systems. Then comes a little copy and paste of the last code used that worked. You think you are golden but forgot that one little tweak for the directory to be used. Oops.

Take this example session. When you run this code, you will not see an error (don’t change anything).

However, if you try to start the session, you will receive a pretty little error. Here is the code to try and start the session, along with it’s associated error.

Msg 25602, Level 17, State 22, Line 25
The target, “5B2DA06D-898A-43C8-9309-39BBBE93EBBD.package0.event_file”, encountered a configuration error during initialization. Object cannot be added to the event session. The operating system returned error 3: ‘The system cannot find the path specified.
‘ while creating the file ‘C:\Database\XEs\FileTargetDemo_0_130670484928250000.xel’.

If you double check in the SSMS GUI, you will see that the session is there. It is just stopped. You can also check that the session exists if you recall my query from a previous post where I show the status of the session.

If you take it a step further and try to read the log file that should be created, you will be greeted once more with a reminder that there is a problem. Try this (sans change again).

And you will receive the following message:

Msg 25718, Level 16, State 3, Line 31
The log file name “C:\Database\XEs\FileTargetDemo*.xel” is invalid. Verify that the file exists and that the SQL Server service account has access to it.

The beauty here is that all is not lost. These are easy problems to get around and resolve. All it takes is to adjust the file path to be the correct path or a path that exists (to which you have permissions).

Read the Log File

If you have been reading this article, then this might seem like a bit of redundancy. That is by design. DBAs love redundancy—well at least when it means there is a means to recovery due to the redundancy.

Unlike reading from the ring_buffer, when you read from the file target, you will use different code. The file target is stored in a binary representation on disk that a function will produce as XML format for human readability. So to read a file target, you would need to do something like the following.

And you definitely do not want to try the following with a file target. That is unless of course you are looking to go bald at an earlier than expected age in this profession.

If you find yourself doing this version, all you will get for results is a short xml string similar to the following:

This is not what you would be looking for when trying to read the event session data. That said, it at least provides some information that could be of use. You will see that buffers are tracked and the file name is tracked. For this last demo, I corrected the FileTargetDemo session to use a directory on the file system that existed and to which I had the appropriate level of permissions.

I just covered two quick tips to help make life a little bit easier when dealing with file targets and extended event sessions. I will try to continue little tips like this over the coming months. I hope you have found this useful and that you will indeed be able to put it to use.

 

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Interested in an article demonstrating the power of XE over Profiler? Check this one out!

This has been the third article in the 2018 “12 Days of Christmas” series. For a full listing of the articles, visit this page.

Upgrading From SQL Server Profiler

I just shared an article describing how to use Extended Events to perform your SQL Server Profiler duties. If not, you can read all about it here.

In that article, I showed one common use-case for SQL Server Profiler and how to achieve the same result via Extended Events. What I didn’t show in that article was how to correlate all of your favorite trace/profiler options into the shiny new XE equivalents. Is there even a way to do that?

As luck would have it, there is a means to correlate trace/profiler events to Extended Event events (yeah that sounds weird).  With the release of SQL Server 2012, Microsoft introduced a couple of catalog views to help with this correlation effort. Those views have a pretty straightforward naming convention. For example, one of the views is named sys.trace_xe_event_map while the other is sys.trace_xe_action_map. For this article, I will be focusing on the former.

Making the Upgrade

When looking to finally make the upgrade away from trace/profiler, a big key is to figure out if your favorite trace events are even an option at the next level. I have talked about finding events in XE in previous articles by trying to search for a specific topic that might apply to your current condition.

But if you are already familiar with specific trace events, you may just want/need to know what the new name is in XE. This is where those catalog views come into play. And in support of that, here is a query that can help in that upgrade effort:

Looking through the results, one would notice that not every trace event maps to an Extended Event event. The events that don’t map, in general, deal with audits (which is actually driven by Extended Events).

This is a really good start to getting you on your way to that much needed upgrade away from profiler/trace. What if there are several traces that are already in use or scripted for various reasons in the environment?

Decision Time

If you happen to have traces that are already deployed in the environment or in script form, it can be a bit of a tedious pain to convert those to Extended Events. Do you manually recreate the traces as XE sessions? Do you abandon the upgrade due to the effort and annoyance it will create? Or do you find some automated means of performing the conversion?

Of those three options, only two are valid options. Those options involve performing the conversion of the traces to XE sessions. There are pros and cons for each. You may opt to take your time and learn more about Extended Events by performing the manual upgrade, or you may choose to save time by using an automated routine.

Should you decide to try the automated routine, there is one already out and available to help you on your way. Jonathan Kehayias wrote the script and you can download it here.

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Interested in another article demonstrating the power of XE over Profiler? Check this one out!

This has been the second article in the 2018 “12 Days of Christmas” series. For a full listing of the articles, visit this page.

How To: XEvents as Profiler

A common excuse for not delving into extended events seems to be the jump from Profiler to extended events.  There appears to be an inherent fear with how to use extended events.  In addition to that fear, is the fear that extended events does not do what Profiler can do so easily.

Today, I would like to share a short tutorial on how to use extended events as if it were a profiling session.  I am only going to show one of the many possible means to profile via extended events.  Just understand that this means you have flexibility and a way to do more with this simple example than what I am going to share.

Requirement

You need to capture TSQL queries that are occurring against your instance in order to try and determine the code that is being passed to SQL Server from the application.

How-To

Rather than jump to Profiler, you are going to need to jump to a fresh, clean query window in SSMS.  Just like in Profiler, you will want to capture certain “statement” oriented events.  For this session, I want to start with sp_statement_starting and sql_statement_starting.  I am not going to use the _completed forms of those events because I want to capture as much as I can – without capturing too much noise.  Sometimes, there may be a _starting without a coordinated _completed and that could throw a wrench in the works.

With the basic information in hand, we are ready to implement some of the things learned in previous XEvents posts (here and here).  Let’s go ahead and start setting up the session that will capture the information we seek.

One caveat to consider is in the sp_statement_starting event, I have specified a specific set option be used.

This particular option is disabled by default.  If you want to know the object_name of code that is being used (triggers, stored procedures, etc), then this option must be enabled.

I have specified the same actions to be used for each of the events I am trapping, but this is purely up to you.  If you do not wish to capture these actions, then you can remove them or add different actions as necessary.  I have also specified a different predicate for each of the events.  This is something that works so much better in XEvents than Profiler – the filtering that can be applied is much more flexible.

Once the session is started, then it is merely a matter of figuring out how to look at the data.  I generally use TSQL to parse the data from the file, but you could also fiddle with the GUI as well.  You can access that by right-clicking the event session in management studio and then selecting “Watch Live Data”.  If everything is configured properly, then you will start to see data after the first event is triggered.  From within the GUI, you can pick and choose which columns to display in your view.  You can pause the display of the session data, or you can even “erase” the data from the current view (just like in Profiler).  You can even filter, aggregate or group the data in the view from the GUI.

The use of the GUI is up to you.  I still prefer to use the script route.  With that route, here is a sample of what you may need to write in order to display the data from the session data file that has been captured.

It really is that simple.  Now you have a means to run a Profiler-like trace on your server without the impact of Profiler.  XEvents provides the means to run a more evolved Profiler session on your data to capture things like queries from an application.  XEvents does all of this without the severe penalty of Profiler and with many additional bonuses.  Try it out and enjoy!

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Interested in another article demonstrating the power of XE over Profiler? Check this one out!

This is the first article in the 2018 “12 Days of Christmas” series. For the full list of articles, please visit this page.

Quick and Easy XE for Azure DB

The Cloud

It has been a minute since I gave much love or attention to Extended Events in Azure SQL DB. Things have changed a touch since then. We will see about some of those changes in a future article. As for this time, let’s focus on how to get a session in Azure SQL DB up and running really quick and easy.

Create a Session

I am going to keep this as easy as possible for creating a session and thus I will demonstrate how to do it from the GUI. And for those more in tune with their scripting side, there is a future article on that as well as a glimpse of a script near the end of this article.

Recall from the previous article that things in Azure SQL DB are different for Extended Events. XE is database scoped rather than server scoped (as is the case with your on-premises servers). Due to this change, finding the GUI for XE is a little different.

In order to find the GUI for XE, you must drill down into the database and then you will see “Extended Events.” (Side note, there is no XE Profiler for Azure SQL DB as of this writing.) If you right-click Sessions, you will get a menu with the option to create a New Session.

After clicking “New Session…” the familiar window for a new session will appear.

Here, you see a session that I started creating already. For this session, I opted to go with the template called “Query Detail Tracking” (more on templates for Azure SQL DB in a future article). I also like to enable “Causality tracking” so that checkbox is ticked in this example. The template has the events I wish to capture already and I am only going to use the ring buffer (file target has some additional requirements for an Azure SQL DB session and should be treated in an entirely separate article) so I will skip those screens straight to the advanced screen before completing this session.

From the advanced screen, all I want to do for now is to decrease the dispatch latency from the default 30 seconds down to 10 seconds. For this session, I just want to capture the events quickly in the target and be able to review them as soon as possible. I won’t be able to utilize the “Watch Live data” option for this session (as shown below), so a quick dispatch is essential.

After I have selected all of the configurations I desire for this session, then I click the script button at the top of the window. From there, I will select “New Query Editor Window.” Once scripted, I will receive the following script for this particular session.

Notice that there is nothing about starting this session in the script to start the session.

Note that “on database” has been specified instead of the traditional “on server” for the session state command. In order to stop the session, it is just as easy as follows.

Conclusion

This has been a very simple introduction into the creation of an Extended Event session using a template for Azure SQL DB. I demonstrated the use of the GUI to configure the session quickly and then to subsequently script that configuration before creating the session.

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Interested in seeing the power of XE over Profiler? Check this one out!

This is a quick pre-cursor to the 2018 “12 Days of Christmas” series. The series will begin on December 25th (the first day of Christmas) and you can easily follow it via this page.

The Mystery of Missing Backup Files

Comments: 1 Comment
Published on: December 10, 2018

Yoink – Backups Gone

Not only is this mystery of missing backup files worth discussing now, but has been a mystery worth talking about previously (albeit in slightly different form). One such exploration into this mystery can be read here. And as a requirement for exploring any mystery some tools are necessary to be able to explore the clues that might be revealed. For this mystery, we will be utilizing this script along with our acutely developed sense of perception, ability to fact-find and our ability to troubleshoot.

Without further ado and all of our tools in hand, let’s begin!

Peter Deebweehay and Tommy Tsisamin work at Global Exchange Wholesale Retailers (any similarity to real entities is wholly unintentional and completely fictitious). Both Peter and Tommy are diligent hard working individuals. Peter happens to be a DBA and is stubborn about ensuring databases are backed up regularly. He has carefully planned out the disaster recovery scenarios and has nit-pickingly designed the backup strategy to ensure business continuity.

Tommy on the other hand is just as head-strong about performing his daily duties as the systems administrator. Tommy is responsible for the virtual machines, network infrastructure, operating systems, domain, mail services and desktop machines (OS to hardware). Tommy has also carefully planned out all of the disaster recovery needs of the business for all of the servers and specific workstations of individuals that might be pertinent to the continuity of the business.

Tommy and Peter get along together glowingly. The two work together any time there is a new SQL Server to be stood up and have worked together previously many times to ensure good performance of the SQL Servers on both physical and virtual machines.

Lately users have been complaining about a performance slowdown. The slowdown passes quickly but it keeps coming back. Peter starts to look into the reports and is slowly developing a pattern. He notices that the slowdown occurs every hour. Like a good DBA, Peter decides to implement an Extended Event session to capture the queries and server activity that occurs for a period of 3 hours to hopefully capture data that could indicate what the offender is.

After the three hour window has passed, Peter pulls out the captured data and starts looking it over. He notices that every hour there is a backup that occurs against this database. Peter knows that shouldn’t be happening because it is out of cycle with what he had implemented. To be absolutely certain, Peter decides to check the SQL Agent jobs and the Windows task scheduler. Nothing indicating a backup in either location. Next Peter pulls out his trusty backup history script to confirm the data he pulled from the XE session data. When he starts looking at the data, his jaw drops as he sees something that looks like the following.

Missing backup files

Looking at these results, Peter starts looking in the Backup share he had created for these backups and can’t find anything representing those backups with the GUIDs in the BackupPath result. Then he ponders for a minute. Peter knows he has seen something like this before and then recalls where he saw it. Those Full backups with the GUID as the path are coming from outside of SQL Server.

Peter is curious if Tommy could possibly help with the new-found mystery on this server. When Peter presents the information to Tommy, Peter is met with another jaw-dropping moment. Tommy, with a wry smile and pride in himself, explains to Peter exactly what those GUIDs happen to be (not knowing the connection). Tommy explains that he has set the virtual machine to perform an incremental backup every hour. This incremental backup is represented to SQL Server as a Full database backup and produces a backup that equates to .005MB. With this data, Peter is curious to know if Tommy is quiescing the database(s) prior to doing this incremental backup. As it turns out, Tommy was not. Peter was mildly frustrated but he was able to work out a better solution with Tommy. Without quiescing the database, these full backups become questionable as to the usability to actually recover this server (an incremental OS backup is not the same as a Database Differential backup).

As far as the database is concerned, these GUID backups are much the same as a missing backup. Any backup that is taken that a DBA cannot explain or find would also be a missing backup.

What does this mean for you?

In the case of Peter and Tommy, this meant that there was a bit of a mis-communication and a bit of a performance hit for these off cycle backups. Had Peter implemented differential backups, the recovery plan he had implemented would be for naught because each of those full backups caused by the incremental backup for the VM would break his differential chain. Also, if Peter had attempted to recover the database to a different server, he might have discovered this sooner and before the users had reported the performance issues that led him to the discovery. Not only is a backup plan essential, but monitoring of the logs and backups as well as routine practice restores is necessary.

After reading this article, enjoy more articles similar in nature such as this series on Extended Events or this series called “Back to Basics

Capture the Flag – The Trace Flag

Many people work in a tightly controlled environment. Every change requires a change control form and approval to make a change. In some environments, things are more lax, but you still wish you could tell when certain changes were made. That is even true in some of those tightly controlled environments. You know what I am talking about. Despite the controls in place there always seems to be some change that is unknown or even some cowboy that makes changes as s/he sees fit and forgets to notify people or follow process.

Then come the times when you are new to an environment and you really need to know some basic information about the servers you have just inherited. Well, truth be told, you don’t necessarily have to be new to a job to inherit some new servers. Sometimes, blackbox servers end up getting dropped into your realm of responsibility.

When you run into something like this, you really need to have some means to figure out what changed, when it changed and who changed it. Having that information as a base, you would then have the tools to inquire as to why it might have been changed. One of the things that would be nice to know is around the trace flags that can be enabled in SQL Server. There are trace flags (or knobs) for so many different things within SQL Server that it would be nice to get a handle on them if they do happen to be enabled or even disabled.

As a DBA in SQL Server, you should know by now that there are multiple different ways to do all sorts of things within SQL Server. Today we will discuss three different methods that you could use to perform this investigation for yourself.

Report Style

Did you know that Management Studio comes with a dashboard report? It does! In that report, one will find a pretty decent amount of overview information as a first pass to get to know your server. If you were to right click the instance within SSMS, you can browse to the Reports/Standard Reports submenus. From there you can select the Dashboard report. The path to the reports is similar to what is displayed in the following screen capture.

The report that is of interest is circled in red in the preceding image.

If you have any trace flags enabled (globally) you can see those listed in the report similarly to the following image:

Once again, I have encircled the relevant data in red. From this instance that I queried, I currently have one trace flag enabled. As it stands, this particular trace flag is the one that disables the noise of successful backups from being recorded to your log.

This is one particularly easy method to retrieve that information. A benefit from this method is that you will be able to see some graphs and charts and so forth by using the report.

SQL Query

As logic would dictate, it would seem that we should be able to trap that information through the use of our most frequently used tool – TSQL. Case in point is that the report itself requires the use of TSQL to produce the dataset for the nice display we would see when the report is rendered.

Using similar code to what is used to produce the Dashboard report, we can get a decent understanding of the trace flags that might be employed on your particular instance.  Let’s take a quick look at what we could do.

As you can see, that code is terribly simple. The crux of the code boils down to a DBCC command to return the results of TRACESTATUS. Granted, this is a bit over the top and designed to dump the data into a table for later consumption (for good reason). You can legitimately get away with simply running DBCC TRACESTATUS. The results of this would produce the following for the same instance of SQL Server shown in the Server Dashboard report we already explored. Those results look like the following.

queryout

So far, so good. If I had several trace flags enabled globally on this instance then I would see every single one of them in the result set just displayed. Cool!

To this point, this is wonderful for anything that might be currently enabled on the instance. Should any of those trace flags be disabled shortly after being enabled, or even be disabled months after being enabled but before you run either of the two preceding methods, then you are up a creek without a paddle. That is a problem. How would you trap that cowboy sysadmin or cowboy DBA in that case? Well, you could go with a full blown audit. Or you could try a similar approach but on a somewhat smaller scale.

If you are paying close attention, you are probably thinking to yourself that the query just demonstrated seems more complex than it need be.  After all, the documentation for DBCC TRACESTATUS says that I can run that command with a -1 parameter and just have it return the global trace flags.  And in the script I am showing, I filter on Global = 1.  Well, as it turns out, the -1 option in DBCC TRACESTATUS does not work as documented.  The -1 parameter does not return the appropriate result set and that is reproducible.  If you were curious, the MSDN reference is here.  The use of that parameter value still returns session flags as well.

Extended Event

Oh my, we are talking about something really abstract and difficult now! No, not really.  This is a pretty straightforward approach and easy enough to implement. Within XE, there is an event called trace_flag_changed. If you were curious, you could use a query such as the following to find any events that might be related. The key is finding the correct key terms to find what you need/want.

Running that query will return just one result. You might find a few more if you expand your search criteria to include the term “flag” within the description field. It’s up to you to explore a bit more. As the description and name imply for this particular event, you can presume correctly that it tracks when a TF is either enabled or disabled (the state of the TF changes). Let’s call this a mini audit for a very targeted type of event.

Knowing the name of the event, let’s go ahead and look at a possible setup for an event session to trap data:

This gets us to a point where we can now trap all changes to a trace flag status (enabled/disabled). At this point, a bit of a caution is that the event appears to fire twice for every time it occurs. There is one case where it will not produce a double result for each change. That case happens to be if the TF is already set to the same setting to which you are attempting to change it to. So if it is enabled, trying to enable it again just records one event to fire. However, trying to enable or disable the TF when it is in the opposite state, you will see two occurrences of the event firing. Here is a sample output to help show what happens.

xe_results Trace Flag

Displayed in the preceding image, you will notice both scenarios. In the scenario where the TF was not enabled, and I attempted to enable it, you will see two entries. And then when I disabled that TF after enabling it, I once again got a double entry. A corresponding example is encircled by the green. Now, when I tried to disable a TF that was not currently enabled, you see there is a single entry. That occurrence is marked via the orange circle.

A problem does present itself here with this result set and the use of XE.  How in the heck did I get the results from the XE Session so I could peruse it? Here is a sample script that will help parse this session data.

Now isn’t that really helpful?

Conclusion

Today I covered three different means to help isolate the trace flags in use or the changes to the state of those trace flags. This is good information that any DBA should want to know. It is information to help get an understanding of the health and operating environment of your instance. It is also a means to help identify what might be changing under your nose!

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Need a little help with extensive audits of your servers, check out these articles.

«page 2 of 12»

Calendar
April 2019
M T W T F S S
« Mar    
1234567
891011121314
15161718192021
22232425262728
2930  

Welcome , today is Monday, April 22, 2019