Ghosts – an eXtrasensory Experience

ghostrip_fireThis is the last article in a mini-series diving into the existence of ghosts and how to find them within your database.

So far this has been a fun and rewarding dive into Elysium to see and chat with these entities.  We have unearthed some means to be able to see these things manifesting themselves in the previous articles.  You can take a look at the previous articles here.

For this article, I had planned to discuss another undocumented method to look into the ghost records and their existence based on what was said on an msdn blog.  But after a lot of research, testing and finally reaching out to Paul Randal, I determined that won’t work.  So that idea was flushed all the way to Tartarus.

Let it be made very clear that DBTABLE does not offer a means to see the ghosts.  Paul and I agree that the other article that mentioned DBTABLE really should have been referring to DBCC Page instead.

Despite flushing the idea to Tartarus, it was not a fruitless dive.  It just was meaningless for the purpose of showing ghosts via that DBCC command.  I still gained value from the dive!!

All of that said, the remainder of the plan still applies and it should be fun.

Really, at this point what is there that hasn’t been done about the ghosts?  Well, if you are well tuned to these apparitions, you may have received the urge to explore them with Extended Events – sometimes called XE for short.

As has been done in the past, before we board Charon’s boat to cross the River Styx to Hades to find these ghosts in Elysium, one really needs to run the setup outlined here.

With the framework in place, you are now ready to explore with XE.

Look at that! There are several possible events that could help us track these ghosts.  Or at the least we could get to know how these ghosts are handled deep down in the confines of Hades, err I mean the database engine.

Ghost_XE

 

From these possible events, I opted to work with ghost_cleanup and ghost_cleanup_task_process_pages_for_db_packet.  The sessions I defined to trap our ghost tracks are as follows.

You can see there are two sessions defined for this trip down the Styx.  Each session aptly named for our journey.  The first (GhostHunt) is defined to trap ghost_cleanup and sends that information to a histogram target.  The second (SoulSearch) is defined to use the other event, and is configured to send to the ring_buffer.  Since the second event has a “count” field defined as a part of the event, it will work fine to just send it to the ring buffer for later evaluation.

Once I have the traps, I mean event sessions defined, I can now resume the test harness from the delete step as was previously done in previous articles.  The following Delete is what I will use.

Prior to running that delete though, I checked the Event Session data to confirm a starting baseline.  Prior to the delete, I had the following in my histogram target.

 

predelete_count

 

After running the delete, and checking my histogram again, I see the following results.

post_count

 

You can see from this that in addition to the 25 pre-existing ghosts, we had another 672 ghosts (666 of which were from the delete).

This is how I was able to investigate the GhostHunt Extended Event Histogram.

But what about looking at the other event session?

Let’s look at how we can go and investigate that session first and then look at some of the output data.

ghostclean

 

Cool!  Querying the SoulSearch session has produced some information for various ghosts in the database.  Unlike the histogram session that shows how many ghosts have been cleaned, this session shows us some page ids that could contain some ghosts – in the present.  I can take page 1030111 for instance and examine the page with DBCC PAGE as follows.

 

 

pagealtLook at that page and result!! We have found yet another poltergeist.

RIP

Once again we have been able to journey to the depths of the Database engine and explore the ghosts that might be there.  This just happens to illustrate a possible means to investigate those ghosts.  That said, I would not necessarily run these types of event sessions on a persistent basis.  I would only run these sessions if there seems to be an issue with the Ghost cleanup or if you have a strong penchant to learn (on a sandbox server).

Some good information can be learned.  It can also give a little insight into how much data is being deleted on a routine basis from your database.  As a stretch, you could even possibly use something like this to get a handle on knowing the data you support.  Just be cautious with the configuration of the XE and understand that there could be a negative impact on a very busy server.  And certainly proceed at your own risk.

Database Ghosts

phantasmripRecently you may have seen my first article on this topic over at SQL Solutions Group.  If not, here is the link so you can read that article first before proceeding here.  This article is intended as a first of two part follow-up to that article.

Now that you have read that other article, we can recap it a little bit.

You will recall that in that article, I discussed the presence of Ghosts in your database.  I also discussed that those ghosts are a good thing.  They have some benefits to them, of which is a bit of a performance boost for some operations like the rollback of a delete.

In that article I discussed one method with which you could see these ghost records.  In this article, I would like to share an alternate method to be able to see these ghosts.

In order to explore this alternate means, let’s go ahead and follow the first few steps from the other article to get the setup complete so our second ghost hunting foray can begin.

Now with the setup complete, we should once again confirm that we have appropriate data available for the hunt.  Once again a simple query can suffice to show the data.

Upon execution of the check script, we should see something similar to the following data-set.

datasample

 

Great, the data is confirmed and we should have a wonderful chance to find some ghosts once again.  In the previous attempt, we needed an additional plasma blaster in the form of a trace flag.  In this attempt we will hunt these ghosts without that tool and see what we can find.  Why attempt it in this way?  Well, it is simply because I’d rather not use a trace flag if it is not necessary.  If you recall, that trace flag had a couple of noted effects.  One of the effects was that it turned off the ghost cleanup process.  If I can avoid it, I’d rather leave the containment unit in tact.

Now, due to that clerical error of putting Halloween on the wrong date, we need to follow the prescribed policy to delete records prior to creating the proper records.

Notice that we issued the delete in a transaction this time around.  We are going to leave that transaction open for a bit while we hunt those ghosts.

The first step is to verify that some ghosts might be present.  To do that we should run a query such as the following.

From that query we should see something like the following.

idxstats_ghosts

Once again we are on the right track.  We can further confirm the existence of these ghosts through a little more investigation and monitoring.  Let’s try the trick with the dblog function again.

With that query, we should see something like the following result set.

dblog_ghosts

 

Very cool.  We once again can see that these phantasms are in the database.  We have enough information that we can proceed on to the next stage.  We can pass the PageID into DBCC PAGE in order to investigate the ghosts on the page.  If we use the PageID that is circled with the green in the preceding result set, we can get a better feel for these specters.

And we may see results such as the following.

page_ghosts2

Recall that the log says this page has ghost records on it.  When we check the page with DBCC PAGE we can definitely see that there are ghosts on the page.  This is very cool.  Now, had we tried to check for ghost records on the PFS page we would not be able to see the ghost count like we were able to see by enabling the TF.

Once again we have been able to display the existence of ghosts in the database.  In order to get these ghosts to move on from the database to their afterlife, we merely need to commit the transaction or roll back the transaction.

Stay tuned for the next article in this mini-series about ghosts.  Who knows, we may even have a bonus Halloween article since this is Halloween month.

Murder In Utah

I am about to set sail on a new venture with my next official whistle stop.  This year has been plenty full of whistle stops and I plan on continuing.  You can read (in full) about previous whistle stops and why they are called whistle stops here.

Suffice it to say at this point that it all started with a comment about a sailing train a few months back.

goldspike

 

Time to sink or sail, so to speak.  SQL Saturday 349 in Utah will mark the next attempt at what I hope to be a repeat performance – many times.  I will be tag-teaming with Wayne Sheffield in this all day workshop event.  The session is one of two all day sessions for the event in Lehi, UT (just south of Salt Lake City).

If you are a DBA or a database developer, this session is for you.  If you are managing a database and are experiencing performance issues, this session is a must.  We will chat with attendees about a horde of performance killers and other critical issues we have seen in our years working with SQL Server.  In short, some of these issues are pure murder on your database, DBA, developer and team in general.  We will work through many of these things and show some methods to achieve a higher state of database Zen.

Description

Join Microsoft Certified Masters, Wayne Sheffield and Jason Brimhall, as they examine numerous crazy implementations they have seen over the years, and how these implementations can be murder on SQL Server.  No topic is off limits as they cover the effects of these crazy implementations from performance to security, and how the “Default Blame Acceptors” (DBAs) can use alternatives to keep the developers, DBAs, bosses and even the end-users happy.

Presented by:

wayneWayne Sheffield, a Microsoft Certified Master in SQL Server, started working with xBase databases in the late 80’s. With over 20 years in IT, he has worked with SQL Server (since 6.5 in the late 90’s) in various dev/admin roles, with an emphasis in performance tuning. He is the author of several articles at www.sqlservercentral.com, a co-author of SQL Server 2012 T-SQL Recipes, and enjoys sharing his knowledge by presenting at SQL PASS events and blogging at http://blog.waynesheffield.com/wayne

 

 

 

JasonBrimhall

Jason Brimhall has 10+ yrs experience and has worked with SQL Server from 6.5 through SQL 2012. He has experience in performance tuning, high transaction environments, as well as large environments.  Jason also has 18 years experience in IT working with the hardware, OS, network and even the plunger (ask him sometime about that). He is currently a Consultant and a Microsoft Certified Master(MCM). Jason is the VP of the Las Vegas User Group (SSSOLV).

 

 

 

 

Course Objectives

  1. Recognize practices that are performance pitfalls
  2. Learn how to Remedy the performance pitfalls
  3. Recognize practices that are security pitfalls
  4. Learn how to Remedy the security pitfalls
  5. Demos Demos Demos – scripts to demonstrate pitfalls and their remedies will be provided
  6. Have fun and discuss
  7. We might blow up a database

kaboom

 

There will be a nice mix of real world examples and some painfully contrived examples. All will have a good and useful point.

If you will be in the area, and you are looking for high quality content with a good mix of enjoyment, come and join us.  You can find registration information and event details at the Salt Lake City SQL Saturday site – here.  There are only 75 seats available for this murder mystery theater.  Reserve yours now.

The cost for the class is $150 (plus fees) up through the day of the event.  When you register, be sure to tell your coworkers and friends.

Wait, there’s more…

Not only will I be in Utah for this workshop, I will also be presenting as a part of the SQLSaturday event on October 25, 2014 (the day after the workshop which is Oct. 24, 2014).  You can view the available sessions here.

Shameless plug time

I present regularly at SQL Saturdays.  Wayne also presents regularly at SQL Saturdays.  If you are organizing an event and would like to fill some workshop sessions, please contact either Wayne, myself or both of us for this session.

October 2014 Las Vegas UG Meeting

Comments: No Comments
Published on: October 8, 2014

The Las Vegas User Group is happy to announce our monthly meeting.  The meeting is available for in person and webinar style.  The start time is 6:30 PM Pacific and the details are listed in this post.  We hope to see you there!

invite_oct2014

This month we have a special treat brought to us by fellow MVP Argenis Fernandez (blog | twitter).  If you had a devilish little trick to upgrade SQL Server without an outage, would you do it?  Argenis will show you a good set of tricks to put your upgrade nightmares to rest.

You can read all about what Argenis is planning to present and read all about Argenis on our meetup page.

We hope to see you either in person or virtually for our monthly UG installment.

Microsoft, Me and the MVP

Comments: 2 Comments
Published on: October 2, 2014

Microsoft_MVP_logo_thumbOctober 1st is one of the quarterly dates that Microsoft announces all of the new people into the MVP program.  It is also a day in which Microsoft recognizes many already in the program and renews their status for the upcoming year.

This time around, I had the distinct honor and pleasure of joining the ranks of so many people who have been a part of this program for some time or who might have also been recently named to the program.

The MVP program is a program designed to recognize people for their efforts in teaching about various Microsoft technologies.  In my case, that happens to be SQL Server.  It is a humbling feeling to have been nominated and to be welcomed into the program.

To top it all off, I found it really cool that my welcome letter came in French rather than the standard English.  This is perfectly acceptable to me since it is a language I speak fluently and do try to keep up to date with it.  It was also interesting to see the MVP Lead call me within minutes of receipt of one of NDA email to make sure it was ok.  He was thrown by the NDA being in French and was concerned whether I could understand it or not.  At which point we had a good chuckle because I had no qualms with the language in use.

Attached are a couple of screen grabs with the welcome letter and so forth.  I geek out every time I look at them and it is totally cool.

In addition, I have also attached a link to my updated MVP profile.  You can find that here.

notice

CongratsSo what does it mean now that I am an MVP?

Well, to be truthful, it means status quo.  Having this means that a recognition was given.  It also means that, in my opinion, I am on the right track in doing good things for the community.  Because of that, I will continue to do what I can where and when I can to help teach others about SQL Server.  I am indeed grateful to be in pretty good company with people who also try to teach about SQL Server.

To all who helped this happen, I am grateful for your support.

I am also grateful to SQL Solutions Group who first broke the news.  You can read about that here.

 

September 2014 Las Vegas UG Meeting

Comments: 1 Comment
Published on: September 10, 2014

Who is up for a little free learning this week? Besides the opulence and feast that was the 24 Hours of PASS (Summit Preview), we have more training in store for you from the people in Las Vegas. Let’s call this a preview for next week which happens to be DevConnections (which also happens to be in Vegas)!!

The Las Vegas User Group is happy to announce our monthly meeting.  The meeting is available for in person and webinar style.  The start time is 6:30 PM Pacific and the details are listed in this post.  We hope to see you there!

Capture

Abstract: PowerShell: The Way of the DBA Dragon

In this introduction to PowerShell, attendees will learn how to start from scratch with PowerShell 3.0 or newer, use the pipeline, run T-SQL against multiple instances, use transcripts, and be shown martial arts usage of one of the SQLPSX cmdlets.  Scripts will be provided.

BIO

Lars Rasmussen was born in Illinois, but considers Utah home.  He does not play video games, is learning to camp and hike, and is happy to have shared the summit of Mt. Timpanogos with two of his sons.  Lars’ wife and four children help him smile and laugh, and the family dog is teaching him patience.  Playing board games is one his favorite pastimes.  He considers SQL Server, PowerShell, and CMD.EXE some of his dearest frenemies.  Lars enjoys the company of SQL Server professionals and sysadmins – he used to be one of the latter, and is employed as a database administrator for HealthEquity.

LiveMeeting Info

Attendee URLhttps://www.livemeeting.com/cc/UserGroups/join?id=MR7C92&role=attend

Meeting ID: MR7C92

T-SQL Tuesday #58 – Security Phrases

Comments: No Comments
Published on: September 9, 2014

TSQL2sDay150x150Today is once again TSQL Tuesday.  This month the event and topic are being hosted by Sebastian Meine (blog | twitter).  You can read all about the topic this month on his blog inviting all to participate.

Despite Sebastian being a real cool kid, I was not too hip to the topic this month.  Not because it is a bad topic or anything, it’s just that I really had nothing that seemed to stand out as easy to write for the blog party.

Then all of a sudden, a nice fat, juicy pork chop landed right in my lap.

The Pork Chop

A client requested that I make some changes to a task on a development server for them.  As it happens, the task is a powershell script that was being run on a schedule via a Scheduled Task in the Windows “Scheduled Tasks” control panel.  Making the requested change is a no-brainer of a change – or it should have been.

The change was to change the owner/executor of the scheduled task to the service account for the SQL Service.  By doing that, they would be less likely for the job to fail in the future due to an employee leaving the company.

As luck would have it the client DBA happened to know the password for the service account.  When changing the task to use the service account with the supplied password, we soon discovered that the supplied password caused the service account to become locked.  OUCH!

Maybe it was just fat fingered?  Nope, no dice!  As it turns out the DBA had the incorrect password and did not know the correct password.  Worse, nobody else knew what the correct password was.  Due to this issue, I proposed that the sysadmins and I work together to get the password changed.  That is to be done at a future date.

In addition to this, we decided that the passwords need to be more accurately documented.  These should be stored in an encrypted vault (the application is your choice).  But the mere use of an encrypted vault is far better than the use of a sticky note to document passwords (and I have seen that far too often at client sites).

This is just a short and sweet post for the day.  I think that it demonstrates problems that can arise from bad password management and also the risks that could come from that password management.  In our case, it was at least a Dev server with minimal users.

Effects of sp_rename on Stored Procedures

There comes a time when mistakes are made.  Sometimes those mistakes can be as annoying as a spelling mistake during the creation of a stored procedure.  When a mistake such as that happens, we are given a few choices.  One could either rename the stored procedure, drop and recreate the stored procedure or simply leave the mistake alone.

When choosing to rename the stored procedure, one may quickly reach for the stored procedure that can be readily used for renaming various objects.  That procedure was provided by Microsoft after-all and is named sp_rename.  Reaching for that tool however might be a mistake.  Here is what is documented about the use of sp_rename to rename a stored procedure.  That documentation can be read at this link on MSDN.

We recommend you do not use this statement to rename stored procedures, triggers, user-defined functions, or views; instead, drop the object and re-create it with the new name.

And later in the same documentation, one can read the following.

Renaming a stored procedure, function, view, or trigger will not change the name of the corresponding object name in the definition column of the sys.sql_modules catalog view. Therefore, we recommend that sp_rename not be used to rename these object types. Instead, drop and re-create the object with its new name.

Now, a chief complaint against dropping and recreating the stored procedure, as recommended, is that process can cause permissions issues.  I am less concerned about the permissions issues and see that as more of a nuisance that is easily overcome due to great documentation and a few quick script executions to restore the permissions.  Despite that, I think we might have a means to address the rename and permissions issue that will be shared later in this article.

Using sp_rename

When using sp_rename, it would be good to understand what happens and what one might expect to see.  Let’s use the following script to create a stored procedure to step through an exercise to rename a stored procedure and evaluate the results.

When I execute that series of batches, I will get an output that matches the following.

renameme

 

When looking at the results we can see that the use of sp_rename does indeed change the name of the stored procedure as it is represented via sys.objects and metadata.  We can also see that the definition of the stored procedure does not change as it is held within the metadata.

If I choose to check the definition through the use of OBJECT_DEFINITION()  instead of sys.sql_modules, you will be pleased to know that sys.sql_modules calls OBJECT_DEFINITION() to produce the definition that is seen in the catalog view.

Well, that does pose a potential problem.  We see that the object definition is unchanged and may report the name as being different than what the object name truly is.  What happens if I execute the stored procedure?  Better yet, if I can execute the stored procedure and then capture the sql text associated to that plan, what would I see?

Yes!  The renamed stored procedure does indeed execute properly.  I even get three results back for that execution.  Better yet, I get an execution plan which I can pull a plan_hash from in order to evaluate the sql text associated to the plan.  In case you are wondering, the execution plan does contain the statement text of the procedure.  But for this case, I want to look at the entire definition associated to the plan rather than the text stored in the plan.  In this particular scenario, I only see the body of the procedure and not the create statement that is held in metadata.

plan

For this particular execution and plan, I can see a plan_hash of 0xE701AFB2D865FA71.  I can now take this and provide it to the following query to find the full proc definition from metadata.

And after executing that query, I can see results similar to the following.

execplancache_text

 

Now is that because in some way the query that was just run was also running OBJECT_DEFINITION()?  Let’s look at the execution plan for both OBJECT_DEFINITION() and the query that was just run.

obj_def_plan

 

Looking at the XML for that particular plan and we see xml supporting that plan.  There is no further function callout and the plan is extremely simple.

Now looking at the plan for the query involving the query_plan_hash we will see the following.

fngetsql

 

Looking at this graphical plan, we can see that we are calling FNGETSQL.  Looking at the XML for this plan, we can verify that FNGETSQL is the only function call to retrieve the full sql text associated to this plan execution.  FNGETSQL is an internal function for SQL server used to build internal tables that might be used by various DMOs.  You can read just a bit more about that here.

What now?

After all of that, it really looks pessimistic for sp_rename.  The procedure renames but does not properly handle metadata and stored procedure definitions.  So does that mean we are stuck with drop and create as the Microsoft documentation suggests?

If you have access to the full procedure definition you could issue an alter statement.  In the little example that I have been using, I could issue the following statement.

After executing that script, I could check sys.sql_modules once again and find a more desirable result.

And my results…

finallymatching

 

If you don’t have the text to create the proc, you could use SSMS to script it out for you.  It is as simple as right-clicking the proc in question, selecting modify and then executing the script.  It should script at with the correct proc name (the beauty of SMO) and then you can get the metadata all up to snuff in your database.

Of course, if you prefer, you could just drop and recreate the procedure.  Then reapply all of the pertinent permissions.  That is pretty straight forward too.

SQL Server UG in Vegas August 2014 Meeting

Comments: No Comments
Published on: August 13, 2014

evite

 

Another Great meeting and topic is coming to the folks in Las Vegas.  This month we have had the luck of finding Mike Fal (blog | twitter) step up and fill our speaker void.

Yes, it happens to be the second Thursday of the month already.  Being that time of month, the SQL Server UG of Las Vegas will be meeting at the Tahitti Village Resort and Spa to take in some great info on SQL Server and Powershell.

You can read the information about the meeting on our Meetup page here.  Or you can continue reading here.

Improving Database Restores with Powershell

Database restores are a key function of any database administrator’s job. However, it can be an extremely time consuming task to sort through your backups, find the right files, and then get your database up and running. In an emergency this will have a disastrous impact on your Recovery Time Objective(RTO) and lead to the dreaded angry-CTO-in-your-cube effect. By leveraging some easy-to-use Powershell scripts, you can avoid the second disaster and the pain that comes with it. By attending this session, you will understand how you can use the Powershell automation framework for database restores, see scripts that will let you restore faster, and learn techniques to extend these tools for migrating data and testing backups.

Michael Fal  

Mike Fal is a musician turned SQL Server DBA, with 10+ years of experience as a database administrator. He has worked for several different industries, including healthcare, software development, marketing, and manufacturing and has experience supporting databases from 1 GB to 4 TB in size. Mike received his a Bachelor’s Degree from the University of Colorado at Boulder in 1996 and has been caught playing trombone in public on more than one occasion.

LiveMeeting Info

Attendee URL:https://www.livemeeting.com/cc/UserGroups/join?id=4RD8NP&role=attend

Meeting ID: 4RD8NP

Whether you are in Vegas or you are somewhere else, you are welcome to join us.  We hope to see you Thursday evening.

T-SQL Tuesday #57 – SQL Family and Community

Comments: 1 Comment
Published on: August 12, 2014

TSQL2sDay150x150Look at that, it is once again that time of the month that has come to be known as TSQL Tuesday.  TSQL Tuesday is a recurring blog party that occurs on the second Tuesday (most generally) of the month.  This event was the brainchild of Adam Machanic (Blog | Twitter).  

Anybody who desires to participate in this blog party is welcome to join.  Coincidentally, that open invitation is at the base of this months topic – Family and Community.  The invitation, issued by Jeffrey Verheul (blog | twitter), for this month said the following.

This month I would like to give everyone the opportunity to write about SQL Family. The first time I heard of SQL Family, was on Twitter where someone mentioned this. At first I didn’t know what to think about this. I wasn’t really active in the community, and I thought it was a little weird. They were just people you meet on the internet, and might meet in person at a conference some day. But I couldn’t be more wrong about that!

Once you start visiting events, forums, or any other involvement with the community, you’ll see I was totally wrong. I want to hear those stories. How do you feel about SQL Family? Did they help you, or did you help someone in the SQL Family? I would love to hear the stories of support, how it helped you grow and evolve, or how you would explain SQL Family to your friends and family (which I find hard). Just write about whatever topic you want, as long as it’s related to SQL Family or community.

What is it?

We have all likely seen SQL Family thrown about here and there.  But what exactly is this notion we hear about so often?

I think we have a good idea about what family might be.  I think we might even have a good idea of what a friend is.  Lastly, I might propose that we know what a community is.  When we talk of this thing called SQL Family, I like to think that it is a combination of family, friends and community.

mushroom

These are people that can come together and talk about various different things that span far beyond SQL Server.  We may only see each other at events every now and then.  Those events can be anything from a User Group meeting to a large conference or even at a road race (5k, half marathon, marathon).

These are the people that are willing to help where help is needed or wanted.  That help can be anything ranging from well wishes and prayers, to teaching about SQL Server, to lending a vehicle, or anything along that spectrum.

I have seen this community go out of their way to help provide a lift to a hotel or to the airport.  These people will help with lodging in various circumstances when/if they can.  These are the people that have been known to make visits to hospitals to give well wishes for other people in the community.

Isn’t that what friends / family really boils down to?  People that can talk to each other on an array of topics?  People that go out of their way to help?  Think about it for a minute or three.

«page 1 of 45






Calendar
November 2014
M T W T F S S
« Oct    
 12
3456789
10111213141516
17181920212223
24252627282930
Content
SQLHelp

SQLHelp


Welcome , today is Saturday, November 1, 2014