Tags: SQL

Slammer, Alive…Barely

Categories: News, Professional, SSC, SSSOLV
Tags: ,
Comments: 1 Comment
Published on: January 24, 2012

Slammer

By now you must have heard of the SQL Slammer worm.  It was quite an infectious little nuisance.  The harm it caused came largely due to unpatched, unprotected SQL Servers.

We are now 9 years out from the initial discovery of this worm.  The worm has made its way onto the endangered species list – but it is not yet extinct.  I don’t know if I should be surprised by that.

My initial reaction is “No way that worm is still causing problems.  Everybody knows about it.”  But yet, I just caught several infection attempts from remote hosts that were affected by Slammer.  When I take a step back, I recall that many people out there are still running on unpatched servers.  I know of many places that are running SQL 2000.  I know of a large pool of servers across different versions and editions that are not patched.  I even know of a few places that are still running SQL 6.5.

When I take all of that into account, finding that Slammer is still active does not surprise me – but it should.

So for fun, here is what I was able to trap from the recent attempts at my machine with SQL Slammer.

Time:		 1/23/2012 3:59:03 PM
Event:		 Intrusion
IP Address/User: 202.56.192.195
Message:	 Attack type: MSSQL Resolution Service Buffer Overflow (Slammer)

When I trace that IP back to its source, I get a host name of the machine.  If I search on the Host Name of the IP Address, I find this page.  If I were a hacker, I now have a lot of valuable information.  I can also assume that this particular host has many virii.

This entire little foray has made me wonder how many people out there are concerned about security.  Do you know what the patch level is of your server?  Is your AV software up to date?  Are you running any form of HIPS?  If you are in IT and your focus is Data, you may want to check those things.  After all, our focus is to protect the data.

Dedicated Administrator Connection

Categories: News, Professional, SSC
Tags: , , ,
Comments: No Comments
Published on: January 19, 2012

Recently you may have read my article about some hidden functions in SQL Server.  In that article you learned that those functions were in some DMOs and that you could get at them through the resource database.

Today I found myself learning more about the resource database.  Due to what I had learned in my prior foray into the resource database, I was curious if certain other functions might call some hidden functions in that database.

Sadly – they did not.  But in my travels I did happen across something else that is in that database.  Those items are called system base tables.  Unlike the trio of functions from the last article – you can get to these but it is STRONGLY advised to not do it.

Naturally, I want to check these tables out – especially since the MSDN article does say how to get to them.  I will write about some adventures into looking at these tables in the future.  I already found one interesting thing that seemed odd – but first I will need to login using the DAC and start testing to confirm a hypothesis.

For now, I want to cover how to create a Dedicated Administrator Connection.  This should be something that DBAs know how to do.  It isn’t difficult, and I will only cover one method and leave the other method to the Microsoft documentation.

You can create a DAC through either SSMS or through SQLCMD.  You can create one remotely, but you will need to enable that option since it is disabled by default.  You can find the method for creating this connection via SQLCMD here.

To create a connection through SSMS, it is rather easy as well.  You simply add (case insensitive) “admin:” to the beginning of your server as shown in this image.

In order for this to work, you will need to have the browser service running.  If it is not running, you will get an error message.  This error message is informative if you read it.  It will provide a clue to look at the browser service.

Once you have successfully created this connection, you can now use it when necessary to perform administrative tasks or for some learning opportunities.  If you open a query using this connection you will see something like this next image in your query tab.

You can see in the tab of this query tab that there is the label “ADMIN:”.  This is your DAC connection.  You are limited to one of these at a time – period.

If you try to create a second connection, you will get a nasty message.  The message is not entirely informative – just understand that you are getting it because you already have a DAC open.

It is a good idea to become familiar with how to connect via the DAC.  I have a connection saved for quick access.  Luckily I have a development server which I can test and use for learning opportunities.  As the warning MSDN states: “Access to system base tables by using DAC is designed only for Microsoft personnel, and it is not a supported customer scenario.”  If you venture into the system base tables via the DAC – Microsoft will not support it if you break it.

SSRS Export part 2 (Export Data Source)

Categories: News, Professional, SSC
Tags: , ,
Comments: No Comments
Published on: November 29, 2011

Back in August, I published a post on exporting SSRS report files en masse.  That article (which can be read here) detailed an SSIS package I created to export the files.  As it is published, it is only good for exporting the actual report files and not the data sources.

I knew of this short coming for some time and updated my SSIS package shortly after with the expectation of writing an update to that article.  Well, time went by a little too quickly.  Now it has been almost four months and I am finally getting to that update.  I am doing this all while also working out a TSQL only solution to do the same thing.  I hope to have that worked out soon with how to do it being published shortly after.

So, in keeping with the same format as the previous article, let’s start by detailing out the variables that have been added.

FileExportExtension – As the variable name should imply, this is the extension of the xml file that is to be created.  RDL would be for the report file, and RDS would be the data source (as examples).

Then inside the script task we will find the next change to be made.  The new variable we created will need to be added to the readonly variable list as shown.

So far so good.  The changes are simple and straight forward.

The next change is to the script.  Let’s click the edit script button and we can change the Main with the following.

Public Sub Main()
		'
		' Add your code here
		'
        My.Computer.FileSystem.WriteAllText(Dts.Variables("ReportExportPath").Value + Dts.Variables("ReportName").Value + "." + Dts.Variables("FileExportExtension").Value, Dts.Variables("ReportXML").Value.ToString, False)
        Dts.TaskResult = ScriptResults.Success
    End Sub

Looking at this code, you will see once again that variable that we added popping up.

One key to this working effectively is the use of the ReportSourcePath variable.  An appropriate path must be specified that contains Data Sources in the Catalog table.  An example would be /Data Sources/.  Some environments may have a subfolder after the data sources.  Just make sure that the path specified leads to the data sources you want to export.

I had also considered altering the “Execute SQL Task” that starts the flow to this package.  The script there could be altered such that another variable may be added to designate report part type.

SELECT
     ItemID,Name,[Type]
   ,CASE Type
      WHEN 2 THEN 'Report'
      WHEN 5 THEN 'Data Source'
      WHEN 7 THEN 'Report Part'
      WHEN 8 THEN 'Shared Dataset'
      ELSE 'Other'
     END AS TypeDescription
   ,CONVERT(VARBINARY(MAX),Content) AS Content
   FROM ReportServer.dbo.CATALOG
   WHERE Type IN (2,5,8)
	And LEFT(PATH,LEN(@ReportPath)) = @ReportPath

The change would add another variable into this query in the where clause.  Change the type from in to an equality.  Add a variable that would designate the different types listed in the case statement – and it becomes pretty straight forward.  This change would allow more flexibility.  I will update at a later time with the changes I have made to the package to permit that.  But for now, I felt it more of a bonus addition and didn’t need it quite yet.  (Necessity drives functionality, right?)

If you make these suggested changes, you will have more flexibility in being able to export the various files related to reporting.  If you have played with Report Manager, you will know that there is no way to export an RDS file.  Now, you have a means to export the xml into a usable file that can be imported to another server – if you need it.

Check back in the future for that update to do this using TSQL as well as for the update to provide more flexibility to the package.

 

 

SQL Deep Dives 2 on the Kindle

Categories: News, Professional, SSC
Tags: ,
Comments: No Comments
Published on: November 23, 2011

Since Deep Dives 2 came out, I had been putting off getting my copy of the e-book until I wanted the book for travel purposes.  I decided I really needed to have it loaded on the kindle and quickly started running into a few roadblocks.

First roadblock was easy to overcome.  That required an email to Manning to get the beta link for the ebooks.  If you purchased through Manning, I’d recommend checking the beta site for any e-book purchases.

The second roadblock was determining which file to use on the kindle.  E-book formats are .mobi, .pdf and another that escapes me right now.  I didn’t see one for kindle.  In my journeys though I learned that mobi is essentially the same format as the azw format used by Amazon for the Kindle.  That is very good to know.

The next roadblock was how to get the file onto the kindle.  Getting it into my PC Kindle was pretty easy.  Find the \Documents\My Kindle Content folder in your user profile directory.  Then copy the mobi file to that directory.  But despite that, syncing did not put that book onto the rest of my kindle apps.  I soon found two ways of getting that done.

The first method for getting those mobi files onto the kindle was to plug my kindle into the usb port and copy the mobi file onto it.  Still, it didn’t sync to the rest of my devices.  Flip side is that it was pretty fast.

The second method for getting those mobi files onto the kindles is to email your kindle email account.  Amazon will eventually make those files available for you.  This will make it so you can sync all of your kindle devices with the same files.  The drawback is that it is considerably slower.

The same process can be done for any of those SQL books you have that you want to port around with you.  Now, the book is updated on several devices for me (laptop, phones, kindle) and I can reference it much faster than lugging the book around everywhere I go.

Try it and enjoy!

SQL Family – an Update

Categories: News, Professional, SSC
Tags: ,
Comments: 2 Comments
Published on: November 21, 2011

At the beginning of the month, we had a Meme Monday on the topic of SQL Family.  I had a few things to say about the SQL community back then.  And now, I want to give a bit of an update on the topic that supports what I have already said.

The Story

Late last Wednesday I learned from my wife (I was at the time on the other side of the continent) that our two year old daughter needed to have “emergency” surgery on her nose.  My wife was understandably concerned.  I was a bit more freaked out than she was – and yes we were both really worried.

My daughter was having problems breathing and her nose and cheek were swollen.  My wife took her to the doctor suspecting that it may have been broken by a head-on collision with her older brother.  The doctors at the clinic referred her to specialists saying they felt surgery was necessary.  There was a white sliver poking through skin internally in the nostril.

By the next day when my wife had gone to the specialist (this visit was Wednesday and was the one that got us a bit more concerned) that sliver had gotten larger.  Add that my daughter was getting frequent nosebleeds and you may just have the picture now.  The specialist told my wife that they needed to operate Thursday morning and fix it.  The would have to slice this protrusion off and sew the nose.

The doctor tried to pull the white sliver from the nose and nothing moved.  This was kind of weird to me since they had called the sliver “cartilage.”  It also made the panic go up a bit more.

Thursday morning, I commented on twitter that my daughter was having the surgery.  Many thanks again to all of you that replied both publicly and privately.  This is what I mean about community.  I was trying to work but also trying be with my daughter in spirit.

Update

After the surgery I got a text message from my sister in law about almonds.  I was confused by the text and decided to call back.  It happens that she was with my wife at the hospital and the almonds reference was in regards to the nose surgery.  It turns out that my daughter had sneezed while eating some almonds.  Some pieces (large and small) had traversed through that opening between nose and throat at the back of the mouth.  Those pieces became lodged in her nose.  One was too large to completely pass.  Some of the almond skin and cut into her skin and did have to be surgically removed.  In short, without surgery, none of it could have been removed.

That is quite the relief!  It is also something we can look back on and laugh about now – embarrassing as it may seem.

Again, thanks to all who expressed interest and concern.  It is very much appreciated.

SSSOLV November 2011 Meeting

Tags: , ,
Comments: No Comments
Published on: November 7, 2011

Another month, another meeting.  Time really is flying.  And now with more Holidays fast approaching, I am sure that time will warp on us.

The Las Vegas user group is happy to announce that we have a new topic and new presenter (new to us anyway) for the month of November.  Norm Kelm is prepared to teach us a few things about his new hammer.

POWERSHELL, THE NEW SQL HAMMER

You’ve seen all the amazing scripts that use PowerShell, but writing your own is raising more questions. This session will help fill in the gaps by explaining all the moving parts of PowerShell 2.0, the integration with SQL Server and answer the following questions as well as others. Why is Invoke-Sqlcmd necessary? What is and why is there a Minishell for SQL Server? What makes the SQLSERVER: PS Drive so powerful? How does a remote SQL Server get added to the SQLSERVER: PS Drive?

Norms Bio

Norman Kelm is the owner of Gerasus Software, http://www.gerasus.com/, the maker of SSIS-DTS Package Search the only utility that can search SSIS and DTS packages. Norman has been working in IT for over 20 years. He worked as a FORTRAN and C developer for 8 years before making the jump to databases with Sybase. He then moved on to SQL Server working as a production and development DBA on SQL Server since version 6.5. He is also a founding member of the Tampa Bay SQL Server User Group.

We welcome all to attend (as we do every month).  Here is the online meeting information:

LiveMeeting Information:
 Attendee URL:  https://www.livemeeting.com/cc/UserGroups/join?id=2JK8TZ&role=attend
 Meeting ID:  2JK8TZ
In Person Attendees:
The meeting location has changed.  We will no longer be meeting at The Learning Center.  New meeting location is M Staff Solutions & Training / 2620 Regatta Drive Suite 102 Las Vegas, NV 89128.

We are Family

Categories: News, Professional, SSC
Tags: ,
Comments: No Comments
Published on: November 7, 2011

It has been a while since I last wrote something for a meme monday.  This month is a perfect time to jump back in the deep end.

If you don’t know what a meme monday is, then here is a little recap.  Thomas LaRock started this to assist in getting the blog writing juices flowing.  He provides a simple, usually open ended topic.  As with most memes, you should write something if you get “tagged,” but you can participate without the tag.  In fact, I usually forget to tag somebody and have yet to be tagged (to my knowledge anyway).

This month, the topic is SQL Family.

SQL Family

What is the SQL Family?  Well, loosely stated I think of a family as a close knit group of individuals working together with common goals and helping each other – a lot.

How does that translate to the SQL Family?  The SQL Family is very big from my point of view.  I have blogged a few times about this family.  You will usually see me referring to this family as community.

What does this SQL Family do for you and me?  A LOT!!!  Here are some of the things I know that this group of people does for you and me.

Support – Have you ever experienced something that is not SQL related?  This group of individuals will lend a hand, shoulder, limb or what is needed in many circumstances.  From bullying to death to divorce to a hospitalized child at one end of the spectrum to marriage, child birth, awards and recognition at the other end.  This group will be there for you.

Train – what other group of professionals regularly gives of time and money to train and teach you every weekend – somewhere/somehow?  It’s not just the training on the weekends – it goes far deeper than that.  Many write books on the topic of SQL.  And these people make themselves available to train via twitter, forums, email, and in person.  Granted, not all of this training is free – but there is a ton of it that is free.

Help – Have you heard of the twitter hashtag #sqlhelp?  How about the msdn forums?  Have you heard of SQLServerCentral?  How about Stackoverflow?  There are so many people giving of their time to help other database people get better at what they do.  On the other side of the coin, the family is also there to help proof read and tech edit articles and books.  Not to mention that they often times find themselves helping by lending a professional opinion from anything SQL related at work to professional behavior in the workplace.

Fraternize – The SQL family, like most families, is not all work all the time.  There is a lot of play, chit chat and hanging out going on in this group.  This comes in real handy when you are out of town – in a place you don’t know too well.  There is plenty of joking as these people become friends.  There is a level of trust and respect that is gained.  So much so that these people will jump to your side quickly to defend you – just as you would do for your little sister or brother.

Network – Have you ever been in need of a break?  Maybe just a way to get to that next job?  The network that comes with this family is pretty vast.  You will have exposure to many more opportunities than you might have expected looking on your own.

These are all benefits of the SQL Family.  I have talked a lot about what they do for you.  As with good families, it goes both ways.  It’s not required for you to do all of these things.  But, once you start seeing what has been done for you – it is very hard to not reciprocate in some way.  The more everybody contributes, the stronger this family becomes.  Not only does the family become stronger – but you gain and grow far more than a) without the family and b) just leaching from the family.

Data Head

Categories: News, Professional, SSC
Tags: ,
Comments: No Comments
Published on: September 27, 2011

Well, it’s official.  I am a Data Head.  It doesn’t change too much my level of geekiness.  But I may have to rethink one of the answers given during that interview.

The Data Head profiles are being done by a training company called Data Education.  The company is another of Adam Machanics creations and is based out of Boston.

The people at Data Education contacted me a month or so ago and asked me a few questions.  I thought it was a cool idea and answered the questions and provided them with a profile picture.

There are two other Data Head profiles to this point.

Both of these people are MVPs.  The only Microsoft title I could associate to my name is via my certifications (MCITP and MCDBA for instance).  That brings up one small update to that profile – the fact that I have the MCITP too and not just MCDBA (which as you know is expired).

I put more thought into the question on geekiest thing ever done.  If I could do it, I would totally do this.  Well, at least come up with a way (maybe a remote built into the gloves) to make it appear like I had something beyond the TSQL Force.

Swing by Data Education and check things out – at the very least check out the Data Heads.

Performance Tuning – Over the Top Biz Logic

Comments: No Comments
Published on: September 19, 2011

This month, Robert Pearl is helping to host a Performance Story contest.  The contest is being sponsored by Safe Peak, and you can see more about it – here.

The timing on this contest actually fits well with something I have already been working on.  The only delay has been related to getting everything scrubbed and obfuscated.  There is a lot that could be included with this particular experience, I plan on touching on two of the key areas.

Backstory

I have been working a lot lately with building reports in SSRS.  These reports had been working great from IE hitting directly to the SSRS Server in one environment.  When we migrated the reports to the production environment – we started seeing some serious timeout issues.

There was one report in particular that was giving more grief than others.  I had already gone through everything I could find with regards to SSRS in trying to help speed these reports up to acceptable times.

Having gone through what could be done in SSRS, I next turned my attention to the main stored procedure creating the data-set for the report.

Stored Procedure

Although the report was not even making it to this particular stored procedure – I wanted to take a look at it and make any improvements there that I could.  It made sense to eliminate it as a possible source of the problem.

I opened up the stored procedure and found a fair amount if Business Logic.  I came to a section of the proc that contained code that at first sight, made my head spin a bit.  I thought for sure I was seeing double because of all of the logic.

Once I finally got my head wrapped around it, I noticed a lot of similarities and wanted to start from that particular section with my tuning.  Here is an obfuscated version of what I saw.

DECLARE @CompareParam INT = 200
 
		SELECT tccv.MyColumn
			FROM dbo.table1 AS cb
				INNER JOIN dbo.table2 tcfbt
					ON cb.column1 = tcfbt.column1fk
    			INNER JOIN dbo.table3 AS cpd
    				ON cb.column2 = cpd.column2fk
				INNER JOIN dbo.table4 AS tcfcpt
					ON cpd.column3 = tcfcpt.column3fk
				INNER JOIN dbo.table5 AS tcfcpct
					ON cpd.column4 = tcfcpct.column4fk
				INNER JOIN dbo.table6 cp
					ON cb.column5 = cp.column5fk
				INNER JOIN dbo.table7 cal
					ON cp.column6 = cal.column6fk
				INNER JOIN dbo.table8 tgmc
					ON cal.column7 = tgmc.column7fk
				INNER JOIN dbo.table9 tccv
					ON tgmc.column8 = tccv.column8fk
				INNER JOIN dbo.table10 fps
					ON cp.column9 = fps.column9fk
        WHERE
            cb.column5 = @CompareParam
	        and
	        (
	            tcfbt.CodeVal = 'In-Progress'   -- use cases 1,2,3
	            and
                (
                    ( -- use case 1
                        cp.column5fk = @CompareParam
                        and fps.CodeVal not in ('Val1','Val2')
                        and tccv.Date1 between cp.Date2 and ISNULL(cp.Date3, cp.Date4)
                    )
                    --use case 2
                    or
                    ( cp.column5fk = @CompareParam
						and tccv.Date1 < cp.Date2
						and not exists (
							SELECT 1
								FROM dbo.table11 cvcp
									INNER JOIN dbo.table6 prevcp
										ON cvcp.column5 = prevcp.column5fk
									INNER JOIN dbo.table10 prevfps
										ON prevfps.column9fk = prevcp.column9
								WHERE prevcp.column6 = cp.column6
									and prevcp.Date4 < cp.Date2
									and cvcp.MyColumnfk = tccv.MyColumn
									and prevfps.CodeVal in ('Val1','Val2')
						)
						and exists (
							SELECT 1
								FROM dbo.table6 AS pastcp
									INNER JOIN dbo.table10 AS pastfps
										ON pastcp.column9 = pastfps.column9fk
								WHERE pastcp.column6 = cp.column6
									and pastcp.Date4 < cp.Date2
									and tccv.Date1 between pastcp.Date2 and pastcp.Date4
									and pastfps.CodeVal in ('Val1','Val2')
						)
                    )
                )
             )
             or
             (
				-- same as the other side of 'OR' just above
                 tcfbt.CodeVal = 'Live'
				and not exists (
					SELECT 1
						FROM dbo.table12 cvh
						WHERE tccv.MyColumn = cvh.MyColumnfk
				 )
                 and
                 (
                    ( -- use case 1
                        cp.column5fk = @CompareParam
                        and fps.CodeVal not in ('Val1','Val2')
                        and tccv.Date1 between cp.Date2 and ISNULL(cp.Date3, cp.Date4)
                    )
                    --use case 2
                    or
                    (
						cp.column5fk = @CompareParam
						and tccv.Date1 < cp.Date2
						and not exists (
							SELECT 1
								FROM dbo.table11 cvcp
									INNER JOIN dbo.table6 prevcp
										ON cvcp.column5fk = prevcp.column5fk
									INNER JOIN dbo.table10 AS prevfps
										ON prevcp.column9 = prevfps.column9fk
								WHERE prevcp.column6 = cp.column6
									and prevcp.Date4 < cp.Date2
									and cvcp.MyColumnfk = tccv.MyColumn
									and prevfps.CodeVal in ('Val1','Val2')
						)
						and exists (
							SELECT 1
								FROM dbo.table6 AS pastcp
								INNER JOIN dbo.table10 AS pastfps
									ON pastcp.column9 = pastfps.column9fk
							WHERE pastcp.column6 = cp.column6
								and pastcp.Date4 < cp.Date2
								and tccv.Date1 between pastcp.Date2 and pastcp.Date4
								and pastfps.CodeVal in ('Val1','Val2')
						)
                    )
				)
	          )

I want you to look closely at both sides of that OR condition in the where clause before proceeding to look at the image of the execution plan that this query generates.

Again, my head started spinning when I first looked at the execution plan.  It was the size of the overall plan that was causing it this time.  Looking through it though, I quickly saw what I thought was the main point of slowness with this query (yes it was slow).  If you open the image of that execution plan and compare the red section to the yellow section, you will see that it is duplicated.  These sections also correlate to each side of the OR condition that was already mentioned.

When browsing this execution  plan in SSMS, I was also lured into these sections due to a bad estimate on the query cost (which you don’t see in this image).  All of the Index Spools and Index scans/seeks had a cost of 89% attached to them (within the red and yellow regions).  Seeing that both sections were identical – I was certain that the OR condition could be optimized.

DeDuping

Since the queries were soooo similar, I knew there had to be a way to combine both sides and make it less costly.  After working on it a bit, I came up with the following query.

DECLARE @CompareParam INT = 200
 
		SELECT tccv.MyColumn
			FROM dbo.table1 AS cb
				INNER JOIN dbo.table2 tcfbt
					ON cb.column1 = tcfbt.column1fk
    			INNER JOIN dbo.table3 AS cpd
    				ON cb.column2 = cpd.column2fk
				INNER JOIN dbo.table4 AS tcfcpt
					ON cpd.column3 = tcfcpt.column3fk
				INNER JOIN dbo.table5 AS tcfcpct
					ON cpd.column4 = tcfcpct.column4fk
				INNER JOIN dbo.table6 cp
					ON cb.column5 = cp.column5fk
				INNER JOIN dbo.table7 cal
					ON cp.column6 = cal.column6fk
				INNER JOIN dbo.table8 tgmc
					ON cal.column7 = tgmc.column7fk
				INNER JOIN dbo.table9 tccv
					ON tgmc.column8 = tccv.column8fk
				INNER JOIN dbo.table10 fps
					ON cp.column9 = fps.column9fk
        WHERE
            cb.calendarPeriodID = @CompareParam
	        and
	        (
	           ( tcfbt.CodeVal = 'In-Progress'   -- use cases 1,2,3
						OR (tcfbt.CodeVal = 'Live'
							and not exists (
								SELECT 1
									FROM dbo.table12 cvh
									WHERE tccv.MyColumn = cvh.MyColumnfk
							)
						)
					)
	            and
                (
                    ( -- use case 1
                        cp.column5fk = @CompareParam
                        and fps.CodeVal not in ('Val1','Val2')
                        and tccv.Date1 between cp.Date2 and ISNULL(cp.Date3, cp.Date4)
                    )
                    --use case 2
                    or
                    ( cp.column5fk = @CompareParam
						and tccv.Date1 < cp.Date2
						and not exists (
							SELECT 1
								FROM dbo.table11 cvcp
									INNER JOIN dbo.table6 prevcp
										ON cvcp.column5fk = prevcp.column5fk
									INNER JOIN dbo.table10 prevfps
										ON prevfps.column9fk = prevcp.column9
								WHERE prevcp.[calendarID] = cp.[calendarID]
									and prevcp.Date4 < cp.Date2
									and cvcp.MyColumnfk = tccv.MyColumn
									and prevfps.CodeVal in ('Val1','Val2')
						)
						and exists (
							SELECT 1
								FROM dbo.table6 AS pastcp
									INNER JOIN dbo.table10 AS pastfps
										ON pastcp.column9 = pastfps.column9fk
								WHERE pastcp.column6 = cp.column6
									and pastcp.Date4 < cp.Date2
									and tccv.Date1 between pastcp.Date2 and pastcp.Date4
									and pastfps.CodeVal in ('Val1','Val2')
						)
                    )
                )
             )

Now take a close look at the first part of the where clause.

WHERE
            cb.calendarPeriodID = @CompareParam
	        and
	        (
	           ( tcfbt.CodeVal = 'In-Progress'   -- use cases 1,2,3
						OR (tcfbt.CodeVal = 'Live'
							and not exists (
								SELECT 1
									FROM dbo.table12 cvh
									WHERE tccv.MyColumn = cvh.MyColumnfk
							)
						)
					)
	            and

That little change represents the only difference between the two sides of the Or condition in the prior query.  The result set is exactly the same.  Now we get a new execution plan that looks like this.

In this execution plan, you can see that there is now only one section matching the red and yellow sections from the previous query.  This is a very good thing.  We can see that the plan looks simpler than the last one.  What about if we execute the two queries in a single batch for comparison?

 

The improvements we see are substantial.  Look at the difference in cost comparison between the two!!  The old query consumes 97% of the cost.  To further illustrate the improvements by making this little change in logic, let’s take a look at the time.

Timecompare_withExecplan

Timecompare_withExecplan

Timecompare_withoutExecplan

Timecompare_withoutExecplan

 

 

 

 

 

 

 

 

Looking at the time it takes to run these queries should also be a strong indicator of the performance gains made by simplifying that where clause.  Let’s also take a look at another couple of screen-shots that helps to show some of the different improvements made by simplifying that “OR” condition.

Notice the old plan has an index scan, sort operation and 61 parallel operations.  The new plan has a key lookup.  Now look at the glaring difference between the two in estimated rows.  That is significant.

That is not all that was done with the query itself to improve performance.  Other things like index tuning and more work on the rest of the query was also done with some improvements in overall execution of the query.  But none were as significant as this.

TCP/IP

The other place where significant improvement was made in the performance of this query came from something outside of SQL Server.  A key part of this exercise has not yet been mentioned.  The reports were all taking 30-60 seconds to even load without any parameters having been selected.  Then they would take another 30 seconds or so after that to bring up the next parameter.

We ran a trace between the two servers to see what we could find.  After some more troubleshooting, we decided to try disabling the TCP Chimney.  This change had a significant impact in loading the reports from SSRS.  From within IE, these reports were now immediately loading and the prompt flow became very responsive.

Conclusion

Taking a little bit of extra time when writing out the logic for your queries can have a profound impact on the responsiveness of your queries.  In this case, simplifying the code but retaining the logic had a tremendous effect on overall performance.

In addition to what can be done in SQL Server, sometimes one needs to look outside of SQL to the network or disk layer for further tuning.  Working with the appropriate teams to accomplish those goals can go miles to having a positive effect on co-workers and application experience by the business users.

T-SQL Tuesday #22 – Data Presentation

Tags: ,
Comments: No Comments
Published on: September 13, 2011

Tuesday is upon us.  It also happens to be the second Tuesday of the month and you know what that means.  It is time for TSQL Tuesday.  This month, Robert Pearl, a friend, is hosting.  Read his invitation here.

As the invitation suggests, this topic is wide open.  There are many ways to present data.  Some of those methods may be better than others.  There are a few things of note with Data Presentation that will impact how much better your Data Presentation will be.

The items I will discuss that will help your data be better presented are: Performance, Accuracy, Display, and Business Requirements.  I will use a few scripts to progress through each of these topics.  Let’s start with Performance.

Performance

Why is performance included in this discussion?  How does performance relate to data presentation?  Well, have you ever had an end-user complain that a report was utterly useless because it was too slow?  If the report is too slow, it won’t matter if the data is accurate in the eyes of some.  They needed to have the data yesterday and it simply took too long.

Here is an example of a query that could be optimized a bit more (admittedly this query does not perform soooo slowly that a user would give up on it – by the end you will see that it could perform better).

DECLARE @StartDate DATETIME = '2011-09-08'
		,@EndDate DATETIME = '2013-11-09'
		,@numMonths TINYINT
DECLARE @calendarTable TABLE (monthNum TINYINT, monthDays TINYINT)
 
SET @numMonths = (SELECT DATEDIFF(m,@StartDate,@EndDate) + 1)
 
WHILE @numMonths >= 1
        BEGIN
                INSERT INTO @calendarTable SELECT MONTH(@StartDate), DATEDIFF(d, @startDate, DATEADD(m, 1, DATEADD(d, 1 - DAY(@StartDate), @StartDate)))
                SET @numMonths = @numMonths - 1
                SET @StartDate = (SELECT DATEADD(m, 1, DATEADD(d, 1 - DAY(@StartDate), @StartDate)))
                --SELECT @StartDate
        END
 
SELECT * FROM @calendarTable
 
Go

The requirements for this script are simple.  Provide the number of days in a month in a given date range.  If the starting date provided is not the first of the month, then we must only provide the number of days from that date to the end of the month.

As you can see, this script utilizes a looping mechanism to provide this information.  The loop inserts into a table variable one record at a time.  When examining the execution plan and the execution time on this query, one would see that the insert is the most expensive part of the query.  One would also find that this query does take a fair amount of time to run – despite its’ simplicity.  On my machine, it takes about 1.3 seconds to execute.

Certainly, if this were a more complicated query, one would see that this type of query could cause some delays in data rendering and subsequently cause grief for the end-user and you.

There is one more issue with the provided query in that it doesn’t meet all requirements.  I neglected to mention that the date format needs to accept date/month/year format.  Yes it is nitpicking, but it was a requirement and an invalid date is far too easy to submit with this query.

Accuracy

There should be no question on how accuracy affects data presentation.  No matter how pretty the rest of the data/report may look, wrong data will render the report useless in the eyes of many business users.

Again, this example is not extreme – but it does create sufficient concern that the query should not be used – unless fixed.

SET DATEFORMAT DMY
 
DECLARE @StartDate      DATE = '05/06/2011'
        ,@EndDate       DATE = '31/08/2012'
 
;
WITH getmonths AS (
        SELECT CAST(CONVERT(VARCHAR,'01/' + CAST(MonthNum AS VARCHAR) + '/' + CAST(yr.yr AS VARCHAR)) AS DATE) AS Moy
                FROM ( VALUES(1),(2),(3),(4),(5),(6),(7),(8),(9),(10),(11),(12)) Mnths (MonthNum)
                Cross Apply (SELECT YEAR(@StartDate) AS yr
                                                UNION
                                                SELECT YEAR(@EndDate) AS yr
                                        ) yr
), inputdates AS (
        SELECT dates FROM (VALUES (@StartDate),(@EndDate)) D (Dates)
        )
 
SELECT DATENAME(m,g.moy) AS MonthInRange,YEAR(g.moy) AS YrInRange
                ,DATEPART(d,DATEADD(mm, DATEDIFF(mm, 0, g.moy)+1, 0)-1)
                        - CASE WHEN DATEPART(d,id.Dates) = DATEPART(d,DATEADD(mm, DATEDIFF(mm, 0, g.moy)+1, 0)-1)
                                                THEN 0
                                        WHEN DATEPART(d,g.moy) < DATEPART(d,id.Dates)
                                                THEN DATEPART(d,id.Dates)
                                        ELSE Isnull(DATEPART(d,id.Dates),DATEPART(d,g.moy)-1)
                                END AS DaysInMonthInRange
        FROM getmonths g
                LEFT Outer Join inputdates Id
                        ON DATEPART(m,g.moy) = DATEPART(m,id.Dates)
                        And YEAR(g.moy) = YEAR(id.Dates)
        WHERE g.moy between @StartDate and @EndDate
                Or id.Dates between @StartDate and @EndDate
 
Go

Though this query works faster than the first query, it is not entirely accurate.  This query only supports a max of 12 months.  Also, this query is overly complex.  There is value in keeping this particular query more simple.

Display

This topic could be deemed to be largely a matter of preference.  That said, there is great value in how you display the data to the end user.  Displaying the data involves such things as useful column headings, and meaningful data.  For instance, naming a column “Month” but displaying a year value in it – is not very useful.  This could also overlap with the prior topic of accurate data.

Personally, when I am displaying the month, I like to see month names.  In my experience, displaying the name of the month is more easily recognizable than the month number for many users.

SET STATISTICS TIME ON
SET DATEFORMAT DMY
 
DECLARE @StartDate   DATETIME = '05/06/2011',
        @EndDate     DATETIME = '31/08/2013'
 
BEGIN
	WITH Nbrs_2( n ) AS (SELECT 1 UNION all
					SELECT 1 UNION all
					SELECT 1 UNION all
					SELECT 1 UNION all
					SELECT 1 UNION all
					SELECT 1 UNION all
					SELECT 1 UNION all
					SELECT 1 UNION all
					SELECT 1 UNION all
					SELECT 0),
			Nbrs_3( n ) AS ( SELECT 1 FROM Nbrs_2 n1 CROSS JOIN Nbrs_2 n2 CROSS JOIN Nbrs_2 n3)
	, tally (n) AS (SELECT ROW_NUMBER() OVER (ORDER BY n)
							   FROM Nbrs_3
							   )
	,cteMonthEnd AS
	(
	 SELECT t.N,
			MonthEnd = DATEADD(mm,DATEDIFF(mm,0,@StartDate) + t.N, 0) - 1
	   FROM Tally t
	  WHERE t.N BETWEEN 1 AND DATEDIFF(mm,@StartDate,@EndDate) + 1
	)
	 SELECT MonthEnd
		,NumDays     = DATEPART(dd,MonthEnd) - CASE WHEN N > 1 THEN 0 ELSE DATEPART(dd,@StartDate) END
	   FROM cteMonthEnd
	   ORDER BY MonthEnd ASC
END

In this case, the Column Heading matches the data and works.  However, a little change could make this query and output more useful to an end user who is quickly scanning over the results.

SET STATISTICS TIME ON
SET DATEFORMAT DMY
 
DECLARE @StartDate   DATETIME = '05/06/2011',
        @EndDate     DATETIME = '31/08/2013'
 
BEGIN
	WITH Nbrs_2( n ) AS (SELECT 1 UNION all
					SELECT 1 UNION all
					SELECT 1 UNION all
					SELECT 1 UNION all
					SELECT 1 UNION all
					SELECT 1 UNION all
					SELECT 1 UNION all
					SELECT 1 UNION all
					SELECT 1 UNION all
					SELECT 0),
			Nbrs_3( n ) AS ( SELECT 1 FROM Nbrs_2 n1 CROSS JOIN Nbrs_2 n2 CROSS JOIN Nbrs_2 n3)
	, tally (n) AS (SELECT ROW_NUMBER() OVER (ORDER BY n)
							   FROM Nbrs_3
							   )
	,cteMonthEnd AS
	(
	 SELECT t.N,
			MonthEnd = DATEADD(mm,DATEDIFF(mm,0,@StartDate) + t.N, 0) - 1
	   FROM Tally t
	  WHERE t.N BETWEEN 1 AND DATEDIFF(mm,@StartDate,@EndDate) + 1
	)
	 SELECT DATENAME(m,MonthEnd) AS [MONTH],YEAR(MonthEnd) AS [YEAR]
		,NumDays     = DATEPART(dd,MonthEnd) - CASE WHEN N > 1 THEN 0 ELSE DATEPART(dd,@StartDate) END
	   FROM cteMonthEnd
	   ORDER BY MonthEnd ASC
END

To this point, each query has progressively increased in performance.  This last query (both versions) also matches the need to accurately display the data.

The display of data is closely coupled with the next topic – Business Requirements.

Business Requirements

When looking at the results of the last two queries, which one is more accurate?  Well, that entirely depends on what the business has specified as acceptable output for this query.  In this case, both could be correct or both could be wrong.

It is essential to have the requirements written down and understood by all parties so as to avoid any misconception or misunderstanding of what the query should display.  Matching your output to the specified business requirements will help to provide an overall impression of accuracy and usefulness.  It will also help to achieve faster sign-off by the business.

Conclusion

Data Presentation is not just about the look of a report.  Data Presentation involves performance, accuracy, display and the business requirements.  By giving proper attention to each of these facets, the Data presented will be both remarkable and acceptable to the end-user.

page 1 of 7»
Calendar
February 2012
M T W T F S S
« Jan    
 12345
6789101112
13141516171819
20212223242526
272829  
Follow me on Google+
Jason Brimhall

In 246 people's circles

Add to circlesi
Content
Categories

Categories

Now Reading

Now Reading

Planned books:

Current books:

  • ChiRunning: A Revolutionary Approach to Effortless, Injury-Free Running

    ChiRunning: A Revolutionary Approach to Effortless, Injury-Free Running by Danny Dreyer, Katherine Dreyer

  • Advanced Marathoning – 2nd Edition

    Advanced Marathoning – 2nd Edition by Peter Pfitzinger, Scott Douglas

  • SQL Server MVP Deep Dives

    SQL Server MVP Deep Dives by Nielsen Paul, Delaney Kalen, Machanic Adam, Tripp Kimberly, Randal Paul, Low Greg

  • A World Without Heroes (Beyonders)

    A World Without Heroes (Beyonders) by Brandon Mull

Recent books:

View full Library

SQLHelp

SQLHelp


Welcome , today is Saturday, February 4, 2012