Using SQL Server 2005+ index included columns to improve performance of an ASP.net resource

This post shows a real scenario on using included columns on SQL Server 2005/2008 to improve the response time of an ASP.net resource.

When developing DotNetBurner, i wanted to make sure that the site performance was the best it could be. Targeting this goal, i have used indexes with included columns to improve the response time of the "burn!" count image. The burn count image shows how many "burns" (votes) a particularly link has. So, to retrieve this vote count, the database must be queried, like this:

SELECT VoteCount
FROM Story
WHERE UrlHash = @urlHash

The query result is filtered by the UrlHash column, which is a MD5 hash of the URL. Assuming that we have thousands of stories, it's recommended to create an index on the UrlHash column.

The following queries will create a Story table and populate it with some items, so we can query it and get the execution plan:

IF OBJECT_ID('Story') IS NOT NULL
 DROP TABLE Story
GO

CREATE TABLE Story
(
 StoryId INT PRIMARY KEY NOT NULL,
 Description NVARCHAR(2000),
 Url VARCHAR(255),
 UrlHash UNIQUEIDENTIFIER, /* This is not MD5, but can be used for this example */
 VoteCount SMALLINT
)
GO

DECLARE @count INT

-- Inserts the first story, so we know the UrlHash
INSERT INTO Story (StoryId, Description, Url, UrlHash, VoteCount)
VALUES(1, 'This is a story description where the quick brown fox jumped over the lazy dog which had a tatoo written lorem ipsum dolor.',
    'http://www.someurl.com', '5946D715-38FA-42C5-8056-F8F0C86B8AE8', 2)

-- Inserts more 9998 stories
SET @count = 2
WHILE @count < 10000
BEGIN

 INSERT INTO Story (StoryId, Description, Url, UrlHash, VoteCount)
 VALUES(@count, 'This is a story description where the quick brown fox jumped over the lazy dog which had a tatoo written lorem ipsum dolor.',
     'http://www.someurl.com', NEWID(), @count + 10)

 SET @count = @count + 1
END


CREATE UNIQUE INDEX IDX_Story_1 ON Story(UrlHash)

SELECT VoteCount
FROM Story
WHERE UrlHash = '5946D715-38FA-42C5-8056-F8F0C86B8AE8'

This is the resulting execution plan (with a index on the UrlHash column):

Looking at the execution plan, you will see a key lookup. I bet with you that i can reduce it to one index seek :)

DROP INDEX IDX_Story_1 ON Story

CREATE UNIQUE INDEX IDX_Story_1 ON Story(UrlHash) INCLUDE (VoteCount)

SELECT VoteCount
FROM Story
WHERE UrlHash = '5946D715-38FA-42C5-8056-F8F0C86B8AE8'

Now we have replaced the index which one that includes the VoteCount column. Let’s look at the execution plan:

Now this executes faster and uses less resources. This happens because when we added the included column all the values necessary for the query (UrlHash and VoteCount) can be found on the Index, so SQL Server doesn't need to look for the value of VoteCount at the clustered index.

You could, of course, just add the VoteCount directly on the index, like this:

CREATE UNIQUE INDEX IDX_Story_1 ON Story(UrlHash, VoteCount)

This would work, but would also use more disk space, since the index key would be bigger. This happens because when you use the INCLUDE option SQL Server just stores the value at the leaf node of the index, because this value is just returned, not filtered.

Using the PIVOT operator on SQL Server 2005+

The PIVOT operator is used to transform rows into column by an aggregate function. One big limitation is that you have to know the columns, but still, it's useful on some use cases, like when you want to aggregate something by hours (for example).

About having to know the column names limitation: This is probably a design decision, because someone could write a query that has too many columns and cause all sort of problems, like exceeding the limit of columns or return column that the application is not prepared to receive.

To keep the example easy to understand we will create a simple Authentication table (which holds authentications):

IF OBJECT_ID('Authentication') IS NOT NULL
 DROP TABLE Authentication

CREATE TABLE Authentication
(
 AuthenticationId INT,
 Date SMALLDATETIME,
 DocumentId UNIQUEIDENTIFIER
)

INSERT INTO Authentication (AuthenticationId, Date, DocumentId) VALUES (1, DATEADD(hh, +1, GETDATE()), NEWID())
INSERT INTO Authentication (AuthenticationId, Date, DocumentId) VALUES (2, DATEADD(hh, +1, GETDATE()), NEWID())
INSERT INTO Authentication (AuthenticationId, Date, DocumentId) VALUES (3, DATEADD(hh, +2, GETDATE()), NEWID())
INSERT INTO Authentication (AuthenticationId, Date, DocumentId) VALUES (4, DATEADD(hh, +3, GETDATE()), NEWID())
INSERT INTO Authentication (AuthenticationId, Date, DocumentId) VALUES (5, DATEADD(hh, +4, GETDATE()), NEWID())
INSERT INTO Authentication (AuthenticationId, Date, DocumentId) VALUES (6, DATEADD(hh, +4, GETDATE()), NEWID())
INSERT INTO Authentication (AuthenticationId, Date, DocumentId) VALUES (7, DATEADD(hh, +5, GETDATE()), NEWID())
INSERT INTO Authentication (AuthenticationId, Date, DocumentId) VALUES (8, DATEADD(hh, +6, GETDATE()), NEWID())
INSERT INTO Authentication (AuthenticationId, Date, DocumentId) VALUES (9, DATEADD(hh, +7, GETDATE()), NEWID())
INSERT INTO Authentication (AuthenticationId, Date, DocumentId) VALUES (10, DATEADD(hh, +7, GETDATE()), NEWID())
INSERT INTO Authentication (AuthenticationId, Date, DocumentId) VALUES (11, DATEADD(hh, +8, GETDATE()), NEWID())

Now we have some authentications. Let's say some manager wants a report of authentications by hour to know how many people comes to pay bills on the company stores (assuming that some kind of authentication is done when receiving payments). The information can be obtained by doing a query like this:

SELECT [0], [1], [2], [3], [4], [5], [6], [7], [8], [9], [10],
  [11], [12], [13], [14], [15], [16], [17], [18], [19],
  [20], [21], [22], [23]
FROM (SELECT AuthenticationId, DATEPART(HOUR, Date) AS Hour FROM Authentication) AS TBL
PIVOT (
 COUNT(AuthenticationId)
 FOR Hour IN
 ([0], [1], [2], [3], [4], [5], [6], [7], [8], [9], [10],
  [11], [12], [13], [14], [15], [16], [17], [18], [19],
  [20], [21], [22], [23])
) AS PVT

The result will show the authentication count for each hour. Now you can copy it and paste on Excel. Maybe generate some charts (assuming your use case has more data and more rows).

Tracking UPDATE changes on SQL Server 2005+ with the OUTPUT clause

It is common to have to execute a mass update on some table, for example, to update a customer credit classification. When such updates are executed, you may want to know what has changed and maybe log it. There are several ways to do it, but i recently found one that makes it easier: the OUTPUT clause.

First, we create a customer table for the example and populate it with some data:

IF OBJECT_ID('Customers') IS NOT NULL
 DROP TABLE Customers
GO

CREATE TABLE Customers (
 CustomerId INT,
 [Name] VARCHAR(30),
 Income MONEY,
 Classification CHAR(1)
)

INSERT INTO Customers (CustomerId, [Name], Income, Classification) VALUES (1, 'JOHN DOE', 5000, 'C')
INSERT INTO Customers (CustomerId, [Name], Income, Classification) VALUES (1, 'JANE DOE', 6000, 'B')
INSERT INTO Customers (CustomerId, [Name], Income, Classification) VALUES (1, 'JOHN SMITH', 6500, 'B')
INSERT INTO Customers (CustomerId, [Name], Income, Classification) VALUES (1, 'HOMELESS JOHN', 100, 'Z')

Now, let's update the data (imagine that there are thousands of rows):

UPDATE Customers
SET Classification = 'C'
OUTPUT DELETED.CustomerId,
    DELETED.[Name],
    DELETED.Classification AS FromClassification,
    INSERTED.Classification AS ToClassification
WHERE Income BETWEEN 5000 AND 7000

The OUTPUT part will cause the following output:

CustomerId  Name                           FromClassification ToClassification
----------- ------------------------------ ------------------ ----------------
1           JOHN DOE                       C                  C
1           JANE DOE                       B                  C
1           JOHN SMITH                     B                  C

If you want to persist the values:

IF OBJECT_ID('CreditChangeLog') IS NOT NULL
 DROP TABLE CreditChangeLog
GO

CREATE TABLE CreditChangeLog (
 CustomerId INT,
 FromClassification CHAR(1),
 ToClassification CHAR(1)
)

UPDATE Customers
SET Classification = 'C'
OUTPUT DELETED.CustomerId,
    DELETED.Classification AS FromClassification,
    INSERTED.Classification AS ToClassification
    INTO CreditChangeLog
WHERE Income BETWEEN 5000 AND 7000

SELECT * FROM CreditChangeLog

This will save the changed values on the CreditChangeLog table.