CONDG Presentation

I'm very grateful to all those who attended the presentation Scott and I gave last night. I will be doing a screencast next week of the demo that I was unable to get to due to technical difficulties. I guess I need to clear up something that I said last night because I was asked about this more than anything else:

Johnny Cash was an immensely influential American singer and song writer and is probably the most recognized country musician of all time. His 'calling card' and what most people associate with him was the dark clothing he wore, usually all black, hence the crack on myself before I gave my presentation

.NET Garbage Collection PopQuiz Quick Answers

all credit to Tess for this.

the original quiz can be found here

Tess's answers and further explanation can be found here

1. How many GC threads do we have in a .NET process running the Server version of the GC on a dual-core machine?

A: server version = one thread per logical processor = two threads (four if hyper threaded)

2. What GC mode is used in the web development server (cassini) on a quad proc machine? Why? (you can choose from server, workstation or concurrent-workstation)

A: cassini = windows forms app = user interaction = use concurrent-workstation

3. How many finalizer threads do we have in a .NET process running the Server version of the GC on a quad proc machine?

A: Tess states that the answer is one per process. further investigation only lead to more confusion - server version of GC on a quad proc machine means four GC threads (eight if hyper threaded). i believe the answer is either one or four/eight. i'll keep reading and update this when i find an explanation that makes sense to me

UPDATE: the gentleman walker looked into this and found that, at a given time, there is one per .NET process with a maximum equal to the sum of the number of worker process allocated to each application pool. the process will only show up in taskman if the associated app pool/web site is in use

4. When is an object garbage collected?

A: when the object is no longer referenced and a GC occurs for the generation your object is in

5. What causes an object to move from Generation 0 to Generation 1 or to Generation 2?

A: the object is still referenced when a GC occurs for the generation the object is in

6. If you look at the GC sizes for Generation 0, 1 and 2 in perfmon, why is most of the memory in the process in Gen 2?

A: gen 0 and gen 1 have small sizes (though these sizes are not fixed); gen 2 does not. since GC occurs when gen 0 and gen 1 are nearly 'full', referenced objects will eventually end up in gen 2 where they will remain for the remainder of their life (i.e. until they are no longer referenced and eventually cleaned up)

7. How many heaps will you have at startup on a 4 proc machine running the server GC? How many would you have if the same machine was running the workstation GC? Will the memory used for these show up in private bytes or virtual bytes in perfmon or both?

A: one small object heap and one large object heap per logical processor for server GC; one small object heap and one large object heap in workstation GC. as far as the memory part of the question, this is how i understand it - virtual bytes corresponds to memory reserved by a process, whereas private bytes is the memory actually being used by the process. if all goes smoothly, the graphs for these will follow each other which means the process is allocating memory correctly. at startup, virtual bytes may be a lot larger than private because the reserved memory hasn't been used it, but private bytes should eventually catch up to an extent and both should smooth out in perfmon

8. (Leading question:)) Is the fact that you have mscorwks.dll loaded in the process in 2.0 an indication of that you are running the workstation version of the GC?

A: no; in 2.0, both the server and workstation versions of the GC reside in mscorwks.dll

9. Can you manually switch GC modes for a process? If so, how and under what circumstances?

A: the answer is "yes, but...". i'll leave Tess to explain the restrictions:

a) you can not run the server version on a single proc box, it will default to workstation

b) you can not run concurrent while also running server

c) if the runtime is hosted, the hosts GC mode will override the configuration

10. Name at least 2 ways to make objects survive GC collections unnecessarily.

A: create an unnecessary finalize method. create a objects within a method and then have the method perform some action for a long time, preferably one that causes memory pressure which will trigger a GC and the referenced objects will not be collected

11. Can a .NET application have a *real* memory leak? In the C++ sense where we allocate a chunk of memory and throw away the handle/pointer to it?

A: not in the C++ sense, but leaking memory is quite possible. see Tess lab 6 (which was also covered in my presentation)

12. Why is it important to close database connections and dispose of objects? Doesn't the GC take care of that for me?

A: disposing the objects releases their resources and makes them available for GC immediately (note: calling dispose on an object does not trigger a GC)

iTunes and QTFairUse

if you're like me and simply prefer the .m4a file extension to the .m4p file extension that hangs from the end of files newly purchased from iTunes, you probably use either QTFairUse (QT) or myFairTunes. to continue to purchase music from the iTunes store, i was forced to upgrade to at least version 7.4. i also noticed earlier that QT was no longer functioning correctly - it would just spit out the name of all the songs on my list preceded by the test 'could not play track'. i figured an upgrade/reinstall wouldn't hurt and may actually solve both problems. the download page for the iTunes installer only allows you to download version 7.6; however, the latest version that QT 2.5 supports (despite the readme file saying that the max is 7.0.2) is 7.5. luckily, i always keep a local copy of the iTunes installer for the latest version supported by QT.

<tangent>
am i the only one to notice a strange correlation between the timing of QT supporting the most current version of iTunes and iTunes prompting me to upgrade to a newer version that contains just fuckloads of new features. ever wonder why iTunes seems to be the only product that doesn't come with a 'what's new' document, or why you can't find any reference to this on their website? that's because, as far as i can tell, nothing new has been added since 6.0 - maybe even somewhere in the 5.x realm. why can't the assholes just admit that the only new feature is a new encryption key to keep the crackers (the computing type, not white people in general) off their tails for a few months?
</tangent>

that was the background, here is the problem: i upgraded to 7.5 and was able to download my music, but QT kept saying it couldn't find my music library. i navigated to my music folder and found an assload of .itl and .xml files with ' - copy (n)' and even some ' - copy (n) copy' appended to the filename, where n went as high as five or six. i assumed these were created by QT since there is an option to backup the current library and this option is selected by default. what i couldn't find was the non-copied version of my library, which is what QT was looking for. i moved all of these .itl and .xml files to a backup folder thinking that iTunes would prompt me for the location of my library and i could just rename my latest copy to the default name and put it back in the original location and everyone would be happy. this didn't happen. iTunes instead created a new .itl and .xml file with ' - copy (x)' (can't remember which it was) in the default location and the application came up with nothing in my music library.

solution: close iTunes, then hold down shift and launch it again. it will prompt you to use an existing library or create a new one. at this point, you can copy one of the backups to the default location, give it the default library name, and you will have all your music back in your library and the ability to strip your songs of evil, evil copyright protection once again

p.s if anyone needs either the installer for iTunes or QT, feel free to contact me.

classical mechanics problem: box on an incline

for some reason i'm in a physics mood so i thought i'd share a classic problem that anyone that took elementary mechanics would come across at some point and how to solve it .

Q: a box of mass m sits motionless on an incline. what is the maximum angle Θ of the incline before the box begins to slide down?

A: all good little physics students know the first step is to draw a diagram and then add forces:


where Θ is the angle of the incline, μ is the coefficient of friction and N is the normal force

now lets disect the horizontal and vertical forces on the box:

ΣFx = horizontal component of gravity - friction = mgsin(Θ) - μN = 0 (box is stationary)

-> mgsin(Θ) = μN

ΣFy = vertical component of gravity - normal force = mgcos(Θ) - N = 0 (box is stationary)

-> mgcos(Θ) = N

we have two equations and three unknowns (Θ, μ, and N). the second equation gives us a N in terms of Θ, so let's plug it into the first equation and then solve for Θ:

mgsin(Θ) = μ(mgcos(Θ))

cancel out mg on both sides and get μ by itself:

μ = sin(Θ)/cos(Θ) = tan(Θ)

bring tan to the other side and we are done:

Θ = arctan(μ)

the solution tells us that the largest angle of incline before the box moves is dependent on the coefficient of friction of the incline.

mercy i loves me some classic mechanics :)

bubu made the rock band hall of fame

on a personal note, i would like to congratulate bubu on their induction into the rock band hall of fame. most of the credit goes to my singer, YEAHYEAH (who also happens to be the computing services/IT/network guy where i work) and, or course, me, who is most proficient with the axe (both lead and bass) but is also more than competent with the sticks (drums) and has had to take that role when friends with no rhythm whatsoever come over and want to play. a special thanks goes to maneet and good sir thomas playing bass and lead guitar respectively for assisting YEAHYEAH and i on that fateful night. i'd also like to thank diiiiiiiiiidddtttyy, good sir thomas, lonlon and YEAHYEAH for helping the band regain several hundred thousand fans that we lost when i had too much to drink and repeatedly played and failed the song 'gimme shelter' by the stones (alcohol gives me sausage fingers). i'd also like to thank alex (who is a phenom with the sticks), tha dream (who really isn't that good with the sticks), superman, and winnie (who recently became tha dream's wife), for their help along the way.

come to think of it, all the above mentioned people with the exception of good sir and winnie work with YEAHYEAH and i. i think this is one of the reasons that i love where i work - i like to hang out with my coworkers, be it going out boating, going out to a bar, hanging out late at work late and discussing business over a beer or just having everyone come over for some rock band jam'ins. at least half of tha dream's wedding last week was composed of coworkers. i genuinely consider these people friends and am eternally grateful that i work for a company that encourages this kind of environment.

the effects of changing a machine's name on sql server

i decided to post about this because, i shit you not, every single member of our solution delivery department (aka our consultants) and a good number of our developers have approached me with issues after renaming their machine. one may be wondering why so many people are renaming their computer in the first place. simple - in my neck of the woods, we have several baseline virtual machine images built up for anyone to use. when someone begins using it, they typically give the 'computer' a new name (especially if they want to join it to the domain). the problem is that sql server doesn't pick up on this and programs that try to connect to the server by referencing it by the new name (or new name\instance name) will get errors along the lines of 'the server could not be found in the sysservers list. try adding the server using sp_addlinkedserver'. don't do that; it won't work. try running this command against the master database instead:

SELECT @@servername

it will probably return the name of the machine (\instance name) before you renamed it. the fix is this:

sp_dropserver '{oldname}'

where {oldname} is whatever was returned by your first query. after that, run this:

sp_addserver '{newname}(\instancename)', 'LOCAL'

obviously only include (\instancename) if it is a named instance of sql server. restart the sql server service, then re-run your original query and it should return the correct name (you will have to either open a new query editor or re-connect the query editor screen you originally used as restarting the service will cause a disconnect).

damned system named constraints

problem: you need to write a script to alter or drop a column as part of a hotfix for your clients. this column was created with a default constraint, but this default constraint was not explicitly named, thus a system-generated name was given (which looks something like 'DF__(partialtablename)__(partialcolumnname)__(random numbers and letters)' in SQL Server 2005). since the name of the constraint will be different on every client, you can't write a straight forward 'ALTER TABLE DROP CONSTRAINT ' statement.

solution: system tables aaaaaaaand ...

dynamic sql!!! weeeeeeeeeeeeeeeee

(i don't know why i got excited about something i genearlly preach against, but, as i've said before, dynamic sql is a necessary evil and can be very useful)

/*1. i think it is good practice to validate that something exists before dropping it (or does not exist before adding it). that way if a sql query that is part of a hotfix is accidentally run more than once, no errors should occur. this script simply checks to see if there is a constraint on a column named 'col_a' in table_a:*/

IF EXISTS (SELECT OBJECT_NAME(constid) FROM sysconstraints CN
INNER JOIN syscolumns CO
ON CN.id = OBJECT_ID('table_a') AND CO.id = CN.id AND
CO.name = 'col_a' AND CN.colid = CO.colorder)
BEGIN

/*2. the following sample then removes the constraint from the column:*/

DECLARE @Sql AS NVARCHAR(2000)

SELECT @Sql = 'ALTER TABLE table_a DROP CONSTRAINT ' + OBJECT_NAME(constid)
FROM sysconstraints CN
INNER JOIN syscolumns CO
ON CN.id = OBJECT_ID('table_a') AND CO.id = CN.id AND
CO.name = 'col_a' AND CN.coldid = CO.colorder

EXEC(@Sql)

END

/* to be extra safe, you may want to include the type of constraint you are dropping as part of the join. the type of constraint is contained in the pseudo-bit-mask value of the sysconstraints 'status' column. as an example, a default constraint will have a status value of '133141' */

preserving the stack trace when handling exceptions

i just now stumbled across something that i didn't realize (though it makes sense) about the difference between throw and throw ex within a try...catch...finally block and i thought i'd share the wealth. i have the feeling this may be one of the "holes" in my coding knowledge/fundamentals that everyone else knows except for me since i was a physics major and only took 16 hours worth of computer science back in college. if so, you can politely skip this post and be happy for me that i learned something valuable. if you think i'm dumb for not knowing this, just know that i think you're dumb for not knowing that an operator that commutes with the Hamiltonian does not evolve with time. yeah, what now??

here's the gist:
case i) if a method call within a try...catch block results in an exception, calling throw(with or without an exception instance) causes any code in the ...finally block to immediately execute and then execution continues on its jolly way up the call stack. if you want to log anything about the error, do so before calling throw. furthermore, calling throw by itself preserves the call stack, whereas calling throw with an exception instance causes the stack trace to restart from the method that re-throws the exception and you lose whatever precious debugging information you had about what was going on before the call.

case ii) if procedural code within a try...catch block results in an exception, you lose the stack trace whether you throw with an exception instance or without. you should log what you need to know about the error before you throw as suggested in case i) and additionally log the stack trace.

if you are truly masochistic/sadistic (not sure which but i'd lean towards the former), you can follow what some bloke by the name of Chris Taylor outlines here

selective filtering, part iii - dynamic sql filtering

i just thought of this as i posted my last entry and wanted to get it 'on paper' before it slips my mind.

if you are using dynamic sql to filter a query based on an optional/nullable parameter, you have to provide something to your WHERE clause in the case that you receive all nulls. here is an example that will not work using the clever ISNULL technique i discussed in my first post:

DECLARE @sSQL nvarchar(4000)

SET @sSQL = 'SELECT col_a FROM table_a '
SET @sSQL = @sSQL + 'WHERE col_b = ''' + ISNULL(@param1, 'col_b') + ''''

exec sp_executesql @sSQL
GO

what's the problem here? if @param1 is null, the WHERE clause is comparing the value of col_b to a constant, 'col_b', instead of setting it equal to itself. here is a correct way to accomplish the filter:

DECLARE @sSQL nvarchar(4000)

SET @sSQL = 'SELECT col_a FROM table_a '
IF @param1 IS NOT NULL
SET @sSQL = @sSQL + ' WHERE col_b = ''' + @param1 + ''''

exec sp_executesql @sSQL
GO

in this case, you only filter if the @param1 contains a value. however, this once again presents a problem - what if there are multiple optional/nullable parameters provided? without knowing which ones contain values, you don't know where to put the WHERE. for example:

DECLARE @sSQL nvarchar(4000)

SET @sSQL = 'SELECT col_a FROM table_a '
IF @param1 IS NOT NULL
SET @sSQL = @sSQL + ' WHERE col_b = ''' + @param1 + ''''
IF @param2 IS NOT NULL
SET @sSQL = @sSQL + ' WHERE col_c = ''' + @param2+ ''''

exec sp_executesql @sSQL
GO

this obviously breaks if both @param1 and @param2 are not null since you can't specify WHERE more than once. there are two options: write your query such that every combination of parameters containing and not containing values is accounted for (which is exponentially more difficult with each additional parameter), or you can be clever. let's rewrite the above sample:

DECLARE @sSQL nvarchar(4000)

SET @sSQL = 'SELECT col_a FROM table_a '
SET @sSQL = @sSQL + 'WHERE 1=1 '
IF @param1 IS NOT NULL
SET @sSQL = @sSQL + ' AND col_b = ''' + @param1 + ''''
IF @param2 IS NOT NULL
SET @sSQL = @sSQL + ' AND col_c = ''' + @param2+ ''''

exec sp_executesql @sSQL
GO

now that you've specified a condition that's always true (WHERE 1=1), you can dynamically add additional filters as needed. pretty damn slick if i do say so myself

one last thing - i always declare @sSQL as nvarchar(4000) in my examples because sp_executesql only works with variables with data types of nchar, ntext and nvarchar and the maximum size of an nvarchar variable used with sp_executesql is 4000

selective filtering, part ii - dynamic sql sorting 1

my previous post on selective filtering dealt with filtering a select statement if a non-null parameter value is passed to your stored procedure without using dynamic sql. however, there will be times when dynamic sql is required. since the 'sp_executesql' stored procedure was included in SQL 2005 and was not removed in SQL 2008 (to my best knowledge), i dare anyone to dispute that (but if you do so successfully i will be simultaneously amazed, humbled and eternally grateful).

for the uninitiated, dynamic sql is the process of building a query on-the-fly based on parameters passed to the stored procedure. an example of when this is required is if your query accepts the name of a column as a parameter (@param1) and you want to sort your query based on this column/parameter. you can't order a query by a constant, null, or a parameter, even if the parameter evaluates to a valid column name. instead, you write something like this:

DECLARE @sSQL nvarchar(4000)

SET @sSQL = 'SELECT col_a FROM table_a '
IF @param1 IS NOT NULL
SET @sSQL = @sSQL + ' ORDER BY ' + @param1

exec sp_executesql @sSQL
GO

assuming @param1 is a valid column name in table_a, the statement will run and return the values of col_a ordered by the @param1 column (in ascending order by default). notice that i checked to make sure @param1 contained some value since you can't order by null and, if i hadn't checked and @param1 was null, @sSQL would be null (since anyting + null = null). sp_executesql won't throw an error if @sSQL is null; it will just say "Command(s) completed successfully." and return nothing.

that does it for the basics of dynamic sql sorting. the reason i named this post 'dynamic sql sorting 1' is because i can think of at least one more complicated case (e.g. using a case statement to select a column to sort by when another column's value is equal to an input parameter) that i don't want to go into right now since this post is already pretty long. good luck and by all means let me know if you have any comments, corrections or questions. später

thread was being aborted

can it be that this demi-god of t-sql can also code and happens to know a thing or two about debugging? holy shit, how has some beautiful lady not yet scooped him up and made him hers by constantly giving him hot, hot lovin? i ask myself these questions often. i also ask myself who the hell thinks that choking a dude is a good way of saying 'nicely done', but that's a story for another time...

if you are using response.redirect(url) or server.transfer in your web application and receiving a "Thread was being aborted" error despite the fact that your application does absoluetely nothing explicitly with threading, you may be as confused as i was when this happened to me. i'll be the first to admit that, despite all those lovely Microsoft certification letters after my name, i'm far from an expert on page life cycles and the intricacies of what goes on behind the scenes when the application does something as seemingly simple as changing the page that is displayed to a user. here is what i found:

response.redirect and server.transfer both internally call response.end. response.end ends the current page execution, the Application_EndRequest event is fired and a ThreadAbortException is thrown (i'm unsure whether or not resonse.end throws the ThreadAbortException to immediately shift the page life cycle to Application_EndRequest of if the exception is thrown in the Application_EndRequest event). that code which follows response.end (and, therefore, response.redirect and server.transfer) is not executed.

you: "why not just wrap the page redirection in a try...catch block and catch the exception?"

a ThreadAbortException will be rethrown at the end of your catch block whether you like it or not. it will then execute everything in your ...finally block and then kill your current thread. if you feel like dealing with threads, you can call thread.resetabort in your catch block to continue executing your thread, but this discussion assumes you are not dealing with threading, so scratch that.

solution: it turns out this is actually a pretty easy fix. if you are using response.redirect(url), use the overloaded version response.redirect(url, false) instead. if you are using server.transfer, use server.execute.

works cited:
Microsft Help and Support
DotNetJunkies - In the line of Fire -- Shaunak Pandit
Microsoft Visual Studio 2005 Documentation - ThreadAbortClass

deadlocks are bad

what is a deadlock? the way i typically explain it is like so (this is simplified but i think it gets the idea across):

• process A has a lock on table A and is waiting to update it with information from table B
• process B has a lock on table B and is waiting to update it with information from table A

since neither process will release its lock until the other processes releases its lock, we have ourselves a deadlock. most of the time SQL server "resolves" this issue itself because it has a thread dedicated to its lock manager and someone much smarter than myself came up with an algorithm that allows the lock manager to detect this and kill one of the processes. i have no clue how it chooses which process is the victim of its kill statement, but, from my experience, 99% of the time it is does make this decision and it does kill one of the processes. i've not yet had this happen in SQL 2005, but i can count on one hand the number of times when a deadlock occurred in SQL 2000 and SQL left the decision up to me. in all of these cases, the decision for me was simple - i opened up the SQL activity monitor, scrolled to the right, and saw that a whole bunch of locked processes listed the same process in the 'Blocked By' column. by no means should you be cavalier and just kill this single process before you know what it is, but it's a good place to start. i figured out what the process was and had my client log on to the server hosting the application and simply close it.

back to the other 99% of the time SQL Server resolved the deadlock automagically. this is still very bad. some application was trying to perform some operation on a table or its data and SQL Server flat out squashed it before it could finish. there are certainly ways design a database to minimize the chance of this, but no matter how badass you think your 5NF database is (because there is no way you are badass enough to reach 6NF), you are going to experience a deadlock at some point. i recommend you read the following article to better understand what is going on, and how to track down resolve the problem:

http://support.microsoft.com/kb/832524

selective filtering, part i

what if you only want to filter a select statement if a non-null value is passed to your query as a parameter? you have a couple of options. one is to use dynamic sql to build your WHERE clause and only include the filter if the parameter value is not null. however, as i will attempt to establish as law at one point or another, dynamic sql is bad and should be avoided at all costs. from my experience, the easiest and best way to accomplish this without resorting to dynamic sql is to make use of the ISNULL function like so:

WHERE ...
AND a.column_a = ISNULL(@param1, a.column_a)

a.column_a = a.column_a is always true if a.column_a does not contain nulls, thus if a null is passed to @param1, the statement will not be filtered. if the column can contain nulls, we have to be a be more creative depending on what you want to accomplish. what about using the SET ANSI_NULLS OFF option? this allows for the logical comparison of nulls (i.e. null = null evaluatues to true). doing so makes the example above return all rows when @param1 is null. if you do not want rows returned where a.column_a is null, simply add an additional line:

WHERE ...
AND a.column_a = ISNULL(@param1, a.column_a)
AND a.column_a IS NOT NULL

problem solved. one last thing i should note is my example is likely a much simpler example than what you have since i am only concerned with one column; you must take caution to make sure your other comparisons will not return undesired results when using SET ANSI_NULLS OFF.