sides of march » SQL

Executing native SQL using NHibernate named queries

brian — Wed, 11 Feb 2009 14:16:33 +0000

I’ve been doing a lot of work with NHibernate lately, particularly with named queries. It took a while to get it just right, so I figured it would be helpful to others (and to myself in the future) to note some of the gotchas and how-to steps to get it just right.

What is a named query?

An NHibernate named query is essentially a native SQL statement that can be invoked by NHibernate and can return strongly-typed object data. It allows you to leverage native SQL code — parametrized statements or stored procedures — to perform complex data manipulation and retrieval.

In other words, let’s say you’re writing a baseball game, and you have three objects in your domain model: a Player, a Team, and a DrugTestResult model. You write a stored procedure, spSelectPlayersByLastDrugTestDate, which returns all players who haven’t had a drug test since a given date.

Now that you have a domain model and a SQL statement, how do you execute it through NHibernate?

Mapping your named queries

NHibernate needs to know about your named queries in order to execute them. Much like you tell NHibernate about your domain model using XML mapping files (*.hbm.xml), you tell NHibernate about your named queries using those same files.

Since the named query mappings do not relate directly to your domain model mappings, you can create a separate mappings file just for your named queries. A sample mappings file for our spSelectPlayersByLastDrugTestDate could look like the following.


< ?xml version="1.0" encoding="utf-8"?>









exec spSelectPlayersByLastDrugTestDate @LastDrugTest=:LastDrugTestDate

The named query part of the mapping is in the section. I’ll describe its most important sections below.

The sql-query tag defines the named query. The name attribute is the name that NHibernate (and, in turn, your code) will use to reference the query. In our example, we are stating that this named query is called FindPlayerByLastDrugTest.
The query-param tag defines a parameter in your query. The name attribute reflects the name by which you are referring to that parameter, not the parameter name in the actual SQL statement. The type is the data type of that parameter. In our example, we have one parameter, LastDrugTestDate, which is a DateTime.
The return tag defines the type which is returned by the named query. In our example, the return type is a class, Player. Our SQL statement must therefore return fields which map to the Player object. Note that named queries do not have to return domain model objects; they can return ordinals, other types, or nothing at all. (As a result, the return tag is optional.)
The many return-property tags tells NHibernate how to map a column in the query results (the column attribute) to a property name in the return type. NHibernate gets rather crafty here. For normal properties (PlayerID and Name in our example) the mappings are simple column-name-to-property-name. Note, however, the TeamID mapping, where we map a TeamID in our result set (which could be an integer, the primary key for a Team domain object) to a property named Team. In our domain model, the Player.Team property is not an integer, it is a Team. NHibernate, because it knows how a Player relates to a Team, will auto-map the player to the team. This nuance is important in getting your named queries to work with your domain model: you map foreign key values in your result set to objects in your data model. (A similar thing happens for the LastDrugTestResult property.)
Finally, after we close our return tag, we provide our actual SQL statement. This is a standard parametrized SQL statement with a twist. To inject your query-param values into the SQL statement, you specify :Name — that is, a colon followed by the name specified for the query-param.

Got all that? Good. We’re almost done!

Running your named queries

Finally, the easy part — running the named query! Named queries are run against the NHibernate ISession. Various methods are available to set parameters using a fluent interface.

Running our FindPlayerByLastDrugTest query could look like this:


// get your NHibernate session however you normally would
//ISession session = ...

// create an IQuery based on your named query, specifying parameters
IQuery namedQuery = session.GetNamedQuery("FindPlayerByLastDrugTest")
.SetDateTime(new DateTime(2003, 1, 1));

// execute the query
IList list = query.List();

// if you were running a query that returned a single, scalar value, you could say...
var result = query.UniqueResult();

Note that there are generic versions of List() and UniqueResult(), but I had problems getting them to work. Not sure why (I didn’t dig that far), but the above queries will do the same (though they require some type-casting).

Named queries do a lot more. The following links may prove useful in your learning.

NHibernate’s “Stored Procedures” – Named Queries (Ayende)
Using nhibernate’s named queries with ActiveRecord (Ken Egozi)
NHibernate Bulk Updates/Inserts/Deletes with Named Queries (Nathan Scott)

If you find other great resources, be sure to share!

Scripting the deletion of objects in a SQL database (second version)

brian — Thu, 11 Sep 2008 16:46:39 +0000

A few weeks ago, I wrote a post about deleting all tables and constraints in a database. It’s time to take that one step further!

The following SQL code will delete all stored procedures, user-defined functions (UDFs), views, table constraints, and tables (in that order) from a given SQL database. As with the original, you can specify an object name previx by changing the set @tblname = '' as needed.

This is helpful in two situations:

When you want to delete all objects in a database without dropping/recreating it, in which case you’d specify '%' as the @tblname.
When you want to delete all objects in a database whose name starts with a certain sequence of characters. For example, if you want to delete all the ASP.Net membership objects in a database, you can specify '%aspnet_%' as the @tblname, since its SQL objects start with aspnet_ or vw_aspnet_.

Note that you should be very careful any time you use scripts like this, as data loss can happen where you least expect it. I use this script for clearing out development databases, but would never use it for production databases.

Anyway, on to the script.


declare @sql varchar(max)
declare @tbl varchar(255)
declare @tblname varchar(255)

--dpecify the "like" clause for your tables here
--to match all tables, use '%'
set @tblname = '%'

--drop stored procedures
select name
into #procs
from sys.objects
where type = 'P'
and name like @tblname

while (select count(*) from #procs) > 0
begin
    select top 1 @tbl = name from #procs
    set @sql = 'drop procedure ' + @tbl
    exec (@sql)
    delete from #procs where name = @tbl
end

drop table #procs

--drop UDFs
select name
into #funcs
from sys.objects
where type in( 'TF', 'FN' )
and name like @tblname

while (select count(*) from #funcs) > 0
begin
    select top 1 @tbl = name from #funcs
    set @sql = 'drop function ' + @tbl
    exec (@sql)
    delete from #funcs where name = @tbl
end

drop table #funcs

--drop views
select name
into #views
from sys.objects
where type = 'V'
and name like @tblname

while (select count(*) from #views) > 0
begin
    select top 1 @tbl = name from #views
    set @sql = 'drop view ' + @tbl
    exec (@sql)
    delete from #views where name = @tbl
end

drop table #views

--drop constraints
select 'alter table ' + table_schema + '.' + table_name + ' drop constraint ' + constraint_name as query
into #constraints
from information_schema.table_constraints
where constraint_type = 'foreign key'
and table_name like @tblname

while (select count(*) from #constraints) > 0
begin
    select top 1 @sql = query from #constraints
    exec (@sql)
    delete from #constraints where query = @sql
end

drop table #constraints

--drop tables
select name
into #tables
from sys.objects
where type = 'U'
and name like @tblname

while (select count(*) from #tables) > 0
begin
    select top 1 @tbl = name from #tables
    set @sql = 'drop table ' + @tbl
    exec (@sql)
    delete from #tables where name = @tbl
end

drop table #tables

go

Remove anonymous users from ASP.Net Membership tables

brian — Thu, 07 Aug 2008 18:53:47 +0000

If you use the ASP.Net membership tools, have specified in your Web.config, and get lots of anonymous visitors, it’s only a matter of time before your database grows. What’s filling it up is the countless user records for your anonymous users.

If you don’t need to track user and profile information for an anonymous user once they leave the site, you can delete the unneeded data by running a SQL script. The following script will delete from your membership tables all anonymous users whose last activity was more than 7 days ago.


delete from aspnet_profile
where userid in ( select userid from aspnet_users
where isanonymous = 1
and datediff(dd, lastactivitydate, getdate()) > 7
)

delete from aspnet_usersinroles
where userid in ( select userid from aspnet_users
where isanonymous = 1
and datediff(dd, lastactivitydate, getdate()) > 7
)

delete from aspnet_membership
where userid in ( select userid from aspnet_users
where isanonymous = 1
and datediff(dd, lastactivitydate, getdate()) > 7
)

delete from aspnet_personalizationperuser
where userid in ( select userid from aspnet_users
where isanonymous = 1
and datediff(dd, lastactivitydate, getdate()) > 7
)

delete from aspnet_users
where userid in ( select userid from aspnet_users
where isanonymous = 1
and datediff(dd, lastactivitydate, getdate()) > 7
)

Doing this may be very important to those using a shared hosting plan that has limited SQL disk space, as those anonymous users can quickly eat up disk space. Since each row in aspnet_users takes up just over 1kb, having 1,000 anonymous users will eat up 1MB — something that adds up quick if you only have 250MB of SQL disk storage.

Deleting all tables and constraints in a database

brian — Wed, 06 Aug 2008 15:40:32 +0000

We all need to do it from time to time — erase a bunch of tables in a SQL database. Sure, you can do this through Enterprise Manager, one table at a time, or write a bunch of DROP TABLE statements, but when doing it this way, you also need to ensure all foreign key constraints (i.e. table dependencies) are dropped first.

There’s an easier way — the following script, which was derived from a few posts scattered around the Internet. (Sorry, it’s been a while, and I don’t remember what they were!)


declare @sql varchar(max)
declare @tbl varchar(255)
declare @tblname varchar(255)

--Specify the "like" clause for your tables here
--to match all tables, use '%'
set @tblname = ''

select 'alter table ' + table_schema + '.' + table_name + ' drop constraint ' + constraint_name as query
into #constraints
from information_schema.table_constraints
where constraint_type = 'foreign key'
and table_name like @tblname

while (select count(*) from #constraints) > 0
begin
select top 1 @sql = query from #constraints
exec (@sql)
delete from #constraints where query = @sql
end

drop table #constraints

select name
into #tables
from sys.objects
where type = 'U'
and name like @tblname

while (select count(*) from #tables) > 0
begin
select top 1 @tbl = name from #tables
set @sql = 'drop table ' + @tbl
exec (@sql)
delete from #tables where name = @tbl
end

drop table #tables
go

Simply change the set @tblname = '' as needed — it’s used in a LIKE statement to allow you to do these operations on a subset of tables. To run this script against all tables, use set @tblname = '%'.

Bad Programming: How not to use try/catch blocks

brian — Thu, 12 Jun 2008 16:27:31 +0000

I just stumbled across the following SQL 2005 code:

	BEGIN TRY
		UPDATE [tabProgramSite]
		SET
			[Name] = @ProgramSiteName
		WHERE [ProgramSiteID] = @ProgramSiteId
	END TRY
	BEGIN CATCH
	END CATCH

Nothing like catching an error… and doing nothing about it. I hope the user experience isn’t as robust.

Separating SQL script files generated by Microsoft SQL (by type)

brian — Tue, 03 Jun 2008 19:37:58 +0000

There’s one great feature in SQL: the “Generate Scripts” command. Unfortunately, it has one limitation: the default filenames of scripts look something like this:

dbo.fnc_PlayerValue.UserDefinedFunction.sql
dbo.UserSelect.StoredProcedure.sql

I’d much prefer the filenames to match the object name, without the owner (‘dbo’) or object type. In other words, I’d prefer the above two files to look like this:

UserDefinedFunction\fnc_PlayerValue.sql
StoredProcedure\UserSelect.sql

How do we get from point A to point B without a lot of manual file copies and renames? We use the FOR command!

First, create a subdirectory for each object type (Table, StoredProcedure, UserDefinedFunction, View, Schema, Trigger, and User), then run the following from a command prompt.

for %i in (*.User.sql) do for /f "delims=., tokens=1-3" %j in ("%i") do move %i %k\%j.%l
for %i in (*.Schema.sql) do for /f "delims=., tokens=1-3" %j in ("%i") do move %i %k\%j.%l
for %i in (*.Trigger.sql) do for /f "delims=., tokens=1-3" %j in ("%i") do move %i %k\%j.%l
for %i in (*.sql) do for /f "delims=., tokens=1-4" %j in ("%i") do move %i %l\%k.%m

The command above will look for each file with a *.sql extension in the current directory. For each of those files, it copies it to a directory based on the type of file, and ensures the new file only includes the object name and the .sql extension. So much cleaner now!

Note that this will not work if any folder in the full path to the SQL files has a period in it!

Updated 2009.11.10 to support Trigger, User, and Schema scripts.

How to use REPLACE() within NTEXT columns in SQL Server

brian — Wed, 27 Feb 2008 21:10:30 +0000

SQL has an incredibly useful function, REPLACE(), which replaces all occurrences of a specified string with another string, returning a new string. It works great with all forms of NCHAR and NVARCHAR fields. It does not, however, work with NTEXT fields.

Fear not — there’s an easy workaround, thanks to type-casting and SQL 2005′s NVARCHAR(max) datatype. Here’s the process in an nutshell.

Cast the NTEXT field to the NVARCHAR(max) datatype using the CAST function.
Perform your REPLACE on the output of #1.
Cast the output of #2 back to NTEXT. (Not really required, but it does get us back to where we started.

A simple SQL query illustrates this.


select cast(replace(cast(myntext as nvarchar(max)),'find','replace') as ntext)
from myntexttable

If you’re using SQL 2000, you’re out of luck, as NVARCHAR(max) first appeared in SQL 2005. However, if your NTEXT field is less than 8000 characters, you can cast it to VARCHAR(8000) — the largest possible VARCHAR size — to accomplish the same.

[Note #1: This solution below will also work with TEXT fields. Simply replace NTEXT with TEXT, and NVARCHAR with VARCHAR.]

[Note #2: NTEXT fields are depreciated in SQL 2005 in favor of NVARCHAR(max), so avoid using NTEXT and you'll avoid this problem altogether in the future.]

Five simple rules for creating delimited text files

brian — Wed, 16 May 2007 15:02:46 +0000

Here’s a few tips for those people who provide raw data in text files (think CSV files and the like).

Surround all text fields in single quotes, even if a comma does not exist in a specific instance. By failing to do this, you lose field delimiter consistency on a row-by-row basis, forcing the contents of the field to be parsed separately (i.e. stripping the “optional” quotes).
Use consistent line endings. Pick one and stick with it for all your files. Use either (CR/LF), (LF), or (CR) — and use the same in all your files.
Put column headings in the first row only. This is more a convenience than a necessity. If you make your first row column headings, make sure it is only the first row.
Every row past the first should have an identical schema. Don’t try to be fancy and have different row types in one file. Each file should have the same number and sequence of columns.
Provide delimiters for all columns, even when a row does not have data for them. For example, in a comma-delimited file with five columns, if a row has data in only the first two columns, make sure the row ends with three commas. Failure to do so implies missing or incomplete data.

When text files following these guidelines, I can write a script to import them into a SQL table (using BCP and a simple batch file) in a few minutes. Each guideline that is broken requires additional cleanup steps and more complex data import steps, and adds significant development (and debugging) time that shouldn’t be necessary.

Automatically generate (partial) XML format files for BCP

brian — Tue, 08 May 2007 21:14:08 +0000

I’ve been working with a lot of raw data files lately — data files that come in some delimited format which I must move into a SQL database. Most of the files are CSV files, and some of them have over 100 columns.

Using BCP to import CSV files into SQL is a pleasure. The only annoyance is writing those XML format files. Actually, the annoyance is not writing them, it’s going through the CSV files (which, of course, are undocumented) to determine what delimiters are used in each row.

Here’s a sample of what I mean:

2934,128321,2782,"2007-04-32","Excluded",2321,,22

Fortunately, most CSV files are consistent in their use of quotes, but going through dozens of columns to determine the terminators is a pain. The terminators in the above example aren’t just commas; they could also be quotes. Column three, for example, ends with a comma followed by a quote.

To generate the section of my format files, I wrote the following script, which reads the first line of a text file, finds each comma, determines if it is preceded or followed by a quote, and generates the appropriate XML.


//get filename
var args = WScript.Arguments;
if (args.length == 0)
{
	WScript.Echo('Usage: getdelims.vbs ');
	WScript.Quit();
}

var filename = args(0);

//read file
var fso = new ActiveXObject("Scripting.FileSystemObject");
var file = fso.OpenTextFile(filename,1);
var contents = file.ReadLine();
file.Close();
file = null;
fso = null;

//find commas
var cnt = 0;
for (var i = 1; i < contents.length; i++)
{
	if ( contents.substr(i,1) != ',' ) continue;
	cnt++;
	delim = ",";
	if ( contents.substr(i-1,1) == '"' )
		delim = '",';
	if ( i+1 < contents.length && contents.substr(i+1,1) == '"' )
		delim += '"';
	WScript.Echo('\t');
}

The output can be copy/pasted right into your format file. The example content above would generate the following.

A few minutes writing a script, and I won’t be looking at CSV files with too many columns ever again (at least, not for the reason of writing XML format files).

Scripting SQL database builds

brian — Tue, 03 Apr 2007 13:30:12 +0000

A project I’m working on requires importing data from hundreds of megabytes of delimited text files into a staging database, then converting and migrating the data into an operations database. (Forgive the naming conventions. The “staging” database is used as an intermediary between the text files and the final database. The “operations” database is the one the application uses.)

What’s the best way to do automate this? Ideally, I need to be able to re-run the import at any time, preferably with a single command. The process should purge the staging database, rebuild tables (since schemas may change), reload all data, and run the necessary migration scripts. Since the final product is merely a web interface to the imported data, there is no harm in wiping and recreating everything from scratch, as there is no “user data” of note. (Actually there is, but it is isolated from this process.)

The solution I came up with used a series of SQL scripts executed using sqlcmd (part of MS SQL 2005; use osql for older versions) and data import tasks using bcp (part of MS SQL), all tied together using batch files.

The script to build the staging table (which required use of sqlcmd and bcp) was broken down into four steps:

Copy files from a source directory to a temporary working directory.
Clean up file formats.
Build tables from SQL scripts.
Import data using bcp.

Step 1: The first step set working variables, ensured the working directory existed, and copied files.

setlocal
set xsrc=C:devprojectdata
set dest=C:devprojecttrunkData_import
set scripts=C:devprojecttrunkData

@echo Preparing import directory...
if not exist %dest% md %dest%
del /q %dest%*.dat

@echo Copying files for import...
copy %src%players*.dat %dest% /y
copy %src%games*.dat %dest% /y
...

Step 2: The files received need some tweaking to make importing easier, so I use a gsar utility for search-and-replace. Note the use of the for command, which allows you to repeat a command line based on all matching files in a directory (in this case, all *.dest files in the %dest% directory).

@echo Cleaning up files for import...
for %%f in (%dest%*.dat) do gsar -o %%f -sNULL -r""""""

Steps 3 and 4: Again, I use for loops to recursively create tables and import data.

@echo Creating tables...
for %%f in (%root%tables*.sql) do sqlcmd -U user -P pass -S (local) -d database -i %%f
...

@echo Importing data...
for %%f in (%dest%games*.dat) do bcp db1.dbo.games in %%f -f games.xml -T -F 1 -m 1 -h "TABLOCK"
...

That’s pretty much it. It’s nice to be able to click one batch file and automatically build and load data into databases.