The NHibernate optimization debate: partial object queries

Aaron (Eleutian) and Oren (Ayende) have a little backandforth going about optimizing NHibernate. I’m no expert on NHibernate, but I do have experience with O/R mappers and even more experience with raw databases and SQL, and on techniques for improving performance on both, so I feel I could chime in at least in one area of their debate…

Partial object queries

Let me bring you up to date:

  • Aaron wants to run an NHibernate query of select u(Username, Email) from User u and get a User object with only those two fileds filled in.
  • Oren says you can do it: select new UserSummaryDetails(u.Username, u.Email) from User u.
  • Aaron disputes that as a solution, since it returns a UserSummaryDetails object and not a User object, which would require him to rewrite methods that work on a User object but only use a subset of fields (he mentions a SendMailTo method which uses only the username and email).

There’s a solution that perhaps can perhaps make both happy. You specify an interface (IEmailInfo) that exposes the Username and Email properties, implement that interface in your User and UserSummaryDetails classes, and use the interface on your SendMailTo method? At least, this seems plausible considering my limited (but growing) NHibernate exposure, and would work in other O/R mappers (like Paul Wilson’s).

Either way, keeping up with the aforementioned blog debate is a good way to understand the inner workings of NHibernate.

ComputerSims Baseball: Powered by MonoRail and ActiveRecord

For the past seven years, I’ve been maintaining an online baseball game, CSFBL (Computer Simulated Fantasy Baseball League). It’s gone through many changes over the years, but its core is in many ways unchanged since 2002 — and therein lies a problem.

Back in 2002, CSFBL was getting about 50,000 to 100,000 page views per month. By the end of 2003, we were simulating about 1,500 games per day. Today, we get about 4 million page views per month, simulate up to 10,000 games per day, and have over 80GB of historical data (roughly 6 million games spanning 2,700 seasons). That’s significant growth — and the game is suffering as a result. It wasn’t built to scale this far – a victim of its own success and my own limitations as a programmer circa 2002.

I realized long ago that a rewrite from the ground-up was necessary. I’ve done extensive work rewriting the database schema to maximize efficiency, but that’s only part of it — I need to rewrite the web site and simulation engine as well. (They currently run on classic ASP and Visual Basic, respectively.)

Originally, the rewrite was going to use WebForms — but that changed when I found MonoRail. Up until two weeks ago, data access relied on Paul Wilson’s O/R Mapper (WORM) and WilsonORWrapper (WORW) — but I just completed the switch to ActiveRecord.

Why abandon WORM/WORW? Three reasons:

  1. I was concerned that I’d run into situations where I needed more power than WORM could provide.
  2. NHibernate and ActiveRecord have incredibly robust communities that dwarf those of the alternatives.
  3. I needed to pick my first ActiveRecord project. 🙂

I still think WORM/WORW are valid choices, but the scope of CSBB is potentially huge, and I knew I needed to follow the leader.

Converting to ActiveRecord was straightforward (one O/R mapper to another), but I did run into one frustrating problem. I had a problem persisting one ActiveRecord classes. Attempting to save it didn’t throw an exception; rather, it just sat there, as if in an endless loop. I traced the call to an NHibernate method call by ActiveRecord, but couldn’t figure out why it wouldn’t do anything. Turning on logging, I recognized the problem: one of my fields (a not null field in the database) wasn’t marked with the [Property] attribute, so it wasn’t included in the INSERT statement. Running the SQL code in MySQL Query Browser threw an error, but not in NHibernate. Frustrating, and possibly an issue with the MySQL provider for NHibernate, but at least I know what to look out for if I see a similar problem in the future.

Still, it feels good to be in the hands of MonoRail + ActiveRecord for the most significant independent, most rewarding, and most unpaid development project of my life. 🙂

A repository that works with multiple O/R mappers — is it possible?

A recent post on the WilsonORWrapper Google group talked about making WORW multi-database aware by removing static classes and singletons. Well, I’ve kind of been thinking of something similar lately…

For some time I have been toying with the idea of rewriting WORW to support a generic repository service that would work with any O/R mapper, much
like the logger and cache services work over different underlying libraries. By removing the static classes and singletons and adding a provider model for the O/R wrapper element, you would be able to instantiate multiple instances of a “Registry” class, a class which provides a set of services (Repository, Cache, Logger, etc.) based on a given configuration criteira.

As a test I wrote up a simple project (with unit tests!) as a proof-of-concept, using WORM and NPersist as underlying O/R mappers. The WORM part worked fine. The NPersist part almost worked — however, I believe the errors are related to using NPersist’s not-too-well-documented XML provider, not errors with the O/R mapper provider itself (NPersist does initializes properly, which indicates at least something is working right).

If you’re interested in seeing this little stub of a project, download the source (27.5kb). I’ve dubbed it NRepository right now, not for any reason except it’s the first thing that came to mind. Let me know what you think, and whether you think it’s worth pursuing such a project.

Don’t trust the car dealer

nissan_pathfinder In April 2003, I bought a new Nissan Pathfinder from the local dealer, Staten Island Nissan. I also purchased a seven-year extended service plan. Since I typically keep my cars for 7+ years, I figured it’s a worthy investment (as I got a very good price on the car and 1.9% financing).

Fast-forward to July 2007. The car is just over four years old, and has just over 30,000 miles, when I hear a rattling sound. A quick look underneath shows that the muffler and tailpipe have separated from the catalytic converter; the rattling sound is the sound of the metal muffler and tailpipe swaying underneath my car.

Recognizing that an exhaust system which decides to separate itself from the rest of the car is not normal behavior for a four-year-old car, I take it to the service center at the dealer I purchased it from for diagnostics and the annual New York State-required vehicle inspection.

A few hours after dropping off the car, I get a call from the dealer, where I am told that I need a whole new exhaust system (cost: $1,200) and nothing is covered under the warranty or the extended service contract. After a fruitless debate about the reasoning for this (I’ve owned five cars for 4+ years and this is the first one that’s needed a whole exhaust system after four years), I gave up, and decided to call Nissan Consumer Affairs.

Before calling, I did my homework. Reading the details of the extended service plan, I found that numerous exhaust parts are covered. During the conversation with Nissan Consumer Affairs, I enjoyed stimulating dialog such as this:

Nissan: “Sir, I see you have the Service Gold extended plan – that’s the best plan we offer.”
Me: “Then why will you not service what’s clearly broken on my car?”
Nissan: “Sorry, sir, the dealer makes these decisions.”
Me: “What recourse do I have if the dealer is just trying to rip me off?”
Nissan: “Sir, our dealers are all Nissan certified and [bla bla bla]…”

After 30 minutes, I convinced the chap I was speaking with to escalate my case to a “Regional Case Manager” (or something like that). I was told I’d get a call back in 24 hours.

After waiting 24 hours plus three days without getting a return phone call, I called Nissan Consumer Affairs again. Upon providing my information, I was told the case manager already looked into my case. I asked why he didn’t call me with details; they said they’ll send an “internal message” to him requesting he calls me. They also gave me his phone number, which I called; of course, voice mail responded, and I left a message.

Another three days passed without hearing from anyone at Nissan, so I called again, leaving a blistering message for the case manager. Incredibly, I heard back from him in about two hours. The conversation went something like this:

Nissan: “Sorry, the dealer made the determination that the work needed is not covered under the warranty.”
Me: “So you’re telling me that it’s perfectly acceptable to Nissan for their vehicles to require the customer to purchase a $1,200 exhaust system after just four years of ownership?”
Nissan: “Sir, I’m sorry but it’s not covered under the warranty or the extended service plan.”

Nissan (NOT)Realizing I’m getting nowhere, I decide to speak with the Service Manager at the the service center where I brought my car originally. I left a message for him, and … he never called me back.

I decide to take an alternate approach: I take my car to the local mechanic (who I know and trust) and explain the situation. He looks under my car and gives me an estimate… for $193.00, parts and labor. When I tell him the dealer quoted me $1,200, he said, “They’re trying to sell you a whole exhaust system that you don’t need.”

Am I surprised? Not in the least. Now that I have all this information, here’s my next steps:

  1. Have my local mechanic fix my car for over $1,000 less than quoted by the dealer.
  2. Cancel my extended service plan and get a refund.
  3. File a complaint with the New York State Department of Motor Vehicles. You see, the dealer performed a safety and emissions inspection on my car at the same time they recognized that I need an entire exhaust system. My vehicle passed their inspection. I can’t imagine how a car with a disconnected exhaust system and a dangling tailpipe can pass a safety and emissions inspection.

The lessons I learned from this are:

  • Never buy an extended warranty. You lose all your consumer rights, as you are required to use a dealer-authorized service center for all repairs — whether or not they are under warranty. For example, if you have your brakes serviced at an independent repair shop, then take your car to the dealer for brake work, they will not cover any brake work that would otherwise be covered under the extended warranty because you previously had your brakes serviced by an unauthorized service center. How do I know this? They told me.
  • Before buying a new car, talk to the service manager. All new cars come with a manufacturer’s warranty, so you will most likely be dealing with the manufacturer’s service people at some point. You can buy a car in one day in the sales office, but for the next three years, your relationship is with the service center, so be sure you are comfortable with them.

When I have an update, I’ll be sure to let you know. Thanks for listening to my rant!

Use reflection to compare the properties of two objects

In an update to WilsonORWrapper, I added a method which takes two objects of the same type and compares the properties of each, returning a value reflecting the results of the comparison. Any value other than zero would indicate that at least one property on the objects are not equal.

This method may have interest to people who don’t use WilsonORWrapper, so here’s an extracted version of the code that does the comparison.

using System;
using System.Reflection;

public static class ObjectHelper<t>
{
	public static int Compare(T x, T y)
	{
		Type type = typeof(T);
		PropertyInfo[] properties = type.GetProperties(BindingFlags.DeclaredOnly | BindingFlags.Public);
		FieldInfo[] fields = type.GetFields(BindingFlags.DeclaredOnly | BindingFlags.Public);
		int compareValue = 0;

		foreach (PropertyInfo property in properties)
		{
			IComparable valx = property.GetValue(x, null) as IComparable;
			if (valx == null)
				continue;
			object valy = property.GetValue(y, null);
			compareValue = valx.CompareTo(valy);
			if (compareValue != 0)
				return compareValue;
		}
		foreach (FieldInfo field in fields)
		{
			IComparable valx = field.GetValue(x) as IComparable;
			if (valx == null)
				continue;
			object valy = field.GetValue(y);
			compareValue = valx.CompareTo(valy);
			if (compareValue != 0)
				return compareValue;
		}

		return compareValue;
	}
}

With that, if you had a Name class in your code that had two properties, First and Last, you could do something like this:

Name n1 = new Name();
n1.First = "Brian";
n1.Last = "DeMarzo";

Name n2 = new Name();
n2.First = "Brian";
n2.Last = "DeMarzo";

int result1 = ObjectHelper<name>.Compare(n1, n2);
// result1 == 0 because n1 and n2 have equal properties

n1.First = "Alyssa";
// change the first name, so n1 should no longer equal n2

int result2 = ObjectHelper</name><name>.Compare(n1, n2);
// result2 != 0 because n1 and n2 do not have equal properties

Code like this came in handy in my project where I use FileHelpers (see my blog entry from earlier today), where I was able to compare a class loaded from the database (using an O/R mapper) with a class loaded from a text file (using FileHelpers). Since the O/R mapper and FileHelpers used the same class, using this comparison method to determine if the objects were “equal” let me determine whether or not the data loaded from the database was different from the data in the text file.

Even though it used reflection, there wasn’t a huge performance hit, either. Sure beats writing manual comparison methods, which could take a while when you have a few dozen classes!

FileHelpers: the .Net way to import text files

I am working on a project where I have to import data from thousands of text files. These text files vary in formats: they use different delimiters; they use quoted identifiers never, sometimes, or always; they sometimes have missing end-of-row columns. Importing these took a combination of custom scripts (to clean up and reformat) and XML format files for SQL’s BCP (bulk copy) command… until I found FileHelpers.

FileHelpers by Marcos Meli is a .Net library which provides services to import and export text in delimited and fixed-length file formats. It takes a unique approach to the file import problem:

  • You write a class, giving it properties to store the data in the text file.
  • You apply attributes to the class and properties, describing the format of the file and applying import rules.
  • You import the file using a single command, which creates an array of objects populated with the data from the text file.

With support for custom converters, null types, optional fields, before/after import events, extensive parsing rules, and so much more — not to mention a web site with gobs of examples and documentation — FileHelpers has single-handedly driven me away from the script and batch file approach to a much more powerful .Net application to import my files.

The fact that FileHelpers uses regular .Net classes allows you to use the same class to both import data from a file and persist it to a database. For example, consider the following sample text file:

1,Chris,Smith
2,Bob,Jones
3,"Mary Jane",Brown

Clearly we have three fields in this comma-delimited file, so we write the following class to hold the data. Attributes provide instructions to FileHelpers — specifying that it is a comma-delimited file, and that the quotes in the first and last name fields are optional. (Note that you can use fields or properties; I use fields below for brevity.)

[DelimitedRecord(",")]
public class Person
{
	public int PersonID;
	[FieldQuoted('"', QuoteMode.OptionalForRead)]
	public string FirstName;
	[FieldQuoted('"', QuoteMode.OptionalForRead)]
	public string LastName;
}

Importing this file now becomes as easy as writing two lines of code.

FileHelperEngine&lt;Person&gt; engine = new FileHelperEngine&lt;Person&gt;();
Person[] records = engine.ReadFile(@"c:\myfile.csv");

You now have an array of Person objects loaded with data from the text file. Now, let’s say you wanted to persist these objects to a database. You can use the same Person object to persist to your database with an O/R mapper. If you were using Paul Wilson’s O/R Mapper and the WilsonORWrapper, you would decorate your class as follows:

[DelimitedRecord(","), IgnoreInheritedClass]
public class Person : WilsonORWrapper.Entities.EntityBase&lt;Person&gt;
{
	public int PersonID;
	[FieldQuoted('"', QuoteMode.OptionalForRead)]
	public string FirstName;
	[FieldQuoted('"', QuoteMode.OptionalForRead)]
	public string LastName;
}

Note I added the base class and the IgnoreInheritedClass attribute, which tells FileHelpers to ignore any inherited fields and properties (otherwise, the IsReadOnly property from EntityBase would be included).

With that, I can now persist all the records I read using FileHelpers back to my database using the O/R mapper.

FileHelperEngine&lt;Person&gt; engine = new FileHelperEngine&lt;Person&gt;();
Person[] records = engine.ReadFile(@"c:\myfile.csv");
foreach ( Person p in records )
{
	Data&lt;Person&gt;.Track(p);
}
Data&lt;Person&gt;.Save(records);

Incredible! We can even add supporting logic. Let’s say I wanted to only insert new persons into the database. I could do this.

FileHelperEngine&lt;Person&gt; engine = new FileHelperEngine&lt;Person&gt;();
Person[] records = engine.ReadFile(@"c:\myfile.csv");
foreach ( Person p in records )
{
	Person pers = Data&lt;Person&gt;.Retrieve("PersonID = " + p.PersonID.ToString());
	if (pers == null)
	{
		Data&lt;Person&gt;.Track(p);
		Data&lt;Person&gt;.Insert(p);
	}
}

FileHelpers makes this and so much more possible. It’s the backbone of a project I’m writing, and it’s making some heavy data-parsing much easier. To top it all off, FileHelpers is open source — you can download the code from their Subversion repository, and post patches on their support forums. Patches are well-received — I’ve already had two (1, 2) I’ve sent and had applied.

Thanks again to Marcos (and recently-added co-developer, Matt Campbell) for providing this fantastic library. For regular updates, be sure to read the FileHelpers Library Blog.

Supporting free software: TortoiseSVN

It’s a new month, which means it’s time for me to pick a free, open source product to donate to. This month was an easy decision: I picked the product that I use every day to manage my code (and code history): TortoiseSVN.

I’m not the only one who’s ga-ga over TortoiseSVN; they recently were declared the Best Tool or Utility for Developers by the folks at SourceForge. With Subversion being the de-facto source code management standard (even Microsoft is starting to pay attention to it), it’s no surprise that TortoiseSVN is so widely loved.

I’m happy to have joined the list of those who’ve donated to TortoiseSVN!

My donation history to date far is as follows.