REST Levels above 4

There is the Richardson Model of REST says REST APIs can be ranked like so:

1- Plain old XML. You serve up data in a data exchange format at HTTP endpoint. Ignores as much as possible about how HTTP was intended to work.
2- Resources have their own URL
3- Resources can be manipulated with GET, PUT, POST, DELETE
4- Resources return hypermedia, which contains links to other valid actions & acts as the state of the application.

I’ll add these:
5- Metadata. The API supports HEAD, OPTIONS, and some sort of meta data document like HAL
6- Server Side Asynch – There is support for HTTP 202 & an endpoint for checking the status of queued requests. This is not to be confused with client side asych. Server side asynch allows the server to close an HTTP connection and keep working on the request. Client side asych has to do with not blocking the browser’s UI while waiting for a response from the server.
7- Streaming – There is support for the ranges header for returning a resource in chunks of bytes. It is more like resumeable download, and not related to the chunk size when you write bytes to the Response. With ranges, the HTTP request comes to a complete end after each chunk is sent.

#5 is universally useful, but there isn’t AFAIK, a real standard.
#6 & #7 are really only needed when a request is long running or the message size is so large it needs to support resuming.

Clients should have a similar support Level system.
1 – Can use all verbs (Chrome can’t, not without JavaScript!)
2 – Client Side Caches
3 – Supports chunking & maybe automatically chunks after checking the HEAD
4 – Supports streaming with byte ranges

C# Basics– Classes and basic behavior

What upfront work is necessary to create fully defined classes? By fully defined, I mean, there is enough features implemented so that it play well with the Visual Studio debugger, WCF/WebAPI, Javascript and XML, and so on.

Ideally, this boilerplate would always be available to consume on your custom classes. In practices, it is uncommon to see any of the following implemented. Why is that?

So lets simplify reality and imagine that code is of only a few types:

(I’m suffixing all of these with -like to remind you that I’m talking about things that look like these, not necessarily the class or data structure with the same name in the .NET or C# spec or BCL)

  • Primative-like. Single value, appear in many domains, often formatted different in different countries. Sometimes simple, like Int32, sometimes crazy complicated like DateTime, sometimes missing, like “Money”.
  • Struct-like. Really small values, appear in some domains, like Latitude/Longitude pairs.
  • Datarow-like. Many properties,need to be persisted, probably stored in a relational or document database, often exchanged across machine, OS and organizational boundaries.
  • Service-like. These are classes that may or may not have state depending on the programming pradigm. They are classes with methods that do something, where as all the above classes, mainly just hold data and incidentally do something. It might be domain-anemic, like create, read, update and delete or it might be domain-driven, like issue insurance policy, or cancel vacation.
  • Collection-like. These used to be implemented as custom types, but with Generics, there isn’t as much motivation to implement these on a *per type* basis.
  • Tree or Graph-like. These are reference values that contain complex values and collection-like values and those turn also might contain complex values and collections.

All classes may need the following features

  • Equality- By value, by reference and by domain specific. The out of the box behavior is usually good enough and for reference types shouldn’t be modified. Typically if you do need to modify equality, it is to get by-value or by-primary-key behavior, which is best done in a separate class.
  • Ranking- A type of sorting. This may not be as valuable as it seems now that linq exists and supports .Sort(x=>…)
  • String representation- A way to represent this for usually human consumption, with features overlapping Serialization
  • Serialization- A usually two way means of converting the class into string, JSON, XML for persistence or communicating off machine
  • Factories, Cloning and Conversion- This covers creation (often made moot by IOC containers, which sometimes have requirements about what a class looks like), cloning, which is a mapping problem (made moot by things like automapper), and finally conversion, which is mapping similar types, such as Int to Decimal, or more like “Legacy Customer” to “Customer”
  • Validation- Asking an object what is wrong, usually for human consumption
  • Persistence- A way to save an object to a datastore. At the moment, this is nhibernate, EF, and maybe others.
  • Metadata- For example, the .NET Type class, an XSD for the serialized format, and so on.
  • Versioning- Many of the above features are affected by version, such as seralization and type conversion, where one may want to convert between types that are the same but separated by time where properties may have been added or removed. Round trip conversion without data loss is a type of a versioning feature.

How implemented

  • Ad hoc. Just make stuff up. Software should be hard, unpredictable and unmanageable. The real problem is too many people don’t want to read the non-existent documentation of your one-off API.
  • Framework driven. Make best efforts to find existing patterns and copy them. This improves your ability to communicate how your API works to your future self and maybe to other developers.
  • Interface driven. A bit old fashioned, but rampant. For example these:
    //Forms of ToString(), may need additional for WebAPI
    IFormattable, IFormatProvider, ICustomFormatter,
    //Sort of an alternate constructor/factory pattern
    IDisposable, //End of life clean up
    IComparable, IComparable, //Sorting
    //Competing ways to validate an object
    IValidatableObject, IDataErrorInfo,
    //Binary (de)serialization
    ISerializable, IObjectReference
  • Attribute driven. This is now popular for seralization APIS, e.g. DataContract/DataMember and for certain Validations.
  • Base Class- A universal class that all other classes derive from and implement some of the above concerns. In practice, this isn’t very practical, as most of these code snippets vary with the number of properties you have.
  • In-Class- For example, just implement IFormat* on your class. If you need to support 2 or more ways of implementing an interface, you might be better off implementing several classes that depending on the class you are creating features for.
  • Universal Utility Class- You can only pick one base class in C#. If you waste it on a utility class, you might preclude creating a more useful design heirarchy. A universal utility class has the same problem as a universal base class.
  • Code generation. Generate source code using reflection.
  • Reflection. Provide certain features by reflecting over the fields and properties.

All of these patterns entail gotchas. Someday when I’m smarter and have lots of free time, I’ll write about it.

Opinionated Trace

The problem with trace-debug-loggers (I’ll call it trace from now on) is that anything goes. I’m going to try to use the System.Diagnostics namespace with the extensions on codeplex.

Trace is a story, it is the moment to moment diary of the application. The audience of trace is a developer. They want to know what the application is doing. Computers do a lot, so trace volume will have to be carefully managed.

Thou shall not trace in production, unless you have to.
Trace can be expensive. I took a representative 1000 repetition integration test that ran in 1/2 second and turned on verbose logging with DebugView running, and it took 17 seconds. This is why there should be some thought put into logging levels and why there should be multiple trace sources, so that most of them can be off most of the time.

Thou shall be very wary of logging to the same transactional database as the application.
Jeff had a bad experience with this kind of logging and decided to throw the baby & bath water out. I think they just needed to rethink what trace’s promise and possibilities really are.

Thou shall use a TraceSource per class.
There should be a trace source per class. Typically we’re debugging a few classes at a time and turning off the other trace by commenting it out isn’t a practical solution.

Thou shall not use System.Diagnostics.Trace or System.Diagnostics.Debug
Use TraceSource instead. You can’t turn off these sources as easily as a TraceSource

Thou shall not reinvent System.Diagnostics. Extend it. Resist using other people’s re-inventions. Do use other people’s extensions
Trace is for maintenance developers. A maintenance developer shows up on the scene and the last thing they want to see is yet another custom solution for a solved problem. How excited would you be to find a code base that shunned System.IO’s file system API and used an entirely custom one? Your app has a bug. You have one problem. You find out all the trace is written using a odd ball trace infrastructure. Now you have two problems.

Thou shall not do start/end trace with nothing inbetween
Entry exit should be recorded for things that have multiple traced steps. If there is nothing in between start/end, it shouldn’t be added to the story *unless* you are doing performance. If you are recording enter/exit, you should also record the amount of time. You should use a Dispose patter to ensure that the End is written.

Thou shall write a unit/integration test that is has been tuned for a good trace story.
The trace story should be shorter than a novel, longer than a flippant comment.

Thou shall not write a Error trace unless we know it will also be error logged via Elmah or the like
Trace is not error logging. The maintenance developer is obliged to look at the error log, Trace is only on occasionally and even after tuning could have too much info.

Thou shall educate the maintenance developer on how to use the existing trace
The .NET framework has a couple of trace sources. To get at them, you have to just know that they exist. There isn’t an easy way to query an assembly and ask it what trace sources are there and what switches it takes to activate them.

Thou shall look for opportunities to replace comments with trace
We don’t want code to become less readable because of trace. So apply the same reasoning about deciding when to comment to when to log (don’t log the obvious, like most variable assignments)

Thou shall not focus on domain specific events
These would be things like “John editing record B”, or “Sold book to customer Q”.

Thou shall use trace as sort of a poor mans Method-was-called Assertion
For example, if you are caching an expensive value, then on first call, there should be a trace message from the CreateExpensiveValue method and on the second go round there shouldn’t be any trace message. But unlike a unit test, the assertion is verified by a developer reading code. This shouldn’t be a substitute for using mockign frameworks.

Thou shall not bother with Warn. Just use Error and throw an Exception.
Warnings need to have an audience and trace doesn’t always have an audience. Exceptions have an audience. And when an exception is thrown, we may want to add that to the story, since trace and error logs aren’t necessarily going to be together.

Thou shall not bother with Verbose. Just use Info
Lets say I write a trace and I call it information. Years later it is in a tight loop that is executed 10 times a millisecond. You can’t control or predict in advance if a given message is info or verbose.

Thou shall see the link between Trace and step through
Ever step through code that kept going throw a 3rd class and you though, I wish this would stop stepping through that class? You could add attributes (and remember to remove them later) or you could switch to a trace strategy that allows you to turn off trace for the likely-just-fine class.

ClickOnce Experiments

Okay, I used to think click once was a sandbox, kind of like Java applets. I used to think that all click once applications installed from an internet link would be put in a sandbox with partial trust so that certain .NET API and unmanaged code couldn’t be executed.

I was wrong! At least, according to my experiments today with .NET 4.0 and click once.

I took the cassini source code and wrote it so that it would launch and then set up a virtual directory for a website that I bundled with it, essentially as “resource”/”content” files. I figure out how to get that to work in .NET and in ClickOnce. I thought, gee, I thought the APIs necessary to load an AppDomain and host an ASP.NET site and serve files on port 80 would be forbidden by the sandbox, right? Initially I thought it was because I was installing it locally. So I put the files up on a website, downloaded and installed and it still let cassini run as in the ClickOnce local storage area, and serve up a website in Full Trust.

Well, the sandbox is opt-in. If a software publisher doesn’t opt in, the user just gets a warning that doesn’t really make any sense and the application runs in Full Trust.

I did check to see how cassinni runs in ClickOnce after opting-in to Internet level trust. Now, the click once version of Cassini fails as soon as it tries to find out the path to it’s own assembly files. I still got… well not so scary as unintelligible warnings about needing to “trust” the remote website.

Well, so much for sandboxing. Now one thing worth nothing is I only get the browsers warning “Hey this came from the internet, sure you want to run it?” I don’t get teh UAC curtain of “this application will change your machine”. I do get the unintelligible Click once, message “Unknown publisher, this app has access to your machine, start menu, and well, it came from the internet” I’m imaging grandma reading that and thinking, “Well, I don’t personally know them either and I’ve already been told this is from the internet” Where else does software come from? The machine I’m writing from doesn’t event have a CD drive.

So a malicious code writer would distribute code and not opt in to sandboxing, in full expectation that some people will click through the messages.

A non-malicious code writer would only get benefit from this if he opts into sandboxing, didn’t need those other APIs, and if a malicious code writer tried to sneak an assembly into the non-malicious application and execute it, maybe if the sandboxed app has a plug in feature. Why bother with malicious plugins when you can just get people to run your separate full trust app? And besides, to run a plug in .NET you need to be able to load assemblies on demand and I bet a medium or low trust application wouldn’t be able to do that.

ASP.NET Profile Compilation- Mystery Solved!

ASP.NET will kick off a little profile thingy when it finds a <property> section. It appears to go into the ASP.NET Temporary Folder. It seems to be a private method in ProfileBase that loads this.

Some errors that happen:
Sometimes deleting the <property> section and readding it works. This happens all the time for all sorts of reasons, not just profile related, but dynamic compilation and shadow assembly related.

ProfileCommon not available- this is supposed to be compiled either at design time, run time or sometime. It fails to work in a variety of circumstances.

Precompiled apps can have problems with the ProfileBase class
Ambiguous reference errors- can happen when App_Code.Compiled or PrecompiledApp.config exist or fail to exist and ProfileCommon gets compiled twice.
Similarly copying the bin of the new version into the bin of the old is bad. You get a blended set of .dlls
Deleting files in ASP.NET Temporary Files sometimes works.

Interestingly, all the above don’t seem to be addressable by writing a custom provider. It doesn’t seem possible to override the settings part, where it reads web.config and starts trying to create that problematic ProfileCommon.

It turned out to be a dll that was referencing a native dll that wasn’t on the PATH. The profile was just triggering a compile. The compile then check references for everything in the \bin\ folder, even if the profile or invoked page isn’t using it. Oddly, this doesn’t happen on every single page, so there are some code paths where ASP.NET doesn’t feel the need to recompile every !@#$@!#$ thing in the world. This, I suppose is good or else we’d see pathalogical compile churning.

I diagnosed this by watching the list of assemblies that were getting loaded in Visual Studio while the debugger was attached. If it couldn’t load symbols or if it was obviously a COM dll, I considered it a candidate and finally narrowed it down to a SMO library, which happens to have non-managed dependencies.

The sorry state of application security audit logging

Security oriented audit logging records what happened and who did it. So lots of logging frameworks fail because they aren’t very well integrated into the authentication scheme. Ideally, the log entries would non-deniable and read and write-only (i.e. no updates and deletes) by a non-administrator. And super-ideal would be a rich, granular permissions system. For example, NTFS rights are read/write. There isn’t an append-only right for files. And ordinary users need to be able to write to the log. Logging isn’t useful if only the administrator can write a log entry. And ordinary users need to be able to review the log, although not necessarily the same ordinary users that wrote to the log.

This excludes logging techniques where you can write

Log(“I committed a crime”);
DeleteLogs(); //or update logs

Likewise, this should’t happen when running as user John Doe:

Log(“Bill committed a crime”); //Not me!

Windows Event Log.
Requires pinvoke/win32 API calls to record event with user Id.
Can log to Application from ASP.NET with blank user from C#.
You might not even be able to tell which website or application wrote the error log entry.
Looks like it only works with windows authentication, and not custom authentication, e.g. GenericPricincipal.
Only an administrator can create a new log/event source.

Sql Event Log
No user (except SPID) is recorded
Xp_Logevent requires membership in dbo in master, or systeadmin, or explicit rights

EXECUTE master..xp_logevent 50003,’hello world E’, ERROR
RAISERROR (‘hello world from raiserror’, 500000,1) WITH LOG –Requires sysadmin rights to use WITH LOG

Response.AppendToLog(“Action recorded by Austin Powers, really.”);
Has the cs-username field, but it is the service name, not the real user.
For IIS5/6 you need to write an ISAPI filter to hook into the code writing to that log to change cs-username.

The given columnMapping does not match up with any column in the source or destination

The SqlBulkCopy object is good, but probably super twitchy. The fact that it is case sensitive to columnames to me is a sign of likely low code quality, and the error that gives this blog entry its title is another.

Sometimes bulk copy will import a table from text without explicitly setting the columns. Sometimes you have to explicitly set the columns and poof! it works. It may have to do with white space or case sensitivity in the columns, or it may have to do with drivers making inaccurate counts of fields.

SqlBulkCopy bcp = new SqlBulkCopy(
“Server=.;persist security info=True;initial catalog=target;Integrated Security=SSPI”,
SqlBulkCopyOptions.KeepIdentity & SqlBulkCopyOptions.KeepNulls);

bcp.BatchSize = 50000;

for (int i=0;i {
string name = reader.GetName(i).Trim().ToLower();

for (int i = 0; i < bcp.ColumnMappings.Count-1; i++)
bcp.DestinationTableName = file.Name.Split(‘.’)[0];



NOTE: Even explicitly listing the columns often is not enough. I eventually gave up and used alternate techniques. The command is poorly documented with respect to what text formats are acceptable for BIT, DATETIME, IMAGE when doing heterogeneous bulkcopy. Until MSDN documents it, this otherwise fantastic tool is only reliable for copying from SQL to SQL with identical schemas and probably identical version and patch levels.

Error Messages: On_Error_Insult_User

In the good ole days, if we hit an error message, we’d just write to the console a routine insult to the user.

If input_arg<0 then Print “You are such a dumb ass”

We now hide our contempt in secret code

If input_arg<0 then Print “Please contact your administrator”
/* You are such a dumb ass */

Sometimes we prefer the “I’m smarter than the runtime” pattern and send developers on a goose chase.

Sign the Pledge! Sign the Petition! Down with #region!

I beseech the developer community to swear to never use #region or #endregion blocks again, except for possibly machine generated code.

The first step is to stop typing #region and #endregion. This will greatly improve the transparency of your code. Next, because you are the sort of person that uses #region and #endregion to hide the code you are ashamed of– go find all those HACK:, BUG:, and CRAP: comments.

try/finally without catch

You will get a YSOD, but the finally block executes before the YSOD.

int bar = 1 / int.Parse(this.TextBox1.Text);
int z = 21;
int foobar = 1 + 1;

This is kind of like a IDisposable patter at the method and code block level. It means, on the way out of this code block, I want some clean up code to run. Off hand I can’t think of a good reason for this if you aren’t releasing resources. Using it as a control of flow technique looks very iffy especially if it wasn’t the error you were expecting.

Identical behavior to the above, YSOD, but finally block runs just before the YSOD. The following is code you’d see in code generation scenarios where the try/catch/block is code generated with the hope that the developer would fill better error handling later.

int bar = 1 / int.Parse(this.TextBox1.Text);
int z = 21;
catch (Exception) {
int foobar = 1 + 1;

YSOD after line one, no other code after the error is executed

int bar = 1 / int.Parse(this.TextBox1.Text);
int z = 21;
int foobar = 1 + 1;