Comic Book Creator

Comic book creator is a fantastic idea and after one day, I think it is 90% there.

You can use FRAPs to grab a bunch of screens, drop them into comic book panels, add speech balloons and create a comic at such an enormous speed that it boggles the mind. Doing the same with Paint and Word, or more professional tools like Gimp and desktop publishing would take days, plus a substantial learning curve.
The cost you pay for this power is dealing with a product with rough edges. I bought the CD version from Amazon because it cost less than direct download (go figure). The disk comes with an older version plus a CD key and no clear instructions on how to update to the version available online. The trick is to install the online version and use the CD key, as per advice in the CDC forum.

The installer is a bit broken. On two different machines it hangs 0% remaining. The expansion pack only installs to drive C and requires some hacking that I can do, but is inappropriate for the target user of kids and artists.

Moving Bits Faster

Everyone has to move files every once in a while. You might have two machine both with high speed internet connections, but maybe the connection is an unreliable VPN connection, or maybe it is just a very large file. For some reason, windows file shares across VPN just isn’t has high performance as you might want and there is very little that can be done about it, except zip it.

Get 7z. 7z currently has the best compressions ratios of almost any format, it is free and very fast. CPU nowadays is cheaper than bandwidth.

484MB->31MB So I’m going 15 times faster. But download with file shares over VPN is at 40Kb/sec

Get “Get Right“. Get Right is a file download manager. It has an accelerated mode where it opens 4 connections and downloads the file in four parts. It can also talk “Z mode” to FTP servers. “Get Right” costs money, though. And it does resume. And so forth.

Get Filezilla Server. Filezilla is free, supports Z-mode compression and FTP clients that accelerate by using multiple connections.

Bam, 86 Kb/sec and I was able to resume when the VPN connection died.

All together, Compression+FTP+FTP download managers can increase your bit throughput by 30x (neglecting compression and decompression time , which in my case was about a minute or two at 2MB/sec-6MB/sec.

Now what I’d like to hear is, what’s up Microsoft? Why can’t SMB file shares work like that?

Apparently I’m not the only one that has noticed that SMB performance is subpar. And Microsoft has mostly been comparing SMB against Samba. A protocol and a reverse engineering of the same protocol are likely to perform similarly. One of the key finding I see from these links is that there are not a lot of things a computer pro can do to tune SMB, while there are several compression and FTP tools.

MS Access as a Linked Server

I needed to simulate an Oracle server, but didn’t like the idea of trying to install and run Oracle Express. So I created a linked Server pointing to an Access MDB with tables linked to a SQL Server, which had tables dumped from the Oracle Server. This allows for OPENQUERY queries without the “db.dbo” prefix that you’d need if you’d just referenced the SQL database directly. You know a database is hard to use when you can’t convince database professionals to use the free copy installed on their machine.

These are the things I did to get my lined server working:

  1. Switch connection string from SQL authentication to Windows Authentication.

Without that, you get can’t find installable ISAM or authentication errors.

What didn’t work

  1. Giving more rights to IUSER and ASPNET account (this was only failing in my web app, not in SQL Server Management Studio)
  2. Setting the password in the provider string.

MSDTC Considered Harmful

MSDTC creates transactions between two database servers, maybe running on different machines, maybe running on different database engines. That way when you mark down Al’s account in one server for $10 and mark up Bill’s account in the other server for an increase of $10, we don’t end up in weird situations where Bill has $10 and Al still has $10, or Bill has $0 and Al has 0$.

But what if you live in the real world and you want to transfer 1.21 jigabytes of data from server A to B. First of all, this would create explosive log growth and crash your server. For some reason, SQL Server wants me to involve MSDTC when I do a INSERT INTO ServerA.mydb.dbo.mytable SELECT * from ServerB.mydb.dbo.mytable

MSDTC has security problems. It uses RPC, lots of ports and as such MS turned it off in Windows 2003. Re-enabling it is not as easy as flipping in a switch, you have to get into the constipated bowels of the operating system and edit the registry, services, COM+ settings and get your hands covered in binary smoo. After this, you reboot the computer and it still won’t work. This process of trial and error takes more hours than you want to spend on it.

I also tried turning automatic transactions off, no such luck.

Fortunately, a SELECT doesn’t start a transaction and using the ADO.NET SqlBulkCopy
class will not start a transaction. It is very fast and seems to be smart about data types, so a SELECT * works as a source query.

SMO vs TSQL vs Batch vs .NET vs VBS

The prototypical application I have in mind is the backup script, which involves launching a few processes, running a few TSQL commands like “BACKUP” and doing some file system shuffling, like checking for the existance of a file you intend to write, deleting old backup files, moving them to a safe place, etc.  To get anything done, you find yourself doing 3 crimes against computer science:

Code writing code to make up for missing features in the host language.

Parsing untyped output.  The output of DIR is unparsed.

Domain overspecialization.  TSQL does good with tables, it stubles when you need to write a batch script in TSQL.  Batch does good at launching processses, it fails when you need to do loops or handle errors. .NET has a wonderful library, but you have to roll your own 5 line process launcher.

In batch, you can launch a process in 1 line of code:

osql.exe {args}

In TSQL, you can launch a process, BUT, it is just a xp_cmd_shell command, so really you are writing batch and if you have long strings of xp_cmd_shell commands you should write batch.  However, TSQL will put the data into a single column table, so if the output of the batch command is one data element per row, it nicely maps to a table and you can use it as a table.  Still, you have to use TSQL’s string functions to build the command, there isn’t a independent parameter, data returned isn’t typed (it is a table of strings, not a collection of files or some other class).

SQLCMD.  SQLCmd is a SQL2005 tool that is a half measure to make some TSQL scripts easier to write and run, especiall when they need to be execute in different database contexts.  Previously this had been done by writing dynamic EXEC(“use db EXEC myCommand”) type scripts, which were not fun to write–again due to the lousy string handling of TSQL.

The resulting code is clunky and slow to write, a hybrid of a not very good procedural language with something that was never meant to be used as a programming language at all.

VBS has late binding.  This feature is hardly ever used for anything clever, mostly it exists to cripple the language and make it unfriendly to intellisense supporting IDE’s. COPY, DIR, DEL takes many lines of code.  SQLDMO exists, but why put yourself through the pain when you could write VB6 in the uncrippled way?  Well maybe because the environment is COM unfriendly.

VB6. It takes a couple of lines of code to launch a process and it is fairly hard to get the output of the command.  SQL DMO did exist to get object oriented access.  This requires an environment that is COM friendly.  Why deal with COM at all when you can use .NET and not worry if you have the right to register a COM dll?

.NET and SMO. The prototypical backup script in SMO requires instatiating a bunch of objects in a typed language and relying on the .NET framework for the file system shuffling.  In general, it takes more lines of code than the corresponding batch command, eg. COPY, DIR, DEL.  Still it is better than VB6.

WMI. WMI is just so damn hard to use.  I won’t talk about it. It makes me dizzy.WMI via .NET is better, but man you just never know what kind of object you’ll get back when you talk to a WMI object.  Calling everything in the world an object and accessing everthing with a .getProperty(string name) and  .invoke(string name) methods is not really typed programming.

Powershell.  I’m just starting to learn about Powershell.  It looks like the goal is to make batch into a real programming language, with the conciseness of COPY, DIR, DEL, the library of .NET, and some real programming language features. 

Powershell you’re my only hope.

Why I ripped MS Ajax 1.0 out of my app

1. It broke the blowery HTTP compression. Despite having the source code for blowery and attempting to step through it, I couldn’t figure out which event blowery should be hooking into to avoid conflict with MS Ajax. Ajax is supposed to increase subjective speed. Why should I have to remove the one component that was increasing objective speed to increase subjective speed?

2. The event behavior for everything changed. Everything I put into a update panel had different event behavior. In Soviet Ajax, you don’t call events, events call you! So buttons, buttons in columns just stopped working. It is clear the intention of the update panel was to allow developers to drop ordinary code into an update panel and make the post backs go away. In reality, you will have to do meticulous event wiring for anything you put into an update grid. I was just starting to understand the lifecycle of a page and how the life cycle of a page, control and master page interleaved and now I have to come up with a mental model for the regular page life cycle with the Ajax page life cycle.

3. Simply dropping gridviews into updatepanels was anything but simple. The web.config file requires extensive updating, you need a scriptmanager in the master page and see #2, again.

Integrated Security and Kiddie DOS Attacks

This morning my PC says my account is locked out.  Apparently, when I connect to my office by VPN, something on my computer was repeatedly trying to log into my computer using my name, Matthew Martin.  The password and domain were wrong, so Windows XP helpfully locked out my account, which I admit was the behavior I told to have.  I set a limit of something like 1000 failed logins before lockout.  On the “Welcome Screen”, there is no obvious way to log in as administrator.  Guess what feature I plan to start turning off on all the XP machines I come across?  That’s right.  Someday I’m going to need to tell someone in my family how to log on as administrator and they are going to be at a “Welcome Screen” without administrator on the list.  I managed to log on by using remote desktop.  I unlocked my account and found out that MYWORKDOMAIN\Matthew Martin had been attempting to log on about every minute or so all night.

The failed logon continue.  Where could it be coming from?  If VPN is turned off, the logons stop.  Currently, I think the culprit is SQL Server Management Studio, which will send at ‘Are you alive’ connection attempt to every server on your list. It could also have been remote desktop or filesharing, I can’t really prove anything.

If the remote server is a WORKGROUP and not a domain machine and the user names are the same, then a polling app can easily lock out an account just by repeatedly attempting to log into a remote machine.  As long as the logon type =3, there doesn’t appear to be any limits on the remote machine.

The Search for FAST Souce Control

The goal is to check out a big project from the office and edit it on my home PC using the company VPN connection.

Visual Source Safe, while not safe at any speed, isn’t able to reach very high speeds to begin with. I though for a few seconds about using the web services access, but decided slow web services on top of a file based database doesn’t equal faster. (however I will find out what the real numbers are)

Team System has a beautiful source control system, but my company has only 5 licences. And the set up and care and feeding for Team System is a remarkably high barrier to entry. (Soon I will see if it is worse than the others)

CVS is free, but has a user interface only a cave man could love.

Subversion. You are my only hope.

Subversion is a collection of command line applications, one of which can run as a server. The server component can be accessed across the network. There are two protocols that are safe, https:// and svn_ssh://, the former being Apache 2.0 and webdav based, the latter being based on ssh tunneling.

Subversion uses some weird protocols plus webdav. The svnserve process talks TCP/IP but has primitive security: passwords stored in plain text, doesn’t look like it encrypts traffic, etc. Webdav is a HTTP protocol that makes your web server work like a windows file share. Sort of. And at least you can encrypt traffic and use apache’s security.

To use webdav you need Apache 2.0, (not Apache 2.2!), which by the way is a steaming pile of sh*t if you need SSL. A newbie can set ssl running on IIS in a hour or two. I just burned a couple of hours trying to get Apache to run SSL. Openssl is also a steaming pile of sh*t. I don’t see why anyone would go through the significant effort of releasing code to the world without making any effort what so ever to make it work without a day of configuration. [Update, you need to use xca to generate the key and certificate, and define virtual directories in the conf file, that's what finally worked for me.]

Next you want the latest version of subversion, 1.4. Install that and set it up as a windows service.

Finally, you will want Ankh. Get Version 1.0 or higher. It is a Visual Studio plug in. It doesn’t use the same API that the VSS plug in uses, so don’t expect to find it on the options. You will need to have and see a solution to get the context menu where much of the Ankh action happens. Ankh doesn’t cover all the possible Subversion features, it is more for the basic check in, check out and see what’s changed. Alternatively, you could run the underlying commands

Appendix:

Subversion Feature List

* Working folder caches a copy of original, so change detection can be done entirely locally (to see if you change anything since last check out), saving bandwidth

* Changes are submitted as deltas (differences), not the entire file. Again, saves bandwidth.

* Changes are atomic. An update to several files either succeeds entirely or fails entirely. Changes are logged to the client, then executed. If the computer fails midway, changes can be reapplied next time.

* Working copy can become a mixture of versions. This is a feature or defect depending of your viewpoint.

* Locking-Change-Unlock is possible, but not the default workflow.

* Copy-Change-Merge is the default work flow.

* Branching is supposed to be easy.

* A VS2005 plug in exists.

* Not a relational database, but that’s okay because text and document databases are a poor fit for relational stores. It does however have a real database API behind it, Berkley DB.

* Free

* Works with SSH or SSL/Apache/Webdav

* Supports large repositories, so you needed maintain half a dozen separate repositories like in VSS.

* Tool support

Ankh (VS2005 plugin), version 1.0 or higher.

Apache (secure network layer), my advice, use the apache that comes with collabnet’s distribution.

WebSVN (php UI)

Tortoise (Client/Admin tool, stand alone GUI client, Windows Explorer integration)

Exporting Gridviews with ASP.NET

How to export a ASP.NET gridview to excel, plus how to deal with the occasional control not rendered in a form error.

Public Overrides Sub VerifyRenderingInServerForm(ByVal control As Control)
‘Do nothing.
End Sub

Protected Sub btnExport_Click(ByVal sender As Object, ByVal e As System.EventArgs) Handles btnExport.Click
Response.Clear()
Response.AddHeader(“content-disposition”, “attachment;filename=FileName.xls”)
Response.Charset = “”

‘If you want the option to open the Excel file without saving than

‘comment out the line below

‘Response.Cache.SetCacheability(HttpCacheability.NoCache)

Response.ContentType = “application/vnd.xls”
Dim stringWrite As System.IO.StringWriter = New System.IO.StringWriter()
Dim htmlWrite As System.Web.UI.HtmlTextWriter = New HtmlTextWriter(stringWrite)

Me.grdData.RenderControl(htmlWrite)
Response.Write(stringWrite.ToString())
Response.End()
End Sub

Visual Studio Team Server Impressions

The thing has 10 Sql Databases. Most shops have applications with just one database and they have a rough time doing the DBA work to care for and feed them.

It has seven big pieces. By big, I mean, each one you could spend a year accumulating experience before you’d feel a mediocre competency. Reporting Services. SQL Server. Sharepoint. Analysis Services. Visual Stuidio (the IDE). IIS and Web services and off course the Team Server specific functionality behind them all. Add Windows Server to the list if you are used to running code strictly on workstations.

Installation isn’t as bad as the beta. In the beta I never successfully installed a version–the requirements are very picky. For example SQL Developer edition is not good enough for a test install, you need a real SQL Standard Edition. Still, installation took the better part of 4 evenings.

Creating a project takes 10 minutes. I guess web services are a bit slow. Speaking of slow, half of the install struggle was finding a machine powerful enough to run the VPC. It quickly becomes obvious why the install documents have a separate section explaining how to split the server across several physical machines.

Ok, enough kvetching. I like that it has a CMMI process baked in. Team Server supports a massive number of process templates and reports. Firefox seems to work with the web sites.

I filed a bug. It took me a while to realize that you can’t do much in the sharepoint section, for example you can’t file a bug report there. Instead you file bug report in the VS2005 IDE. It started at bug 16. I guess it found 15 bugs before I could check in the first line of code. Unfortunately I was a “Dumb Ass End User”(TM) and the bug report I filed failed to show up on anything except the “All My Team Project Work Items”, it didn’t show on many of the other “List-Bugs” reports. Same thing happened on my “Task Item”. I can just imagine now a pointy haired boss filing 50 bug reports 50 task items and all of them being ignored because he didn’t flip the right bits. And if you are a process person reading this, I can hear you thinking, “Training is the answer.” Come on, get real. Dogs can be trained, but not PHB’s when your living in a Dilbert world.

The source control plug in rocks. This is already miles ahead of VSS and I’ve only been looking at it for 10 minutes. The source code compare is still the VSS one. I can lock a single file and still have multiple check out for others (without creating a brand new repository with a different check in policy, like you have to do in VSS!) It is much easier to view those little comments people are supposed to make when they check in a change set. I never figured out how to do that in VSS. And not only that, I can rename a file and see the history and do compares between the file before and after I changed it!

The build utility left me scratching my head. It is modal and puts a death grip on your instance of visual studio while it is running. It can’t be interrupted. It can’t be stopped. Ha ha! Little does that puny programmer in Redmond who doesn’t want me to stop the build know that I control the electrical supply to the computer! It takes a few minutes to time out.

Also, there needs to be 3rd process, after MSF and CMMI called COWBOY. It means you don’t give a flying foik about the heaps of reports, analysis documents and the overly complex work flows, but you do want to be able to see what your code looked like in case you have to revert, branch or do some other tricky code change.