Tuesday, November 24, 2009

Convert VB.Net to C#

Sometimes I see code snippets online written in VB.Net. Yes, its not hard too work out what the code it doing, but it is still slower than working with C# which I am much more familiar with.

I found this:
http://www.developerfusion.com/tools/convert/vb-to-csharp/

It translates between the two languages.

Friday, November 20, 2009

find .png files larget than 64k, and copy to a temp directory

find /cygdrive/c/ -size +64k -name '*.png' -print0 | xargs -0 -n1 -I % cp % /cygdrive/c/temp/images/png/

Wednesday, November 18, 2009

Recursive Methods, Stack Overflow, IIS & FXCop

Had some serious issues lately around crashing IIS. The logs indicated a Stack Overflow Exception, error code 0x800703E9.

But it wasn't obvious what caused this, since no recent changes seemed to be the culprit, and developers seemed to be unaware of any recursion in our code.

Unfortunately, there is far too much code to aimlessly search through.

I identified the following possible cases where we may have introduced recursion:

1. Intended recursion, a method directly calling itself somewhere
2. Possibly intended recursion, a method indirectly calling itself via another method.
3. Unintended recursion when an overloaded method calls itself instead of one of the other overloads because of too many or few arguments.
4. Unintended recursion because of a typo in a property getter or setter
5. Unintended recursion because of a missing (T) cast when calling 'static bool T.Equals( T x, T y)' inside an 'override bool Equals(object obj)'
6. .Net library classes throwing this exception.


More info:

1.
This is straight forward enough... a method somewhere has a flaw either in the logic that identifies stopping conditions causing infinite recursion, or the depth of recursion depends on the inputs, and the method is processing inputs larger than the developer considered.

2.
See above.


3.
Overloading, without being careful with the arguments can mean a method calling itself instead of the other overload, e.g.

public class X
{
public int A( int f)
{
// infinite recursion - developer should have
// written 'return A(f, 0);'
return A(f);
}

public int A( int f, int g)
{
return f + g;
}
}



4.
The following two properties cause infinite recursion. The first has a typo in the getter, the second has a typo in the setter. (the case is incorrect)

private string abc;
public string Abc
{
get { return this.Abc; }
set { this.abc = value; }
}

private string xyz;
public string Xyz
{
get { return this.xyz; }
set { this.Xyz = value; }
}


5.
It is common practice to implement public override bool Equals(object obj) in classes that are used in collections. It is also not uncommon to implement public static bool Equals(T a, T b) in the same classes too (e.g. the System.string class)

However, if the overridden method directly calls the static method without both of it's arguments cast to the correct type (T), then recursion could result, e.g.
public class T
{
public override bool Equals(object obj)
{
return T.Equals(this, obj);
}

public static bool Equals(T a, T b)
{
return true;
}

}


The above code will cause infinite recursion, since the first method is actually calling static bool object.Equals(object a, object b), not static bool T.Equals(T a, T b) which will then call bool T.Equals( object obj) on each argument.

To fix this, object obj must be cast to type T before calling the static method:
public class T
{
public override bool Equals(object obj)
{
return T.Equals(this, (T)obj);
}

public static bool Equals(T a, T b)
{
return true;
}

}



6.
Possibly Linq to SQL issue?
http://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=355026


But how to identify any methods like this in a large library of code? FXCop to the rescue.

This is my first time using FXCop, but I couldnt find any existing rules that would identify these cases, so I set about writing some custom rules ( Tutorial )

A. Identifying direct recursive methods ( covers cases 1, 3 & 4 above)

I based my custom rule on the "Callees" example class here, with the following 'Check' implementation:
public override ProblemCollection Check(Member member)
{
Method method = member as Method;
if (method != null)
{
string unmangledName = method.GetUnmangledNameWithTypeParameters();

IList callees = Callees.CalleesFor(method);
foreach (Method c in callees)
{
if (string.Equals(c.FullName, method.FullName) && string.Equals(c.GetUnmangledNameWithTypeParameters(), unmangledName) )
{
Resolution res;
if (c.Name.Name.StartsWith("get_") || c.Name.Name.StartsWith("set_"))
{
res = this.GetResolution();
res = new Resolution(string.Format("Check property name typo. {0}", res.Format), new string[] { c.FullName } );
}
else
{
res = this.GetResolution(new string[] { c.FullName });
}
Problems.Add( new Problem(res) );
}
}
}
return this.Problems;
}



B. Identifying indirect recursive methods ( covers case 2 above)

Here's my implementation:
public override ProblemCollection Check(Member member)
{
Method method = member as Method;
if (method != null)
{
//string unmangledName = method.GetUnmangledNameWithTypeParameters();
string fullName = method.FullName;
if (!string.IsNullOrEmpty(fullName) && !fullName.StartsWith("Microsoft.") && !fullName.StartsWith("System."))
{
List methodStack = new List();
methodStack.Add(fullName);

IList callees = Callees.CalleesFor(method);

TraverseCallees(methodStack, callees);
}
}
return this.Problems;
}


private void TraverseCallees(List methodStack, IList callees)
{
foreach (Method c in callees)
{
//string unmangledName = c.GetUnmangledNameWithTypeParameters();
string fullName = c.FullName;
if (!string.IsNullOrEmpty(fullName) && !fullName.StartsWith("Microsoft.") && !fullName.StartsWith("System."))
{
if (methodStack.Contains(fullName))
{
// only match against the head of the stack, since all methods will be processed as the head once
if (string.Equals(methodStack[0], fullName))
{
// only show indirect calls
if (methodStack.Count > 2)
{
StringBuilder callStackDescription = new StringBuilder();

//bool logStack = false;
foreach (string methodFullName in methodStack)
{
//if (!logStack && methodFullName.Equals(fullName))
//{
// logStack = true;
//}

//if (logStack)
//{
callStackDescription.AppendFormat("{0} -> ", methodFullName);
//}
}
callStackDescription.AppendFormat("{0}", fullName);

Resolution res = new Resolution(GetResolution().Format, new string[] { callStackDescription.ToString() });

Problems.Add(new Problem(res, c));
}
else
{
// ignore recursion, where it is a method directly calling itself
}
}
}
else
{
methodStack.Add(fullName);

IList nextCallees = Callees.CalleesFor(c);

TraverseCallees(methodStack, nextCallees);

methodStack.RemoveAt(methodStack.Count - 1);
}
}
}
}



C. Overridden T.Equals Calling Object.Equals ( covers case 5 above )

public override ProblemCollection Check(Member member)
{
Method method = member as Method;
if (method != null)
{
string fullName = method.FullName;
if (!string.IsNullOrEmpty(fullName) && !fullName.StartsWith("Microsoft.") && !fullName.StartsWith("System."))
{
// check of our 'Equals' overrides the defualt 'object.Equals(object obj)', since that may be called by 'static object.Equals(object, object_'
if (string.Equals(method.OverriddenMethod == null ? null : method.OverriddenMethod.FullName, "System.Object.Equals(System.Object)"))
{
IList callees = Callees.CalleesFor(method);

foreach (Method c in callees)
{
if( c.IsStatic && string.Equals( c.FullName, "System.Object.Equals(System.Object,System.Object)") &&
c.Parameters.Count == 2 &&
string.Equals( c.Parameters[0].Type.FullName, "System.Object" ) &&
string.Equals( c.Parameters[1].Type.FullName, "System.Object" ))
//TODO: also check the the VALUE of one of the arguments is really of the same type as the parent caller method
{
Problems.Add(new Problem(GetResolution(), c));
}
}
}
}
}
return this.Problems;
}



Sure enough, my custom rules found new examples of cases 1 and 3. And I am aware of cases 4 & 5 happening in the past, although luckily they were fixed during development.

Monday, October 19, 2009

HTTPS Client Certificate Auth to Redmine and Subversion

Internally, we have been using Redmine (www.redmine.org) for managing development projects, and well as subversion for version control.

The time has come for providing access to both of these from the Internet. However, we do not want anyone who is not unauthorised to do so to be poking around the servers we make available.

The solution is to make the applications available over HTTPS, and to require the correct client certificates be installed on the client browsers, before a connection can be made.


A. Redmine Path.
Initially, Redmine was running through it's own webserver on http://server:3000/
However, we dont want to have Redmine available from a root url like this, but instead have to run from some sub-URI, e.g. .../redmine/, since other paths will refer to other applications.

Do do this, I did as was described in many other places on the Internet(http://www.redmine.org/wiki/1/HowTo_Install_Redmine_in_a_sub-URI), I added the following line to the end of \config\environment.rb

ActionController::AbstractRequest.relative_url_root = "/redmine"

Unfortunately, after restart the CSS and Javascript was not found, giving me an ugly, mostly white page. This was not something I found mentioned much, and the only solution I found was the following: to move everything inside Redmine's \public\ directory into \public\redmine\ to match the sub-URI path expected.

B. Apache and HTTPS

Below is a subset of the configuration directives used to enable HTTPS access. Anything not listed is a default setting in the example provided httpd-ssl.conf. The certificate files listed below were created in my previous post on using XCA to create various types of certificates.


<VirtualHost 192.168.1.250:443>

ServerName www.mydomain.com:443

SSLEngine on

# PEM encoded certificate
SSLCertificateFile "c:/Program Files/Apache Software Foundation/Apache2.2/conf/www.mydomain.com.pem"

# Where to find CA certificates for client authentication
SSLCACertificateFile "c:/Program Files/Apache Software Foundation/Apache2.2/conf/MyCA.crt"

# To force clients to use Client Certificates
SSLVerifyClient require
SSLVerifyDepth 2

...
...

</VirtualHost>


I added the following section to the above to provide access to Redmine:


<Location /redmine>
RequestHeader set X_FORWARDED_PROTO 'https'
ProxyPass http://localhost:3000/redmine
ProxyPassReverse http://localhost:3000/redmine
</Location>


Note: the above allows anyone with a valid trusted certificate (signed by 'MyCA' CA) to connect. Of course, they should not be able to get past Redmine's login page without a valid login. This is the same Basic Auth as we have been using in the office, except external access wraps it all inside a HTTPS connection.

We have allowed limited to our Redmine server to external testers, however they should have no access to our Subversion repository.

So the URL to our subversion, say /svn/ should be more restrictive than for Redmine - a trusted client certificate is still necessary, but it is not enough. The client certificate needs to have certain properties to be granted access to Subversion.

In our case, any client certificates that should be granted access to Subversion, must have 'OurCompany' set correctly in the Organisation field of the client certificates Distinguished Name. Any other client certs, e.g. those of external testers, must have the Organisation field set to something else.

The following snippet fulfils these requirements, where the subversion repository would be in c:\SVN\.


<Location /svn/>
SSLRequire %{SSL_CLIENT_S_DN_O} eq "OurCompany"

DAV svn
SVNPath C:/SVN/
AuthzSVNAccessFile "C:/Program Files/Apache Software Foundation/Apache2.2/conf/svn-acl"
AuthUserFile "C:/Program Files/Apache Software Foundation/Apache2.2/conf/svn-auth-file"

#SSLOptions +FakeBasicAuth +StrictRequire
#SSLUserName SSL_CLIENT_S_DN_CN

AuthName "Subversion"
AuthType Basic
AuthBasicProvider anon
Anonymous "*"
Require valid-user

<limitexcept>
Require valid-user
</limitexcept>

</Location>



The above snippet restricts access based on the particular client certificate in use, however once access have been granted, it still uses Basic Auth for Subversion authentication, with all subversion users configured in svn-acl. This is identical to the non-https internal access used in the office.

Ideally, the client certificate itself could be used to provide those authentication details, without needing a separate Basic Auth login after the HTTPS connection has already been negotiated.

However, this proved difficult. First, I enabling the following two lines (commented out in the above snippet):


SSLOptions +FakeBasicAuth +StrictRequire
SSLUserName SSL_CLIENT_S_DN_CN


The effect of these is to use the client certificate's Common Name as the username for subversion authentication. It then fakes the basic auth process with this username. Details of how to set up subversion with Basic Auth are only a quick search away.

The problem with the above is that the actual username coming from the client certificate is "CN=/bob" instead of "bob", so if I set the certificate Common Name to "Bob", and Bob has an internal username called "Bob", subversion logs for example will not have the same username when commits are made inside and outside the office by the same person.

Creating my own Root CA, HTTPS Server and Client certificates, with XCA

The following assumes XCA 0.7.0

Prelim:
Before doing anything, you need to create the XCA database. Pick a strong password.


A. To create the Root Certificate

1. Select the 'Certificates' tab.
2. Click on 'New Certificate' on the left side.

3. Select the 'Source' tab.
4. Signing -> 'Create a self signed certificate with the serial 1'
5. Signature algorithm -> 'SHA 1' ( I read something about older system having problems with the better 'SHA 256' )
6. Template -> '[default] CA'. Click Apply!! Do not forget to apply.

7. Select the 'Subject' tab.
8. At the very least, fill in 'Internal Name', 'Common name', e.g. 'MyCA' for both.
9. Private Key -> click 'Generate a new key'

10. Select the 'Extensions' tab
11. Make sure the 'Time range' is 10 years or so. If changed, be sure to 'Apply'

12. Select the 'Key Usage' tab.
13. If you clicked 'Apply' to CA template (step 6), you should see 'Certificate Sign' and 'CRL Sign' highlighted.

14. Select the 'Netscape' tab.
15. If you clicked 'Apply' to CA template (step 6), you should see 'SSL CA', 's/MIME CA' and 'Object Signing CA' highlighted.

16. Click 'OK' bottom right corner

'MyCA' should now be listed under the main 'Certificates' tab



B. To create the HTTPS Server certificate

1. Select the 'Certificates' tab.
2. Right Click on the 'MyCA' entry -> 'New Certificate'

3. Select the 'Source' tab.
4. Signing -> 'Use this certificate for signing: MyCA'
5. Signature algorithm -> 'SHA 1'
6. Template -> '[default] HTTPS_server'. Click Apply!! Do not forget to apply.

7. Select the 'Subject' tab.
8. At the very least, fill in 'Internal Name', 'Common name', e.g. 'www.mydomain.com' for both. The common name MUST exactly match the full domain name of the webserver to be protected.
9. Private Key -> click 'Generate a new key'

10. Select the 'Extensions' tab
11. Make sure the 'Time range' is appropriate. The default of 365 days may be too short. If changed, be sure to 'Apply'!!

12. Select the 'Key Usage' tab.
13. If you clicked 'Apply' to HTTPS_server template (step 6), you should see 'Digital Signature', 'Non Repudiation' and 'Key Encipherment' highlighted.

14. Select the 'Netscape' tab.
15. If you clicked 'Apply' to HTTPS_server template (step 6), you should see 'SSL Server' highlighted.

16. Click 'OK' bottom right corner,

The new certificate for 'www.mydomain.com' should now be listed below MyCA under 'Certificates' tab ( once Tree View is selected )



C. To create the HTTPS Client certificate

1. Select the 'Certificates' tab.
2. Right Click on the 'MyCA' entry -> 'New Certificate'

3. Select the 'Source' tab.
4. Signing -> 'Use this certificate for signing: MyCA'
5. Signature algorithm -> 'SHA 1'
6. Template -> '[default] HTTPS_client'. Click Apply!! Do not forget to apply.

7. Select the 'Subject' tab.
8. At the very least, fill in 'Internal Name', 'Common name', e.g. 'Tester' for both.
9. Private Key -> click 'Generate a new key'

10. Select the 'Extensions' tab
11. Make sure the 'Time range' is appropriate. The default of 365 days may be too short. If changed, be sure to 'Apply'!!

12. Select the 'Key Usage' tab.
13. If you clicked 'Apply' to HTTPS_client template (step 6), you should see 'Digital Signature', 'Key Encipherment' and 'Data Encipherment' highlighted.

14. Select the 'Netscape' tab.
15. If you clicked 'Apply' to HTTPS_client template (step 6), you should see 'SSL Client' and 'S/MIME' highlighted.

16. Click 'OK' bottom right corner,

The new certificate for 'Tester' should now be listed below MyCA under 'Certificates' tab ( once Tree View is selected )



D. Exporting the server certificate

For apache, I exported as 'PEM Cert + Key' format, giving me the 'www.mydomain.com.pem' file. I also needed to configure the 'MyCA.crt' so that Apache would trust the client certificates signed my 'MyCA'



E. Exporting the client certificates

In order for a client browser to trust the certificate for 'www.mydomain.com', it needs a copy of the Root Certificate, i.e. 'MyCA'

It is possible to include the Root Certificate with the client certificate, by exporting the client certificate ('Tester') as "PKCS #12 with Certificate Chain"

This gives the file 'Tester.p12' that Internet Explorer is happy to import.

BUT, I encountered problems with Firefox (3.5.4). When importing, it still would not trust 'www.mydomain.com'. It turns out that although the import process itself is successful, and the Client and Root certificate are listed under 'Your Certificates' and 'Authorities', Firefox does not seem to find the Root certificate when it makes a https connections to www.mydomain.com.

The solution to this was to first export 'MyCA' in the 'PEM' format as MyCA.crt, and then export the Client Certificate 'Tester' in the 'PKCS #12' format.

Then I imported both separately into Firefox, MyCA.crt as an 'Authority' and Tester.p12 under 'Your Certificates'

And since this method also works for IE, I do this in all cases, so I dont need different certificates for IE and Firefox.



F. One other point...

In the case where client certificates are being used for https access to subversion via TortoiseSVN, it made things easier (although less secure when transferring the .p12 files), to leave the password blank when exporting to the .p12 file format.

Doing this means there is no need to import private keys into TortoiseSVN to be able to open the .p12 file, or being constantly asked for the password for this file whenever using subversion.

Additionally, in order for TortoiseSVN to trust the https server certificate installed on www.mydomain.com, I needed to install my Root Authority cert in TrotiseSVN.

To do this, I need to edit the client's subversion\servers file. In the [globals] section, right at the end of the file, I pointed the 'ssl-authority-files' parameter at a local copy of the 'MyCA.crt' root certificate file.

Tuesday, October 13, 2009

Remote CMD.exe Shell

I've been using SSH on both Windows and Linux for a long time to get remote access to various machines, and to tunnel TCP connections through firewalls.

On windows, this has generally meant installing Cygwin's SSHD server, which gives you the cygwin Bash shell.

While I'm comfortable in a unix environment, when I log into a windows box I generally prefer to see a shell that looks like a standard windows command line, e.g. I generally don't want the slashes going the wrong way, and I don't want the whole windows filesystem visible a few levels too deep (/cygdrive/c/ for example )

I knew about the http://sshwindows.sourceforge.net/ project, but it has not been active for a long time (although It does now look active again). And I dont want to have to install something new, when I already have a working SSHD server on many machines. There's also WinSSHD, but that is not free.

I just discovered, by editing /etc/passwd I can make my existing cygwin SSHD installations give me a CMD.exe prompt instead of a Bash shell.

All I had to do was find the line corresponding to my user, and edit the end of the line, changing '/bin/bash' to '/cygdrive/c/WINDOWS/system32/cmd.exe'

And thats it!

Unfortunately, this stops SCP working. I do know the above OpenSSH for Windows project does not have this problem by using the 'switch.exe' program, but I dont see that included with a standard cygwin installation.

To get around this issue, I just created another user, same as the my first, but with the original '/bin/bash' shell configured instead of cmd.exe. I log in with this user when I need SCP.

Update: It doesn't like tab file completion, Ctrl-C is killing putty instead of the remote application, and I forgot how important 'vim' is in these remote terminals... so I'm switching back to the default bash shell. I can always just start cmd.exe when inside bash...

Tuesday, October 6, 2009

Locking & Lock Free Collections - Multithreaded Performance

Been exploring Lock-free collections lately, for a highly threaded memory based caching project.

Before I really know how well I'm doing, I need to explore the performance of simpler locking collection classes (simple wrappers around Dictionary, using lock {} and ReaderWriterLockSlim) , as well determine the absolute maximum performance, by testing lockless readonly collections (i.e. read-only operations on pre-filled Dictionary)

1.
So, for the baseline absolute maximum performance, I get approx 55M read operation/sec on a simple Dictinary object, with 10K key/value pairs... this throughput varies little for 4 to 64 parallel threads, on a standard quad-core machine.

2.
I then need tested against a wrapper class, that uses an internal Dictinary, but allows parallel reads, and serialised write access using the ReaderWriterLockSlim class.
100% Reads: 2threads: 5.35M reads/sec, 4threads: 5.12M
reads/sec, 16threads: 4.34M reads/sec, 64threads 3.53M reads/sec

Overriding Enum ToString()

Use the [Description] attribute

http://blogs.msdn.com/abhinaba/archive/2005/10/20/483000.aspx

Some other interesting comments in there too...

Monday, October 5, 2009

Edimax 7318USg (USB) on Windows Server 2003 (Win2K3)

How to modify a driver's INF file to work with different hardware...

1. Inserted the USB adapter
2. Went to Device Manager, Right Clicked on Device with Yellow Exclamation, Properties
3. 'Details' tab -> 'Device Instance Id' -> Made a note of 'VID_148F&PID_2573'
4. Downloaded Driver for D-Link WUA-1340 for Win2k/XP/Vista (ver 1.3 - 20-9-07)
5. Extracted the zip file
6. Went into Drivers\WinXP_2K_9X\ folder
7. Opened Dr71WU.inf
8. Replaced every instance of 'VID_07D1&PID_3C03' with 'VID_148F&PID_2573'
9. Right clicked on yello exclamation, "Update Driver', pointed it at Drivers\WinXP_2K_9X\ folder
10. Associated with my network
11. Feel very pleased with myself...

Roundtrip DateTime to SQL Server

I was playing with something like an SQL transaction version of a CAS operation, which returned a DateTime. I was then comparing the returned DateTime to the value passed in, if they matched, the update was successful.

But the value returned was NOT matching the value I put in, even if the update did actually happen... the cause:

http://seesharper.wordpress.com/2008/07/08/sql-server-datetime-vs-net-datetime-battle-of-accuracy/

From MSDN on the SQL DateTime type: "Accuracy – Rounded to increments of .000, .003, or .007 seconds"

What I was seeing was the millisecond value back very slightly different from the original.

I also noticed the 'Kind' property was different - from 'UTC' to 'Unspecified'... does this also affect the direct comparison of these? I suppose it does... I must check that.

In the mean time, I changed my transaction to return a boolean, indicating whether the update happened, instead of a DateTime. Problem solved