When a Decimal does not equal a Decimal

It all started with a simple bug ticket in Trello that said “When I select 0.125 from the drop down, save, then enter the edit screen again the drop down is set to 0”.

Seemed pretty simple. The clock said 5:45pm and I figured “this should be easy to knock out”. An hour later I was pulling my hair out.

What?

Here’s a good example of the offending code:

public class FooRepository
{
	public decimal GetFooValue() 
	{
		return 0.1250m;
	}
}

public class FooModel
{
	public decimal Value { get; set; }
}

public class FooController
{
	public ActionResult Index()
	{
		var repository = new FooRepository();
		var model = new FooModel { Value = repository.GetFooValue() };
		return View(model);
	}
}

Then in the view:

@Html.DropDownListFor(x => x.Value, new SelectList(new[] {0m, 0.125m, 0.25m}))

Every time I displayed this view the drop down was always on the first item and it was driving me nuts. After playing around in Linqpad I came across a clue:

0.250m.ToString(); // Outputs "0.250"

I stared for a while and finally noticed the trailing zero in the displayed output. I then made sure I wasn’t totally crazy so I tried:

0.250m.ToString() == 0.25m.ToString() // outputs False

I looked in my real database where the value was coming from, and since it is a decimal(18,4) entity framework is bringing it back with 4 decimal places, which means it’s including the trailing zero. Now it makes sense why Asp.net MVC’s helpers can’t figure out which item is selected, as it seems like a fair assumption that it is calling ToString() and doing comparisons based on that.

While trying to figure out a good solution I came across this StackOverflow answer which had the following extension method to normalize a decimal (which removes any trailing zeros):

public static decimal Normalize(this decimal value)
{
    return value/1.000000000000000000000000000000000m;
}

After normalizing my model’s value, the select list worked perfectly as I expected and the correct values were picked in the drop down list.

Of course this is a clear hack based on implementation details and I reverted this fix. There’s no telling if or when a new version of the .Net framework may change this, and from what I’ve read a lot of the internal details of how decimals work are different in Mono, and this hack does not play nicely in Mono (supposedly in mono, 0.1250m.ToString() does not display trailing zeros).

The proper way to resolve this situation is to force the drop down list to do a numerical equality to determine which item should be selected, by manually creating a List instead of using the SelectList() constructor.

Why?

So even though I knew what was going wrong, I was interested in why. So that stack overflow answer pointed me to the MSDN documentation for the Decimal.GetBits() method. That link contained specifications for how a decimal is actually constructed. Specifically that internally a decimal is made up of 4 integers, 3 representing the low, medium, and high bits in the value and the last integer containing information on the power of 10 exponent for the value. Those combined can give exact decimal values (within the specified range and decimal places allowed).

So to start with I tried the decimal of 1m and printed out the bit representation:

new[] {
	Convert.ToString(decimal.GetBits(1m)[0], 2),
	Convert.ToString(decimal.GetBits(1m)[1], 2),
	Convert.ToString(decimal.GetBits(1m)[2], 2),
	Convert.ToString(decimal.GetBits(1m)[3], 2),
}

// 0000000000000000000000000000001
// 0000000000000000000000000000000
// 0000000000000000000000000000000
// 0000000000000000000000000000000

That didn’t show much unexpected. It shows the low bits as containing just a 1 and all other bits containing zeros. Next I tried 1.0m:

new[] {
	Convert.ToString(decimal.GetBits(1.0m)[0], 2),
	Convert.ToString(decimal.GetBits(1.0m)[1], 2),
	Convert.ToString(decimal.GetBits(1.0m)[2], 2),
	Convert.ToString(decimal.GetBits(1.0m)[3], 2),
}

// 0000000000000000000000000101000
// 0000000000000000000000000000000
// 0000000000000000000000000000000
// 0000000000000010000000000000000

This parses the core bits to:
lo: 10
mid: 0
high: 0

So how does it convert a 10 into a 1? Looking back at the MSDN documentation it says that the first 15 bits are always zero (as they are in this case) with the next seven bits being the exponent of the decimal. Bits 16-23 are 0000001 and the last bit is zero, giving us an exponential value of positive one. These mean that to get the final value we take the value of the low + mid + high bits combined (10) and divide them by 10 to the power of positive 1. This gives us a value of just 1,

If we look at our 0.125m example we get:

new[] {
	Convert.ToString(decimal.GetBits(0.125m)[0], 2),
	Convert.ToString(decimal.GetBits(0.125m)[1], 2),
	Convert.ToString(decimal.GetBits(0.125m)[2], 2),
	Convert.ToString(decimal.GetBits(0.125m)[3], 2),
}

// 0000000000000000000000001111101
// 0000000000000000000000000000000
// 0000000000000000000000000000000
// 0000000000000110000000000000000

Like before, this is taking a value of 125 (125 + 0 + 0) and dividing it by 10 to the positive 3 exponent, which gives us 0.125. If we instead use 0.1250m we get:

new[] {
	Convert.ToString(decimal.GetBits(0.1250m)[0], 2),
	Convert.ToString(decimal.GetBits(0.1250m)[1], 2),
	Convert.ToString(decimal.GetBits(0.1250m)[2], 2),
	Convert.ToString(decimal.GetBits(0.1250m)[3], 2),
}

// 0000000000000000000010011100010
// 0000000000000000000000000000000
// 0000000000000000000000000000000
// 0000000000001000000000000000000

This represents 1,250 (1,250 + 0 + 0) divided by 10 to the positive 4 exponent.

So now it’s clear that when you create a new decimal it essentially keeps track of the exact representation it was originally invoked with, including zeros and is pieced together to it’s fully realized value on a ToString() call.

Making .Net Regular Expressions Easier To Create and Maintain

Recently I had the idea of a log parsing and analysis system, that would allow users (via a web interface) to define their own regular expressions to parse event logs in different ways. The main purpose of this is for trend analysis. Logstash is an open source project that does something similar, but it relies on configuration files for specifying regular expressions (and doesn’t have other unrelated features I am envisioning).

One issue I have with regular expressions is they can be very hard to create and maintain as they increase in complexity. This difficulty works against you if you are trying to create a system where different people with different qualifications should be able to define how certain logs are parsed.

While looking at Logstash, I was led to a project that it uses called Grok, which essentially allows you to define aliases for regular expressions and even chain aliases together for more complicated regular expressions. For example, if you needed a regular expression that included checking for an IP address, you can write %{IP} instead of (?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9]). This makes regular expressions much easier to create and read at later on.

The problem with Grok is that it is written in C as a stand alone application. This makes it hard to use in .Net applications without calling out to the shell, and even then it is cumbersome to use with dynamic aliases due to having to reconfigure config files on the fly.

For those reasons, I created an open source library I called RapidRegex. The first part of this library is the freedom to generate regular expression aliases by creating an instance of the RegexAlias class. This class is used to give you full flexibility on how you store and edit regular expression aliases in any way your application sees fit, whether it’s in a web form or in a flat file. It does however come with a class that helps form RegexAlias structures from basic configuration files.

As a simple example, let’s look at the regular expression outlined earlier for IP addresses. With RapidRegex, you can create an alias for it and later convert aliased regular expressions into .net regular expressions. An example of this can be seen with the following code:

            var alias = new RegexAlias
            {
                Name = "IPAddress",
                RegexPattern = @"\b(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b"
            };

            var resolver = new RegexAliasResolver(new[] { alias });

            const string pattern = "connection from %{IPAddress}";
            var regexPattern = resolver.ResolveToRegex(pattern);
            // Resolved pattern becomes "connection from \b(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b"

What makes this even more powerful is the fact that you can chain aliases together. For instance, an IPv4 address can be defined as 4 sets of valid IP address bytes. So we can accomplish the same above with:

            var alias = new RegexAlias
            {
                Name = "IPDigit",
                RegexPattern = @"(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)"
            };

            var alias2 = new RegexAlias
            {
                Name = "IPAddress",
                RegexPattern = @"%{IPDigit}\.%{IPDigit}\.%{IPDigit}\.%{IPDigit}"
            };

            var resolver = new RegexAliasResolver(new[] { alias, alias2 });

            const string pattern = "connection from %{IPAddress}";
            var regexPattern = resolver.ResolveToRegex(pattern);

            // Resolve the pattern into becomes "connection from \b(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b"

The project can be found on Github and is licensed under the LGPL license.

DotNetOpenAuth, OAuth, and MVC For Dummies

I recently was trying to understand OAuth so that I could utilize the LinkedIn API in my Asp.Net MVC application. The LinkedIn API has a pretty good summary of the OAuth process. However, the more I looked at it (and other documentation on OAuth), the more confused I got at implementing OAuth myself. So I went around and looked for a C# library to help me out, and found DotNetOpenAuth.Net. Unfortunately, the DotNetOpenAuth website is horribly designed without any real tutorials. After searching around the internet I was able to piece some things together, and hopefully this will help someone figure this out quicker than I was able to.

Requesting User Authorization

The first step of authorizing with OAuth and DotNetOpenAuth is to redirect to the OAuth provider’s authorization page, so the user can grant your application access to perform queries/service calls on their behalf. DotNetOpenAuth needs several pieces of information to begin this process. The first is to create a ServiceProviderDescription object, that contains the provider’s URL for retrieving the request token, URL for retrieving the access token, URL for requesting user authentication, what OAuth protocol version to use, and details of the tamper protection used to encode the OAuth signature. An example of creating the provider description for connecting to LinkedIn is:

        private ServiceProviderDescription GetServiceDescription()
        {
            return new ServiceProviderDescription
            {
                AccessTokenEndpoint = new MessageReceivingEndpoint("https://api.linkedin.com/uas/oauth/accessToken", HttpDeliveryMethods.PostRequest),
                RequestTokenEndpoint = new MessageReceivingEndpoint("https://api.linkedin.com/uas/oauth/requestToken", HttpDeliveryMethods.PostRequest),
                UserAuthorizationEndpoint = new MessageReceivingEndpoint("https://www.linkedin.com/uas/oauth/authorize", HttpDeliveryMethods.PostRequest),
                TamperProtectionElements = new ITamperProtectionChannelBindingElement[] { new HmacSha1SigningBindingElement() },
                ProtocolVersion = ProtocolVersion.V10a
            };
        }

The next thing that DotNetOpenAuth requires is a token manager. The token manager is a class which DotNetOpenAuth utilizes to store and retrieve the consumer key, consumer secret, and a token secret for a given access key. Since how you will store the user access tokens and token secrets will vary project to project, DotNetOpenAuth assumes you will create your own token storage and retrieval mechanism by implementing the IConsumerTokenManager interface.

For testing, I looked online for an in memory token manager class, and found the following code:

    public class InMemoryTokenManager : IConsumerTokenManager, IOpenIdOAuthTokenManager
    {
        private Dictionary<string, string> tokensAndSecrets = new Dictionary<string, string>();

        public InMemoryTokenManager(string consumerKey, string consumerSecret)
        {
            if (String.IsNullOrEmpty(consumerKey))
            {
                throw new ArgumentNullException("consumerKey");
            }

            this.ConsumerKey = consumerKey;
            this.ConsumerSecret = consumerSecret;
        }

        public string ConsumerKey { get; private set; }

        public string ConsumerSecret { get; private set; }

        #region ITokenManager Members

        public string GetConsumerSecret(string consumerKey)
        {
            if (consumerKey == this.ConsumerKey)
            {
                return this.ConsumerSecret;
            }
            else
            {
                throw new ArgumentException("Unrecognized consumer key.", "consumerKey");
            }
        }

        public string GetTokenSecret(string token)
        {
            return this.tokensAndSecrets[token];
        }

        public void StoreNewRequestToken(UnauthorizedTokenRequest request, ITokenSecretContainingMessage response)
        {
            this.tokensAndSecrets[response.Token] = response.TokenSecret;
        }

        public void ExpireRequestTokenAndStoreNewAccessToken(string consumerKey, string requestToken, string accessToken, string accessTokenSecret)
        {
            this.tokensAndSecrets.Remove(requestToken);
            this.tokensAndSecrets[accessToken] = accessTokenSecret;
        }

        /// <summary>
        /// Classifies a token as a request token or an access token.
        /// </summary>
        /// <param name="token">The token to classify.</param>
        /// <returns>Request or Access token, or invalid if the token is not recognized.</returns>
        public TokenType GetTokenType(string token)
        {
            throw new NotImplementedException();
        }

        #endregion

        #region IOpenIdOAuthTokenManager Members

        public void StoreOpenIdAuthorizedRequestToken(string consumerKey, AuthorizationApprovedResponse authorization)
        {
            this.tokensAndSecrets[authorization.RequestToken] = string.Empty;
        }

        #endregion
    }

Now that we have a Token Manager class to use, and a service description we can begin the authorization process. This can be accomplished with the following code:

        public ActionResult StartOAuth()
        {
            var serviceProvider = GetServiceDescription();
            var consumer = new WebConsumer(serviceProvider, _tokenManager);

            // Url to redirect to
            var authUrl = new Uri(Request.Url.Scheme + "://" + Request.Url.Authority + "/Home/OAuthCallBack");

            // request access
            consumer.Channel.Send(consumer.PrepareRequestUserAuthorization(authUrl, null, null));

            // This will not get hit!
            return null;
        }

This sets up the DotNetOpenAuth consumer object to use our in memory token manager, and our previously defined service description object. We then form the URL we want the service provider to redirect to after the user grants your application access. Finally we tell the consumer to send the user authorization request. The Send() method will end the execution of the Asp.Net page, and thus no code after the Send() call will be called. The user will then see the authorization page on the service provider’s website, which will allow them to allow or deny access for your application.

Receiving the OAuth CallBack

Once the user logs into the service provider and gives your application authorization, the service provider will redirect to the callback URL specified in the previous code. The service provider includes the oauth token and secret, which needs to be processed by DotNetOpenAuth. The following code can be done to process the auth token and store it and the secret in the token manager:

        public ActionResult OAuthCallback()
        {
            // Process result from the service provider
            var serviceProvider = GetServiceDescription();
            var consumer = new WebConsumer(serviceProvider, _tokenManager);
            var accessTokenResponse = consumer.ProcessUserAuthorization();

            // If we didn't have an access token response, this wasn't called by the service provider
            if (accessTokenResponse == null)
                return RedirectToAction("Index");

            // Extract the access token
            string accessToken = accessTokenResponse.AccessToken;

            ViewBag.Token = accessToken;
            ViewBag.Secret = _tokenManager.GetTokenSecret(accessToken);
            return View();
        }

Perform A Request Using OAuth Credentials

Now that we have the user’s authorization details we can perform API queries. In order to query the API though, we need to sign our requests with a combination of the user’s access token and our consumer key. Since we retrieved the user’s access token in the previous code, you need to figure out a way to store that somewhere, either in the user’s record in the database, in a cookie, or any other way you can quickly get at it again without requiring the user to constantly re-auth.

In order to use that access token to call a service provider API function, you can form a prepared HttpWebRequest by calling the PrepareAuthorizedRequest() method on the WebConsumer class. The following is an example how to use an access token to query the LinkedIn API.

        public ActionResult Test2()
        {
            // Process result from linked in
            var LiServiceProvider = GetServiceDescription();
            var linkedIn = new WebConsumer(LiServiceProvider, _tokenManager);
            var accessToken = GetAccessTokenForUser();

            // Retrieve the user's profile information
            var endpoint = new MessageReceivingEndpoint("http://api.linkedin.com/v1/people/~", HttpDeliveryMethods.GetRequest);
            var request = linkedIn.PrepareAuthorizedRequest(endpoint, accessToken);
            var response = request.GetResponse();
            ViewBag.Result = (new StreamReader(response.GetResponseStream())).ReadToEnd();

            return View();
        }

And now, if the user has authenticated going to /Home/Test2 will correctly access LinkedIn on behalf of the user!

Update: For those who are looking for a tool to help test API calls prior to having to write them down formally in code, please see my Testing Oauth APIs Without Coding article!

Why Entity Framework 4.1’s Linq capabilities are inferior to Linq-to-Sql

I have a small-ish website at work that contains several tools we use internally. I originally coded the database layer using Linq-to-Sql, as it was too small to overcome the learning curve of NHibernate and Entity Framework version 4 was not out at the time.

However, lately database schema changes have been harder to incorporate into the Linq-to-Sql edmx, mostly due to the naming differences between database tables and relationships and their C# class names. I decided to convert my database layer to Entity Framework 4.1 CodeFirst based on an existing database, as I use EF CodeFirst in a current project and already familiar with it.

The process of converting my Linq-to-sql was *supposed* to be a simple matter of replacing my calls to my L2S data context to my EF DbContext, and allow Linq to take care of the rest. Unfortunately, this is not the case due to Entity Framework’s Linq capabilities being extremely limited.

Very Limited Function Support

In Linq-to-Sql it was common for me to perform transformations into a view model or an anonymous data type right in the linq. Two examples of this in my code are:

var client = (from c in _context.Clients
                where c.id == id
                select ClientViewModel.ConvertFromEntity(c)).First();

var clients = (from c in _context.Clients
                orderby c.name ascending
                select new
                {
                    id = c.id,
                    name = c.name,
                    versionString = Utils.GetVersionString(c.ProdVersion),
                    versionName = c.ProdVersion.name,
                    date = c.prod_deploy_date.ToString()
                })
                .ToList();

These fail when run in Entity Framework with NotSupportedExceptions. The first one fails because it claims EF has no idea how to deal with ClientViewModel.ConvertFromEntity(). This would make sense, as this is a custom method, except this works perfectly in Linq-to-Sql. The 2nd query fails not only for Utils.GetVersionString(), but it also fails because EF has no idea how to handle the ToString() off of a DateTime, even though this is all core functionality.

In order to fix this, I must return the results from the database and locally do the transformations, such as:

var clients = _context.Clients.OrderBy(x => x.name)
                              .ToList()
                              .Select(x => new
                              {
                                  id = c.id,
                                  name = c.name,
                                  versionString = Utils.GetVersionString(c.ProdVersion),
                                  versionName = c.ProdVersion.name,
                                  date = c.prod_deploy_date.ToString()
                              })
                              .ToList();

No More Entity to Entity Direct Comparisons

In Linq-to-Sql I could compare one entity directly to another in the query. So for example the following line of code worked in L2S:

context.TfsWorkItemTags.Where(x => x.TfsWorkItem == TfsWorkItemEntity).ToList();

This fails in Entity Framework and throws an exception because it can’t figure out how to compare these in Sql. Instead I had to change it to explicitly check on the ID values themselves, such as:

context.TfsWorkItemTags.Where(x => x.TfsWorkItem.id == tfsWorkItemEntity.id).ToList();

It’s a minor change, but I find it annoying that EF isn’t smart enough to figure out how to compare entities directly, especially when it has full knowledge of how the entities are mapped and designed. Yes, I could have gone straight through using TfsWorkitemEntity.Tags, but this is a simple example to illustrate lost functionality.

Cannot Use Arrays In Queries

This issue really caught me by surprise, and I don’t understand why this was omitted. In my database versions consist of 4 parts: major, minor, build, and revision numbers, and is usually represented in string form as AA.BB.CC.DD. I have a utility method that converts the string into an array of ints, which I then used in my Linq-to-Sql query:

int[] ver = Utils.GetVersionNumbersFromString(versionString);
return context.ReleaseVersions.Any(x => x.major_version == ver[0] && x.minor_version == ver[1]
                                    && x.build_version == ver[2] && x.revision_version == ver[3]);

Under L2S, this query works fine, but (as the common theme in this post) fails in Entity Framework 4.1 with a NotSupportedException with the message “The LINQ expression node type ‘ArrayIndex’ is not supported in LINQ to Entities.”.

In order to fix this I had to split my version into individual ints instead.

Dealing With Date Arithmetic

My final issue I have come across is probably the most annoying, mostly because I can’t find a good solution except to do most of the work on the client side. In my database I store test requests, and those requests can be scheduled to run at a certain time on a daily basis. This scheduled time is stored in a time(7) field in the database. When I query for any outstanding test requests one of the criteria in the query is to make sure that the current date and time is greater than today’s date at the scheduled time. In L2S I had:

var reqs = _context.TestRequests.Where(x => DateTime.Now > (DateTime.Now.Date + x.scheduled_time.Value)).ToList();

This fails in Entity Framework for 2 reasons. The first is that DateTime.Now.Date isn’t supported, so i had to create a variable to hold that and use that in the query.

The 2nd issue is then that EF can’t make a query out of the current date added to a specific time value. This causes an ArgumentException with the message “DbArithmeticExpression arguments must have a numeric common type.”.

I have not found a way to do this in EF, and instead had to resort to pulling a list of all TestRequest entities from the database, and locally pull out only the ones that fit that criteria.

Conclusion

I am utterly baffled by how limited Entity Framework is in it’s Linq abilities. While I will not go back to L2S for my internal tool I am definitely having second thoughts of using EF 4.1 in my more complex personal projects. It definitely seems that Microsoft is going more for feature count in their frameworks rather than functional coverage lately.