Tag Archives: c#

Beware of this WCF Serialization Pitfall

Ideally, one should avoid data contracts with complex graphs- especially with repeated references and definitely ones with circular references. Those can make your payload explode on serialization. With repeated references, you may run into an integrity issue on deserialization. With circular references, the serialization will enter a recursive loop and you will probably run into a stack overflow.

Seeing that in certain situations, this becomes unavoidable, WCF has a way that you can tell it to preserve object references during serialization. You do this by setting IsReference to true on the DataContract attribute that you use to decorate the composite type that is your data contract.

So, for example:

[DataContract(IsReference = true)]
public class MyContract
{
   [DataMember]
   public string Member1 { get; set; }
   ...
   ...
}

This solves the problem- however, since WCF achieves this by augmenting the WSDL- beware of this when you are exposing your service to third parties (especially ones that are not using WCF or perhaps not .NET at all) and interoperability is a concern. Without IsReference set to true, the WSDL snippet for the above would look something like:

<xs:complexType name="MyContract">
  <xs:sequence>
    <xs:element minOccurs="0" name="Member1" nillable="true" type="xs:string"/>
    ...
  </xs:sequence>
</xs:complexType>

With IsReference set to true, this is what it looks like:

<xs:complexType name="MyContract">
  <xs:sequence>
    <xs:element minOccurs="0" name="Member1" nillable="true" type="xs:string"/>
    ...
  </xs:sequence>
  <xs:attribute ref="ser:Id"/>
  <xs:attribute ref="ser:Ref"/>
</xs:complexType>

See those two lines that got added (i.e. “Id” and “Ref”)? That could very well cause some other party’s WSDL parser/proxy generator to choke. You have been warned.

Advertisements

Diagnosing MEF Composition Errors

For all its goodness, if something goes wrong, problems with MEF are terribly hard to diagnose. Thankfully, there’s an article out there by Daniel Plaisted at Microsoft that goes into great detail into all the things that can go wrong with MEF and how to get to the bottom of each one. I have it bookmarked, and if you work a lot with MEF, you should too. The one area that I find most useful, though, is figuring out composition-time errors using tracing.

With MEF, if you are using the Import attribute and there are errors during composition – either due to a bad Export, a missing DLL, or even an exception in the constructor, the composition itself will not choke. It will simply not load the Export and therefore anything that depends on it into the container. The problem then manifests itself as a missing Export when you try to use it. By then, it’s too late to know what happened.

An easy diagnosis tool from the article I mentioned above is to drop the following snippet into your application configuration file (App.config, Web.config, what-have-you).

<system.diagnostics>
  <sources>
    <source name="System.ComponentModel.Composition" switchValue="All">
      <listeners>
         <add name="fileListener" type="System.Diagnostics.TextWriterTraceListener" initializeData="composition.log" />
      </listeners>
    </source>
  </sources>
  <trace autoflush="true" indentsize="4" />
</system.diagnostics>

After the composition routine runs, check the file configured above (in this example, “composition.log“) for any errors. You will have to sift through a lot of “everything’s okay” type messages, but errors stand out fairly easily. If something is causing the composition to skip certain parts, it will tell you exactly what and why – so you can fix it and move on rather than scratch your head and wonder what’s wrong. Make sure you disable it after you’re done though – the trace is pretty verbose and if you leave it on, that file will grow extremely large before you know it.

An Easy Service Proxy Executor for WCF

If you have adopted service oriented architecture (SOA) and are using WCF as the hosting/communication mechanism for your internal services, chances are you are doing one of two things: you publish each service like any old WCF service and your other services which are consumers of said published service consume it through its WSDL; or you create shared libraries that include the contract information that both the service and its consumer reference. Both are somewhat cumbersome but can be managed. If all your services are internal, though, going the WSDL route is somewhat of an unnecessary overhead and is just a bit more unmanageable.

Now, if you decide to go the second route, but still stick to the more obvious interface that WCF provides to instantiate and call proxies (ClientBase and the like), that is a bit of a waste – since those classes were built with generated-code proxies in mind. In that case, the better option really is to have a mechanism to obtain a ServiceEndpoint and use that along with the contract information to create your own ChannelFactory – where you can then call CreateChannel to get your proxy. A lot less code and a lot more manageable.

To this end, for my own purposes, I built a bunch of classes that comprise my WCF service executor module. This is part of the Services namespace in the new Commons Library. Here are what a few of the key classes look like – you should be able to surmise how they can be used. The most common usage example would be:

var response = ServiceCallerFactory
   .Create<IMyContract>()
   .Call(x => x.MyOperation(request));

 

IServiceCaller

public interface IServiceCaller<out TChannel>
{
     void Call(Action<TChannel> action);
     TResult Call<TResult>(Func<TChannel, TResult> action);
}

ServiceCaller

public class ServiceCaller<TChannel> : IServiceCaller<TChannel>
{
      private readonly ServiceEndpoint endpoint;
      private readonly EndpointAddress endpointAddress;

      public ServiceCaller() {}

      public ServiceCaller(ServiceEndpoint endpoint)
       {
             this.endpoint = endpoint;
       }

      public ServiceCaller(EndpointAddress endpointAddress)
      {
             this.endpointAddress = endpointAddress;
      }

      public void Call(Action<TChannel> action)
      {
             var channelFactory = this.endpoint != null
                   ? new ChannelFactory<TChannel>(this.endpoint)
                   : new ChannelFactory<TChannel>();

            if (this.endpointAddress != null) channelFactory.Endpoint.Address = endpointAddress;

            var channel = channelFactory.CreateChannel();
             try
             {
                   action(channel);
             }
             catch
             {
                   channelFactory.Abort();
                   throw;
             }
             finally
             {
                   channelFactory.Close();
             }
       }

      public TResult Call<TResult>(Func<TChannel, TResult> action)
      {
             var channelFactory = this.endpoint != null
                   ? new ChannelFactory<TChannel>(this.endpoint)
                   : new ChannelFactory<TChannel>();

            var channel = channelFactory.CreateChannel();
             try
             {
                   return action(channel);
             }
             catch
             {
                   channelFactory.Abort();
                   throw;
             }
             finally
             {
                   channelFactory.Close();
             }
       }
}

ServiceCallerFactory

public static class ServiceCallerFactory
{
       private static readonly object serviceCallerMapLock = new object();

      private static readonly IDictionary<Type, ServiceCaller> serviceCallerMap = new Dictionary<Type, ServiceCaller>();

      public static Func<Type, ServiceEndpoint> ServiceEndpointAccessor { get; set; }

      public static IServiceCaller<TChannel> Create<TChannel>(EndpointAddress endpointAddress = null)
      {
             ServiceCaller caller;
             if (serviceCallerMap.TryGetValue(typeof (TChannel), out caller))
                   return (IServiceCaller<TChannel>) caller;

            lock (serviceCallerMapLock)
            {
                   if (serviceCallerMap.TryGetValue(typeof (TChannel), out caller))
                         return (IServiceCaller<TChannel>) caller;

                   if (ServiceEndpointAccessor != null)
                   {
                         var serviceEndpoint = ServiceEndpointAccessor(typeof (TChannel));
                         if (endpointAddress != null) serviceEndpoint.Address = endpointAddress;
                         caller = new ServiceCaller<TChannel>(serviceEndpoint);
                   }
                   else
                   {
                         caller = endpointAddress == null
                               ? new ServiceCaller<TChannel>()
                               : new ServiceCaller<TChannel>(endpointAddress); 
                   }

                  serviceCallerMap[typeof (TChannel)] = caller;
            }

            return (IServiceCaller<TChannel>) caller;
      }
}

Writing your own LINQ provider, part 4

This is the last in a short series of posts on writing your own LINQ provider. A quick outline of the series:

  1. A primer
  2. Provider basics
  3. A simple, pointless solution
  4. A tiny ORM of our own (this post)

A tiny ORM of our own

In the previous post, we took a look at a simple, albeit pointless example of a LINQ provider. We wrap the series up this time by looking at something a little less pointless – a LINQ-based ORM, albeit a very rudimentary one. As with the previous one, it helps to take a look at the source code first:

This is a very simple example implementation and has its limitations. It only works with SQL Server. It only supports reads. It only supports these methods:

  • Any
  • Count
  • First
  • FirstOrDefault
  • Select
  • Single
  • SingleOrDefault
  • Where
  • OrderBy
  • OrderByDescending
  • ThenBy
  • ThenByDescending

There are a few more limitations, but again the point is not to redo NHibernate or Entity Framework. There is also a simple fluent mapping interface that you can use as so:

Mapper.For<MyThing>("my_thing_tbl")
    .Member(x => x.Id, "id")
    .Member(x => x.Name "thing_name")
    .Member(x => x.Date "thing_date");

Once you’ve got your mappings in place, it is up to you to create the DB connection. Once you’ve done that, you can create an IQueryable<T> out of aSqlConnection instance and then do LINQ on top of it.

using (var conn = new SqlConnection("..."))
{
    conn.Profile(Console.WriteLine); // Write generated query to console.
    conn.Open();

    var query = conn.Query<MyThing>();

    var things = query
        .Where(x => x.Id < 1000)
        .OrderBy(x => x.Name)
        .Select(x => new {x.Id, x.Name, x.Date})
        .ToArray();
}

Or, using the “other” syntax:

var things = (from item in query
                where item.Id < 1000
                orderby item.Name
                select new { item.Id, item.Name, item.Date }).ToArray();

If you recall Step 2 from provider basics, there were two options. The last solution used Option 1, i.e. there is one queryable that just builds up the expression and the enumerator does the parsing. For this one, we’re using Option 2, where we have a separate implementation of IQueryable<T> for each type of query operation to support.

When you first obtain a queryable, you get an instance of TableQueryable (which is descended from SqlQueryable) corresponding to the table that the type is mapped to. Each call on top of it then wraps the queryable in another descedant of SqlQueryable (e.g. WhereQueryable for Where operations, and so on). This logic is in SqlQueryProvider. Similarly, for executable methods, the appropriate type of ExecutableBase is created and called. Beyond this, the actual query creation logic is implemented in the individual queryables and executables defined within the Constructs namespace.

The queryables and executables work with classes within the QueryModel namespace that represent parts of a SQL query. Based on what operation they stand for, they convert an Expression into a Query, which can then be converted into a SQL string. Each queryable implements a Parse method that does this, and as part of doing this, it parses the queryable it wraps and then works on top of the Query object returned by the inner Parse, and so on until the top-most enumerable gives you the final query. The innermost or leaf queryable is always TableQueryable which simply adds the name of the source table to the query model.

LINQ is undoubtedly awesome, but knowing how it works gives you new appreciation for just how powerful it can be. Man, I love LINQ.

Writing your own LINQ provider, part 3

This is the third in a short series of posts on writing your own LINQ provider. A quick outline of the series:

  1. A primer
  2. Provider basics
  3. A simple, pointless solution (this post)
  4. A tiny ORM of our own

A simple, pointless solution

In the previous post, we took a look at what happens when you call LINQ methods on IQueryable<T>, and how you can use that to build your own provider. We take that a step further this time by building an actual provider – albeit a somewhat pointless one, in that it adds LINQ support to something that doesn’t really need it. The point, though, is to keep it simple and try to understand how the process works.

The best way to understand is to take a look at the source code first:

Now, a quick summary of what this is.

We have an interface, INextProvider. It has one method, GetNext that is supposed to get the next one in a sequence of items. An example implementation that uses a simple array as the underlying store is also included. Once you have an instance of INextProvider<T>, say, callednextProvider, you can then extract an IQueryable<T> out of it with this call:

var query = nextProvider.AsQueryable();

You can then use standard LINQ on top of it. Now, I know what you’re thinking: this INextProvider seems uncomfortably similar to IEnumerator – why would we need a query provider for this? We don’t, hence the “pointless” part, but again – the idea is to examine how building a provider works.

The entry point is NextProviderQueryable which implements IQueryable<T> and uses NextProviderQueryProvider as its Provider and returns aNextProviderEnumerator from its GetEnumerator() call. This means that whenever one of the LINQ methods are called on an instance ofNextProviderQueryable, one of the following happens:

  • If the method is something that creates another queryable out of the existing one (e.g. WhereSelectSelectManyCast, etc.),NextProviderQueryProvider.CreateQuery() is called. That, in turn, creates a new instance of NextProviderQueryable, but with the Expression set to what has been passed in. Thus, every call to CreateQuery ends up creating a new queryable with the Expression property representing the complete call.
  • If the method is something that enumerates a queryable (e.g. ToListToArray, etc. or a foreach loop), the GetEnumerator() method is called and enumeration starts. This means that NextProviderEnumerator takes place. This object is initialized with the current value of Expression as of the time of enumeration, thus it has complete information to parse it, figure out what needs to be done, and then do it using the INextProvider that it is assigned. The class ExpressionParser is used to convert the expression into a series of “nodes” that act on each item in the underlyingINextProvider and do the appropriate thing based on what it is (e.g. if it’s a WhereNode, it will have a predicate that it will run on each item).
  • If the method is something that returns a scalar (e.g. AnyAllFirst, etc.), NextProviderQueryProvider.Execute is called. In our case, we simply pass control to NextProviderEnumerator to enumerate as mentioned in the previous point, and then perform the appropriate action. We do this by getting an IEnumerable<T> that uses NextProviderEnumerator as its enumerator (and that is the NextProviderEnumerable class), and then calling the appropriate IEnumerable version of the IQueryable method that has been called. All of this is handled by the ExpressionExecutor class.

As of now, only the following methods are supported: AllAnyCastCountDistinctElementAtElementAtOrDefaultFirstFirstOrDefaultLast,LastOrDefaultLongCountOfTypeSelectSelectManySingleSingleOrDefaultSkipTake and Where. If you try to use any other methods, you will get an exception. Even within these methods, if you try to use a variant that is not supported, you will get an exception.

Next time, we’ll try our hands at a more real world implementation, i.e. a tiny, tiny ORM.

Writing your own LINQ provider, part 2

This is the second in a short series of posts on writing your own LINQ provider. A quick outline of the series:

  1. A primer
  2. Provider basics (this post)
  3. A simple, pointless solution
  4. A tiny ORM of our own

Provider Basics

In the previous post, we took a look at the two flavors of LINQ methods, i.e. the methods and classes around IEnumerable<T> and the methods and classes around IQueryable<T>. In this post, we expand upon what happens when you call LINQ methods on IQueryable<T>, and how you can use that to build your own provider.

Once you have an instance of IQueryable<T>, you can do one of three things with it:

  1. Enumerate it, using one of the following methods:
    1. Call a method such as ToListToArray or ToDictionary on it.
    2. Use it in a foreach loop.
    3. Call GetEnumerator() and then use the enumerator you get in the usual way.
  2. Call a LINQ method that returns a scalar result (this also results in the queryable getting enumerated) such as AnyFirstAllSingle, etc.
  3. Call a LINQ method (such as WhereSelectOrderBy etc.) that returns another IQueryable with some rules added that you can again do one of these very three things with.

For the first situation, IQueryable behaves just like any IEnumerable in that the GetEnumerator() method is called – so this is where you implement what you want to happen when the final enumeration happens. Usually, you do this by writing your own implementation of IEnumerator<T> for this purpose that you return from the GetEnumerator() method.

For the remaining two situations, the Provider and Expression properties of IQueryable<T> come into play. When you implement IQueryable<T>, you need to implement the getter for the Provider property to return an implementation of IQueryProvider<T> that does what you want.

In both cases, here is what the LINQ methods do:

  1. Create a lambda expression that represents the LINQ method call.
  2. Get a reference to the Provider for the target IQueryable<T>.
  3. For the first case, call Execute on the IQueryProvider<T> from step 2. For the second case, call CreateQuery on the ‘IQueryProvider` from step 2.
  4. The only thing that is different across different LINQ methods is the type parameters that are passed in, e.g. Any<T> will call Execute<bool> whileFirst will call Execute<T>. Similarly, Where<T> will call CreateQuery<T> while Select<TSource, TResult> will call CreateQuery<TResult>.

To drive the point home, here is the simplified source code for Where<T>:

public static IQueryable<T> Where<T>(
    this IQueryable<T> source, 
    Expression<Func<T, bool>> predicate)
{
    var currentMethodOpen = (MethodInfo) MethodBase.GetCurrentMethod();
    var currentMethod = currentMethodOpen.MakeGenericMethod(new[] {typeof (T)});
    var callArguments = new[] { source.Expression, Expression.Quote(predicate) };
    var callExpression = Expression.Call(null, currentMethod, callArguments);

    return source.Provider.CreateQuery<T>(callExpression);
}

And here is the simplified source code for Any<T>:

public static bool Any<T>(this IQueryable<T> source, Expression<Func<T, bool>> predicate)
{
    var currentMethodOpen = (MethodInfo) MethodBase.GetCurrentMethod();
    var currentMethod = currentMethodOpen.MakeGenericMethod(new[] {typeof (T)});

    return source.Provider.Execute<bool>(Expression.Call(
        null, currentMethod, new[] {source.Expression}));
}

Note how neither method body does anything specific to what a “Where” or “Any” operation should do. It just wraps that information in an expression and calls the appropriate method on the Provider. It is up to the provider to understand the expression (which is passed in as a parameter to bothCreateQuery and Execute) and perform the correct operation. This is why when you build a LINQ provider, it is up to you to write the translation logic for each LINQ operation as it relates to your data source, or write a fallback that says “this operation is not supported.”

Creating a new LINQ provider, then, can be boiled down to the following steps:

Step 1

Create a class that implements IQueryable<T> (say, YourQueryable<T>).

  1. There should be a way to construct this class and pass in some sort of an interface to the underlying data source to use (e.g. in NHibernate,session.Query<T> on the NHibernate ISession object does this).
  2. The call to GetEnumerator() should return your implementation of IEnumerator<T>, (say, YourEnumerator<T>). It should be initialized with the value of YourQueryable.Expression at the time of the call.
  3. The getter for the Provider property should return an instance of IQueryProvider<T> (say, YourQueryProvider<T>). The provider should have access to the underlying data source interface.

Step 2: Option 1

The logic to parse the final expression can go in YourEnumerator<T>. In this case, YourQueryProvider.CreateQuery simply returns a new instance ofYourQueryable<T> but with the Expression set to what is passed in to CreateQuery. The very first instance of YourQueryable<T> would then set theExpression to Expression.Constant(this). This way, when the time comes to enumerate and you get to YourEnumerator<T>, you have an expression that represents the complete call chain. That is where you then put the hard part of parsing that so that the first call to MoveNext does the right thing against the underlying data source.

Step 2: Option 2

Another option is to have a dumb YourEnumerator<T> and instead have a separate implementation of IQueryable<T> for each type of query operation to support (e.g. WhereQueryable',SelectQueryable’, etc.) In this case, the parsing logic is spread out across these classes, andYourQueryProvider.CreateQuery needs to examine the expression then and there return the correct type of IQueryable<T> with all the necessary information wrapped within. In any case, though, the expression as a whole must be parsed before enumeration happens.

Step 3

YourQueryProvider.Execute then needs to have logic that parses the expression passed in, figures out what needs to be done and return the result. This may involve enumerating the underlying IQueryable<T>. Going back to an ORM that is based on SQL Server, say, you would need to know to generate an WHERE EXISTS clause if you spot an Any in the expression.

Now, granted, all of this sounds pretty convoluted and can be hard to get a grip on without an example. So, we will do just that in the next post. We will start with a simple but pointless solution that does LINQ just for the sake of LINQ. Then, we’ll try to build a rudimentary ORM of our own.

Writing your own LINQ provider, part 1

This is the first in a short series of posts on writing your own LINQ provider. While LINQ is the best thing that ever happened to .NET, and using it is so much fun and makes life so much easier, writing your own LINQ provider is “complicated” to say the least (context- the LINQ interface to NHibernate, RavenDB or Lucene – those are all providers).

A quick outline of the series:

  1. A primer (this post)
  2. Provider basics
  3. A simple, pointless solution
  4. A tiny ORM of our own

A Primer

If you’ve used LINQ, you know there are two distinct syntaxes:

The “query” style:

from item in items
where item.Id == 2
select item.Name

And the “method chaining” style:

items.Where(item => item.Id == 2).Select(item ==> item.Name);

Except for the style, they’re pretty much the same in that the former is really syntactic sugar that compiles down to the latter. Now, the latter, as we know, is a series of extension methods that become available when you import the namespace System.Linq. Within this, though, there are two flavors of LINQ that are very different in terms of their internals:

  • IEnumerable<T> and everything that supports it
  • IQueryable<T> and everything that supports it

This means that when you call the same LINQ methods on an IEnumerable<T> versus an IQueryable<T>, very different things happen.

IEnumerable<T> and everything that supports it

These are simpler to use in that all the work has already been done as part of the .NET Framework. You simply use it. If you want to extend this to a data source of your own, you simply build an enumerator for it (e.g. if you wanted to slap LINQ on top of flat files, you could build a FileEnumerable that uses a FileEnumerator that, in turn, deals with a FileStream).

The extension methods are defined in System.Linq.Enumerable and the way they work is: each method, when called, wraps the IEnumerable it’s called on within a new implementation of IEnumerable that has knowledge of what operation is to be performed. These implementations are all private within theEnumerable class (e.g. Where on an array yields a WhereArrayIterator). When the final enumeration happens, the pipeline executes and gives you the desired result. The scalar-returning methods such as Any and First in this case are simple calls to the enumerator or foreach on top of the underlying enumerable.

All methods in this category deal with Func‘s when it comes to predicates or mapping functions that are passed in.

IQueryable<T> and everything that supports it

This is the focus of this series. You’ll notice that all methods in this category are defined within another class, System.Linq.Queryable and deal not withFunc‘s, but with Expression<Func<>>‘s when it comes to predicates or mapping functions that are passed in. You use this when you are working with a data source that has its own way of extracting data that either does not yield well to the IEnumerable way of doing things, or its own way of extracting data is just better-suited or superior than just enumerating away using IEnumerable. An example is relational databases, where rather than enumerating throw each row in a table and applying predicates or mapping to it, you’re better off running SQL.

The core idea here is to boil the method calls down into a lambda expression tree, then when the time comes to enumerate, parse that expression tree into something the underlying data source understands (using the relational database example, the expression tree needs to be parsed into SQL- that is what ORMs with LINQ providers such as NHibernate or Entity Framework do).

More on this to follow in future posts to come in this series.