After reading this you should know enough to write a pretty good socket server; and hopefully where to start looking when bad things happen. This post is written progressively, meaning I may use certain bad practices in the first few sections. Don’t just go and copy and paste – make sure you read the whole thing.

IOCP (I/O Completion Ports)

IOCP is probably the most important part of a socket server. If you have ever heard of the Berkely socket pattern; this is the exact opposite. Berkely relies on a tight loop that queries each socket asking it if it has any data ready; it doesn’t scale. Another often-used (and incorrect) pattern is the multi-threaded Berkely socket pattern. Instead of a single tight loop querying each socket, the server assigns a thread-per-client. While in theory this is a good idea; realise that spinning up 1000 tight loops is going to bring your server to a screeching halt very quickly.

So what is IOCP? Put simply; it’s a way for a program to say to the kernel “Hey, I am in charge of these streams. When data arrives on ANY of them please resume this thread I have created for you.” The .Net BCL team took IOCP a step further and shield developers from having to manage that thread. Using IOCP in .Net is as easy as implementing your server using the Begin/End async pattern. Using this pattern is quite simple:

1. Create your own Begin* method (as a bad example, BeginReceive).
2. Create a stub for the End* method. This method should accept a single IAsyncResult as a parameter.
3. In your Begin* method, call socket.Begin* passing your End* as a delegate parameter.
4. In your End* method, call socket.End* passing the IAsyncResult you got as a parameter.
5. Then call your Begin* method.

This sets up an async loop; there is a shorter form of this loop (designed for DRY error handling) – which looks like this:

Note that you should use the async pattern for everything: Accept, Receive, Send, Connect. As a side-note – using BeginSend with a null callback is fine (fire and forget); but it will play havoc with the socket performace counters for your process. You may also want to set UseOnlyOverlappedIO to true.

I also suggest you look at Reactive Extensions; as one of the things it deals with is the async pattern.

Use Streams

Don’t use the built-in Receive and Send methods, construct a NetworkStream over the socket. This is important because it allows you wrap your socket in a SslStream or DeflateStream (side note: always wrap the data in compression BEFORE encryption).

Side note: DeflateStream was implemented in a really strange way (the existence of the CompressionMode parameter). You are typically going to write a Stream that uses one DeflateStream (in CompressionMode.Compress) to write and another (in CompressionMode.Decompress) to read.

The Evils of Pinning

The Microsoft CLR uses P/Invoke to provision sockets. A side-effect of P/Invoke is that any reference types (and a Byte Array a.k.a buffer is a reference type) that you pass as parameters are pinned for the duration of the call. Pinning is a process whereby the Garbage Collector is informed that it should not move an object around (the garbage collector moves objects around to ensure that large continuous areas of free space are available). What this means is that if you pass a new buffer to each Send/Receive call you will wind up with a lot of pinned objects, which can lead to fragmentation. There have been stories of processes getting so fragmented that a process using 800MB of memory and 1.2GB of free memory throwing an OutOfMemoryException (because it could not find a large enough space to allocate an object). Remember that an OutOfMemoryException kills your .Net process without ever calling any catch blocks – not good. To further compound the problem, TCP has this problem where it can go into a black-hole state – this means that even though the connection was lost; the operating system (and your process) think it’s still there. A black-hole can last for as long as 15 minutes in certain worst-case scenarios.

The way to get around this is to pre-allocate large blocks of memory for buffers. ArraySegment is a good way to get handles on ‘sub-arrays’. Here is an effective buffer pool to get you started:

Note that you can also use BufferManager (which doesn’t support ArraySegment).

Unfortunately this one is actually really hard to get 100% right in the scenario where you do use SslStream and DeflateStream. These two streams allocate their own buffers; which means even if you do write to the “targetStream” using pooled buffers, unpooled buffers will still be sent to the underlying socket. There is no easy way to put this, if you do run into this scenario you are going to land up writing quite a bit of code (subclassing Stream; placed directly above your NetworkStream) that swaps out the buffers passed to it for pooled buffers (keeping in mind you need to do this async). I have some code that does this, but it is tied into some pretty proprietary stuff at the moment, so I have to isolate it before posting. If you go down this route just remember to use Buffer.BlockCopy and not ArrayCopy as Buffer.BlockCopy is blindingly fast compared to the former (IIRC it uses DMA).

There are also two ways to manage these pooled buffers. If your server behaves in a streaming fashion (which is probably the case) you will want to experiment with allocating a buffer to each client; instead of a buffer to each Read/Write call. Whether this will work out better or not depends on the nature of your server.

DOS

Denial-Of-Service (not that I didn’t mention DDOS; pretty-much the only way to deal with that is horizontal scaling) is one you need to look out for. DOS is really simple to pull off if you don’t protect yourself against it. It merely comes down to asking “why is this client connected to me?”, “can I handle this much data?” and for the love of God staying away from byte arrays (except for, obviously, at the socket level). For instance, what is a client doing connected to you if it has not authenticated itself in 30 seconds? Why is this client sending you 200MB of data? Can the data be written to disk instead?

PUSH Parsing

Make sure that your server can PUSH parse. One form of DOS is to send a really big (but valid) packet. Naive servers will build up a large buffer in memory to attempt to store this packet before parsing it. Don’t do this, instead parse the bytes as you get them. For instance instead of using a StreamReader look at the following:

If you need to build up the packet before parsing it (say it’s XML and you don’t want to write/test your own PUSH parser) rather write it to a temporary file.

Hope This Helps

If you can think of anything I should add or correct please let me know in the comments.

I have a logging system that contains one or more loggers. These logger should NOT be initialized (or ever constructed) for child domains; this is a problem because each AppDomain in .Net recieves it’s own static instances.

Typical Solution

Most people resort to the MSCOREE library and enumerate all the current AppDomains to find the primary AppDomain. After that they can retrieve the singleton instance from within the primary AppDomain.

This has it’s problems, most notably interop ties you to an architecture and requires certain CAS policies. Furthermore, I just don’t like too much interop.

My Solution

My solution involves creating a custom AppDomainManager that is aware of the primary AppDomain. The principle is that the AppDomainManager creates a new AppDomain for primary execution. This new domain uses the custom AppDomainManager (which can then control the creation of child domains; and therefore wire itself into them). The code is self-explanatory (hopefully):

    /// <summary>
    /// Represents a <see cref="AppDomainManager"/> that is
    /// aware of the primary application AppDomain.
    /// </summary>
    public class PrimaryAppDomainManager : AppDomainManager
    {
        private static AppDomain _primaryDomain;

        /// <summary>
        /// Gets the primary domain.
        /// </summary>
        /// <value>The primary domain.</value>
        public static AppDomain PrimaryDomain
        {
            get
            {
                return _primaryDomain;
            }
        }

        /// <summary>
        /// Sets the primary domain.
        /// </summary>
        /// <param name="primaryDomain">The primary domain.</param>
        private void SetPrimaryDomain(AppDomain primaryDomain)
        {
            _primaryDomain = primaryDomain;
        }

        /// <summary>
        /// Sets the primary domain to self.
        /// </summary>
        private void SetPrimaryDomainToSelf()
        {
            _primaryDomain = AppDomain.CurrentDomain;
        }

        /// <summary>
        /// Determines whether this is the primary domain.
        /// </summary>
        /// <value>
        /// 	<see langword="true"/> if this instance is the primary domain; otherwise, <see langword="false"/>.
        /// </value>
        public static bool IsPrimaryDomain
        {
            get
            {
                return _primaryDomain == AppDomain.CurrentDomain;
            }
        }

        /// <summary>
        /// Creates the initial domain.
        /// </summary>
        /// <param name="friendlyName">Name of the friendly.</param>
        /// <param name="securityInfo">The security info.</param>
        /// <param name="appDomainInfo">The AppDomain setup info.</param>
        /// <returns></returns>
        public static AppDomain CreateInitialDomain(string friendlyName, Evidence securityInfo, AppDomainSetup appDomainInfo)
        {
            if (AppDomain.CurrentDomain.DomainManager is PrimaryAppDomainManager)
                return null;

            appDomainInfo = appDomainInfo ?? new AppDomainSetup();
            appDomainInfo.AppDomainManagerAssembly = typeof(PrimaryAppDomainManager).Assembly.FullName;
            appDomainInfo.AppDomainManagerType = typeof(PrimaryAppDomainManager).FullName;

            var appDomain = AppDomainManager.CreateDomainHelper(friendlyName, securityInfo, appDomainInfo);
            ((PrimaryAppDomainManager)appDomain.DomainManager).SetPrimaryDomainToSelf();
            _primaryDomain = appDomain;
            return appDomain;
        }

        /// <summary>
        /// Returns a new or existing application domain.
        /// </summary>
        /// <param name="friendlyName">The friendly name of the domain.</param>
        /// <param name="securityInfo">An object that contains evidence mapped through the security policy to establish a top-of-stack permission set.</param>
        /// <param name="appDomainInfo">An object that contains application domain initialization information.</param>
        /// <returns>A new or existing application domain.</returns>
        /// <PermissionSet>
        /// 	<IPermission class="System.Security.Permissions.SecurityPermission, mscorlib, Version=2.0.3600.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" version="1" Flags="ControlEvidence, ControlAppDomain, Infrastructure"/>
        /// </PermissionSet>
        public override AppDomain CreateDomain(string friendlyName, Evidence securityInfo, AppDomainSetup appDomainInfo)
        {
            appDomainInfo = appDomainInfo ?? new AppDomainSetup();
            appDomainInfo.AppDomainManagerAssembly = typeof(PrimaryAppDomainManager).Assembly.FullName;
            appDomainInfo.AppDomainManagerType = typeof(PrimaryAppDomainManager).FullName;

            var appDomain = base.CreateDomain(friendlyName, securityInfo, appDomainInfo);
            ((PrimaryAppDomainManager)appDomain.DomainManager).SetPrimaryDomain(_primaryDomain);

            return appDomain;
        }
    }

Note that it isn’t possible to use the ‘real’ primary domain for anything; otherwise any domains created by it would be outside of the control of PrimaryAppDomainManager. Using this we can implement cross-domain thread-safe singleton class:

    /// <summary>
    /// Represents a cross-domain singleton.
    /// </summary>
    /// <remarks>
    /// The singleton always lives in the primary domain.
    /// </remarks>
    public abstract class CrossDomainSingleton<TSelf> : MarshalByRefObject
        where TSelf : CrossDomainSingleton<TSelf>, new()
    {
        // Used to alter the primary domain
        // in a thread safe-manner.
        private class Locker : MarshalByRefObject
        {
            private static object _lock = new object();
            public void Lock()
            {
                Monitor.Enter(_lock);
            }

            public void Exit()
            {
                Monitor.Exit(_lock);
            }

            public TSelf GetSelf()
            {
                // At this point we always have a lock;
                // just check that the instance is still null.
                if (_instance == null)
                {
                    _instance = new TSelf();
                }
                return _instance;
            }
        }

        private static TSelf _instance;
        /// <summary>
        /// Gets the instance.
        /// </summary>
        /// <value>The instance.</value>
        public static TSelf Instance
        {
            get
            {
                Contract.Ensures(Contract.Result<TSelf>() != null);
                if (_instance == null)
                {
                    SetInstance();
                }
                return _instance;
            }
        }

        private static void SetInstance()
        {
            Locker locker;
            if (PrimaryAppDomainManager.IsPrimaryDomain)
            {
                // No need for the remoting.
                locker = new Locker();
            }
            else
            {
                // Get the primary domain locker.
                locker = PrimaryAppDomainManager.PrimaryDomain.CreateInstanceAndUnwrap<Locker>();
            }

            try
            {
                locker.Lock();
                _instance = locker.GetSelf();
            }
            finally
            {
                locker.Exit();
            }
        }
    }

You will need the following extension method somewhere:

/// <summary>
/// Creates a new instance of the specified type.
/// </summary>
/// <typeparam name="T">The type of object to create.</typeparam>
/// <param name="appDomain">The app domain.</param>
/// <returns>A proxy for the new object.</returns>
public static T CreateInstanceAndUnwrap<T>(this AppDomain appDomain)
{
    Contract.Requires<ArgumentNullException>(appDomain != null);
    Contract.Ensures(Contract.Result<T>() != null);

    var res = (T)appDomain.CreateInstanceAndUnwrap(typeof(T));
    return res;
}

In order for all of this to work you need to make sure that your ‘real’ primary domain is never used:

    class Program : MarshalByRefObject
    {
        /// <summary>
        /// The main entry point for the application.
        /// </summary>
        static void Main(string[] args)
        {
            new Program().Run(args);
        }

        void Run(string[] args)
        {
            var domain = PrimaryAppDomainManager.CreateInitialDomain("PrimaryDomain", null, null);
            if (domain == null)
            {
                // Original Main() code here.
            }
            else
            {
                domain.CreateInstanceAndUnwrap<Program>().Run(args);
            }
        }
    }

And the singleton classes are simple (thanks to the generic-TSelf trick; which I need to do a post on some time):

    class Foo : CrossDomainSingleton<Foo>
    {
        public Foo()
        {
            // Singleton initializer.
        }
    }

• If you are migrating existing code and worrying only about contracts, switch off background checking. Firstly, this means that the results you get are comprehensive and secondly because of the build time you will think twice about hitting that build key again. Long builds == fewer builds == more time thinking == better code.
• Use code snippets. The ones provided by Microsoft are great! Think about creating your own or changing them. For example cren and cresn no longer throw ArgumentExceptions for me; they are both ArgumentNullExceptions. I also have snippets like create contract class for (cc – scaffolding for ContractClassFor) and contract class implementation (cci – ContractClass attributation).
• Invariants are your friend. Figure out what state a class should really be in and depend on that rather than Contract.Requires everywhere.
• A well placed Contract.Ensures will save you a lot of grief.
• Don’t loose hope. It may seem like it’s daunting at first – but as you start thinking in terms of contracts things become a lot faster. A single well-thought out Contract call will often clear off 5 warnings (for the bigger projects).
• Some classes can be skipped ([ContractVerification(false)]) – especially if they are internal. Don’t gung-ho and try and get everything in contracts; because sometimes they may be ill-suited for the task (and you would by ‘lying’ just to get the stuff to pass). These situations are few and far between; but they are there.
• New development = contracts. Full stop.

Overall I am finding fewer and fewer Contract failures during each build. This is mainly because CodeContracts is getting me into a better and more disciplined mindset. It’s a great tool and I highly recommend it.