Writing a socket server in .Net is one of the most rewarding things you can do in .Net. Unfortunately at the same time there is a lot of pre-requisite knowledge – and the knowledge isn’t easy to find online (in fact it is very easy to examples and tutorials that are blatantly wrong).
After reading this you should know enough to write a pretty good socket server; and hopefully where to start looking when bad things happen. This post is written progressively, meaning I may use certain bad practices in the first few sections. Don’t just go and copy and paste – make sure you read the whole thing.
IOCP (I/O Completion Ports)
IOCP is probably the most important part of a socket server. If you have ever heard of the Berkely socket pattern; this is the exact opposite. Berkely relies on a tight loop that queries each socket asking it if it has any data ready; it doesn’t scale. Another often-used (and incorrect) pattern is the multi-threaded Berkely socket pattern. Instead of a single tight loop querying each socket, the server assigns a thread-per-client. While in theory this is a good idea; realise that spinning up 1000 tight loops is going to bring your server to a screeching halt very quickly.
So what is IOCP? Put simply; it’s a way for a program to say to the kernel “Hey, I am in charge of these streams. When data arrives on ANY of them please resume this thread I have created for you.” The .Net BCL team took IOCP a step further and shield developers from having to manage that thread. Using IOCP in .Net is as easy as implementing your server using the Begin/End async pattern. Using this pattern is quite simple:
- Create your own Begin* method (as a bad example, BeginReceive).
- Create a stub for the End* method. This method should accept a single IAsyncResult as a parameter.
- In your Begin* method, call socket.Begin* passing your End* as a delegate parameter.
- In your End* method, call socket.End* passing the IAsyncResult you got as a parameter.
- Then call your Begin* method.
This sets up an async loop; there is a shorter form of this loop (designed for DRY error handling) – which looks like this:
Note that you should use the async pattern for everything: Accept, Receive, Send, Connect. As a side-note – using BeginSend with a null callback is fine (fire and forget); but it will play havoc with the socket performace counters for your process. You may also want to set UseOnlyOverlappedIO to true.
I also suggest you look at Reactive Extensions; as one of the things it deals with is the async pattern.
Use Streams
Don’t use the built-in Receive and Send methods, construct a NetworkStream over the socket. This is important because it allows you wrap your socket in a SslStream or DeflateStream (side note: always wrap the data in compression BEFORE encryption).
Side note: DeflateStream was implemented in a really strange way (the existence of the CompressionMode parameter). You are typically going to write a Stream that uses one DeflateStream (in CompressionMode.Compress) to write and another (in CompressionMode.Decompress) to read.
The Evils of Pinning
The Microsoft CLR uses P/Invoke to provision sockets. A side-effect of P/Invoke is that any reference types (and a Byte Array a.k.a buffer is a reference type) that you pass as parameters are pinned for the duration of the call. Pinning is a process whereby the Garbage Collector is informed that it should not move an object around (the garbage collector moves objects around to ensure that large continuous areas of free space are available). What this means is that if you pass a new buffer to each Send/Receive call you will wind up with a lot of pinned objects, which can lead to fragmentation. There have been stories of processes getting so fragmented that a process using 800MB of memory and 1.2GB of free memory throwing an OutOfMemoryException (because it could not find a large enough space to allocate an object). Remember that an OutOfMemoryException kills your .Net process without ever calling any catch blocks – not good. To further compound the problem, TCP has this problem where it can go into a black-hole state – this means that even though the connection was lost; the operating system (and your process) think it’s still there. A black-hole can last for as long as 15 minutes in certain worst-case scenarios.
The way to get around this is to pre-allocate large blocks of memory for buffers. ArraySegment is a good way to get handles on ‘sub-arrays’. Here is an effective buffer pool to get you started:
Note that you can also use BufferManager (which doesn’t support ArraySegment).
Unfortunately this one is actually really hard to get 100% right in the scenario where you do use SslStream and DeflateStream. These two streams allocate their own buffers; which means even if you do write to the “targetStream” using pooled buffers, unpooled buffers will still be sent to the underlying socket. There is no easy way to put this, if you do run into this scenario you are going to land up writing quite a bit of code (subclassing Stream; placed directly above your NetworkStream) that swaps out the buffers passed to it for pooled buffers (keeping in mind you need to do this async). I have some code that does this, but it is tied into some pretty proprietary stuff at the moment, so I have to isolate it before posting. If you go down this route just remember to use Buffer.BlockCopy and not ArrayCopy as Buffer.BlockCopy is blindingly fast compared to the former (IIRC it uses DMA).
There are also two ways to manage these pooled buffers. If your server behaves in a streaming fashion (which is probably the case) you will want to experiment with allocating a buffer to each client; instead of a buffer to each Read/Write call. Whether this will work out better or not depends on the nature of your server.
DOS
Denial-Of-Service (not that I didn’t mention DDOS; pretty-much the only way to deal with that is horizontal scaling) is one you need to look out for. DOS is really simple to pull off if you don’t protect yourself against it. It merely comes down to asking “why is this client connected to me?”, “can I handle this much data?” and for the love of God staying away from byte arrays (except for, obviously, at the socket level). For instance, what is a client doing connected to you if it has not authenticated itself in 30 seconds? Why is this client sending you 200MB of data? Can the data be written to disk instead?
PUSH Parsing
Make sure that your server can PUSH parse. One form of DOS is to send a really big (but valid) packet. Naive servers will build up a large buffer in memory to attempt to store this packet before parsing it. Don’t do this, instead parse the bytes as you get them. For instance instead of using a StreamReader look at the following:
If you need to build up the packet before parsing it (say it’s XML and you don’t want to write/test your own PUSH parser) rather write it to a temporary file.
Hope This Helps
If you can think of anything I should add or correct please let me know in the comments.





