Introduction

Whether you're interested in making an FPS or RTS, you've probably heard that you should use UDP. It's probably because of its speed. Using TCP with the TCP_NODELAY option (which means it doesn't wait for enough data to be buffered before sending) may not be enough as TCP also does congestion control and you may want to do voice chat, which is better done using a lossy protocol like UDP. TCP doesn't allow you to adjust the “sliding window”, which means you might not reach the full speed of the communication channel if it has a high delay (search for “bandwidth-delay factor” for more information). If one packet doesn't arrive in TCP and is lost, TCP stops all traffic flow until it arrives, resulting in pauses. The packet header in TCP is also 20 bytes, as opposed to 6 bytes in UDP plus a few for reliability. Combining TCP and UDP is not an option as one induces packet loss in the other. So you should use UDP, but how do you guarantee packets are delivered and in the order they were sent in? Also, if you're making an RTS, how do you make sure that clients are running exactly the same simulation? A small change can cause a butterfly effect and be the difference between one player winning and losing. For this you need reliable UDP (RUDP) and lockstep. In addition, you probably want to use the latest client-server methodology, which allows the game to not be held back by the slowest player, as is done in the older peer-to-peer model. Along the way we'll cover parity bit checking to ensure packet correctness and integrity. We'll also cover LAN networking and setting up a matchmaking server.

Note:  This free article version covers only the UDP implementation. If you are interested in the full-breadth of the topic discussed above, you can purchase the complete 70-page PDF document on the GDNet Marketplace.

Library

The library that I use for networking is SDL2_net. This is just abstraction of networking to allow us to deploy to different platforms like Windows, Linux, iPhone, and Android. If you want, you can use WinSock or the native networking functions of Linux. They're all pretty much the same, with a few differences in initialization and function names. But I would recommand SDL2_net as that is cross-platform. Watch for a possible article on setting up and compiling libraries for details on how to set up SDL2_net for use in your projects if you aren't able to do this yourself. By the way, if you're doing voice chat, try PortAudio for getting microphone input on Windows, Mac, and Linux. For transmitting you'd need Speex or the newer Opus speech codec for encoding the data.

Packet Header

So let's jump in. The way you will keep track of the order packets are sent in and also guarantee delivery is by using an ack or sequence number. We will make a header that will be at the beginning of each packet we send. We also need to know what kind of packet this is.

struct PacketHeader
{
	unsigned short type;
	unsigned short ack;
};

To make sure the compiler packs the packet data as tightly as possible, to reduce network data usage, we can remove byte padding by putting this around all of our packet definitions.

// byte-align structures
#pragma pack(push, 1)

//.. packets go here ÃÃÃâÃÃÃæ

// default alignment
#pragma pack(pop)

By the way, they're not really packets; packets are what we use in TCP, but the subset in UDP is called a datagram. UDP stands for “user datagram protocol”. But I call them packets.

Control/Protocol Packets

Let's define some control/protocol packets that will be part of our reliable UDP protocol.

#define PACKET_NULL						0
#define PACKET_DISCONNECT				1
#define PACKET_CONNECT					2
#define PACKET_ACKNOWLEDGMENT			3
#define PACKET_NOCONN					4
#define PACKET_KEEPALIVE					5
#define PACKET_NACK						6

struct BasePacket
{
	PacketHeader header;
};

typedef BasePacket NoConnectionPacket;
typedef BasePacket AckPacket;
typedef BasePacket KeepAlivePacket;

struct ConnectPacket
{
	PacketHeader header;
	bool reconnect;
	unsigned short yourlastrecvack;
	unsigned short yournextrecvack;
	unsigned short yourlastsendack;
};

struct DisconnectPacket
{
	PacketHeader header;
};

We will send a ConnectPacket to establish a new connection to a computer that is listening on a certain port. If you're behind a router and have several computers behind it, your router doesn't know which computer to forward an incoming connection to unless that computer has already sent data from that port outside. This is why a matchmaking server is needed, unless playing on LAN. More will be covered later.

We will send a DisconnectPacket to end a connection. We need to send an AckPacket (acknowledgment) whenever we receive a packet type that is supposed to be reliable (that needs to arrive and be processed in order and cannot be lost). If the other side doesn't receive the ack, it will keep resending until it gets one or times out and assumes the connection to be lost. This is called “selective repeat” reliable UDP.

Occasionally, we might not need to send anything for a long time but tell the other side to keep our connection alive, for example if we've connected to a matchmaking server and want to tell it to keep our game in the list. When this happens, we need to send a KeepAlivePacket.

Almost all of these packets are reliable, as defined in the second paragraph on this page, except for the AckPacket and NoConnectionPacket. When we're first establishing a connection, we will use the ack number to set as the start of our sequence. When there's an interruption and the other side has dropped us due to timeout, but we still have a connection to them, they will send a NoConnectionPacket, which is not reliable, but is sent every time we receive a packet from an unknown source (whose address we don't recognize as having established a connection to). Whenever this happens, we have to do a full reconnect as the sequence/ack numbers can't be recovered. This is important because we will have a list of packets that we didn't get a reply to that we will resend that need to be acknowledged. You will understand more as we talk about how ack/sequence numbers work.

Lastly, the AckPacket's ack member is used to tell the other side which packet we're acknowledging. So an ack packet is not reliable, and is sent on an as-needed basis, when we receive a packet from the other side.

User Control/Protocol Packets

The rest of the packet types are user-defined (that's you) and depend on the type of application needed. If you're making a strategy game, you'll have a packet type to place a building, or to order units around. But there are also some control/protocol packets needed that are not part of the core protocol. These are packets for joining game sessions, getting game host information, getting a list of game hosts, informing a game client that the game room is full or that his version is older. Here are some suggested packet types for a multiplayer RTS or any kind of strategy game, but may also apply to FPS. These are used after a connection has been established.

#define PACKET_JOIN						7
#define PACKET_ADDSV					8
#define PACKET_ADDEDSV					9
#define PACKET_GETSVLIST					10
#define PACKET_SVADDR					11
#define PACKET_SVINFO					12
#define PACKET_GETSVINFO					13
#define PACKET_SENDNEXTHOST				14
#define PACKET_NOMOREHOSTS				15
#define PACKET_ADDCLIENT					16
#define PACKET_SELFCLIENT				17
#define PACKET_SETCLNAME				18
#define PACKET_CLIENTLEFT				19
#define PACKET_CLIENTROLE				20
#define PACKET_DONEJOIN					21
#define PACKET_TOOMANYCL				22
#define PACKET_MAPCHANGE				23
#define PACKET_CLDISCONNECTED			24
#define PACKET_CLSTATE					25
#define PACKET_CHAT						26
#define PACKET_MAPSTART					27
#define PACKET_GAMESTARTED				28
#define PACKET_WRONGVERSION				29
#define PACKET_LANCALL					30
#define PACKET_LANANSWER				31

Lockstep Control Packets

Once a game has been joined and started, RTS games also need to control the simulation to keep it in sync on all sides. A class of packet types that control the lockstep protocol we will call the lockstep control packets.

#define PACKET_NETTURN					29
#define PACKET_DONETURN					30

User Command / In-Game Packets

Any user command or input that effects the simulation needs to be bundled up inside a NetTurnPacket by the host and sent to all the clients as part of lockstep. But first it is sent to the host on its own. More on this later. Here are some example packets I use.

#define PACKET_PLACEBL					32
#define PACKET_CHVAL					33
#define PACKET_ORDERMAN					34
#define PACKET_MOVEORDER				35
#define PACKET_PLACECD					36

Initialization

You should really read some C/C++ UDP or SDL2_net UDP example code before you try to proceed with this, and the basics of that are not what I'm covering here. But nevertheless, I will mention that you need to initialize before you use SDL2_net or WinSock.

	if(SDLNet_Init() == -1) 
	{ 
		char msg[1280]; 
		sprintf(msg, "SDLNet_Init: %s\n", SDLNet_GetError());
		SDL_ShowSimpleMessageBox(SDL_MESSAGEBOX_ERROR, "Error", msg, NULL);
	}

Here are some examples for low-level (not using SDL) UDP networking:

https://www.cs.rutgers.edu/~pxk/417/notes/sockets/udp.html
http://www.binarytides.com/programming-udp-sockets-c-linux/
http://www.codeproject.com/Articles/11740/A-simple-UDP-time-server-and-client-for-beginners

They all cover the same material. You just need to know the basics of how to intialize, send data, and receive data using UDP (not TCP).

Net Update Loop

So somewhere in your frame loop, where you update your game state and render everything, you will add a call to UpdNet, which will process received packets. The skeleton of it will look roughly like this.

//Net input 
void UpdNet() 
{ 
	int bytes; 

	UDPpacket *in; 

	UDPsocket* sock = &g_sock; 

	if(!sock) 
		return; 

	in = SDLNet_AllocPacket(65535); 

	do 
	{ 
		in->data[0] = 0; 

		bytes = SDLNet_UDP_Recv(*sock, in); 

		IPaddress ip; 

		memcpy(&ip, &in->address, sizeof(IPaddress)); 

		if(bytes > 0) 
			TranslatePacket((char*)in->data, bytes, true, &g_sock, &ip); 
	} while(bytes > 0); 

	SDLNet_FreePacket(in); 
}

Following, inside the do-while loop, is the equivalent code in regular Berkeley sockets using recvfrom, in case you're not using SDL2_net:

//Net input 
void UpdNet() 
{ 
	int bytes; 

	int* sock = &g_sock; 

	if(!sock) 
		return; 

	do 
	{
		struct sockaddr_in from; 
		socklen_t fromlen = sizeof(struct sockaddr_in); 
		char buffer[65535]; 
		bytes = recvfrom(g_socket, buffer, 65535, 0, (struct addr *)&from, &fromlen); 

		if(bytes > 0) 
			TranslatePacket(buffer, bytes, true, &g_sock, &ip); 
	} while(bytes > 0); 
}

This basically loops while we still have data to process in the buffer. If we do, we send it to the TranslatePacket function, which takes as parameters:

• the buffer of data (“buffer”)
• the number of bytes received (“bytes”)
• whether we want to check the acknowledgement/sequence number and process it in order (“checkprev”), as we might choose to send a dummy packet that we stored to reuse our command switch functionality for lockstep batch packets, or whether we want to process it right away
• the socket (“sock”)
• and the IP address and port it came from (“from”).

void TranslatePacket(char* buffer, int bytes, bool checkprev, UDPsocket* sock, IPaddress* from)

We will get to TranslatePacket next, but first take a look at these function calls we will also have at the end of the UpdNet function.

	KeepAlive(); 
	CheckConns(); 
	ResendPacks(); 

#ifndef MATCHMAKER 
	CheckAddSv(); 
	CheckGetSvs(); 
#else 
	SendSvs(); 
#endif 

KeepAlive() tries to keep connections alive that are about to time out. CheckConns() checks for connections that we've closed or that have timed out from unresponsiveness and recycles them. ResendPacks() tries to resend packets that we haven't received an acknowledgement for, once they've waited long enough.

Then we have a preprocessor check to see whether we're compiling for the matchmaking server or the client program/app. If we're a game client or host, we only care about CheckAddSv() and CheckGetSv().

CheckAddSv() checks whether our game host address is up on the matchmaker list and getting the information on each one of the received hosts. It also removes hosts from our list that we have lost a connection too (due to time out).

CheckGetSv() makes sure the matchmaker will send us the next host from its list if we've requested to get a host list.

If we're the matchmaker however, we only care about SendSvs(), which sends the next host in the list for each requesting client when the last one has been ack'd.

Connection Class

Because there's no concept of a connection in UDP, we need to make it ourselves.

class NetConn
{
public:
	unsigned short nextsendack;
	unsigned short lastrecvack;
	bool handshook;
	IPaddress addr;
	//TODO change these to flags
	bool isclient;	//is this a hosted game's client? or for MATCHMAKER, is this somebody requesting sv list?
	bool isourhost;	//is this the currently joined game's host? cannot be a host from a server list or something. for MATCHMAKER, it can be a host getting added to sv list.
	bool ismatch;	//matchmaker?
	bool ishostinfo;	//is this a host we're just getting info from for our sv list?
	//bool isunresponsive;
	unsigned long long lastsent;
	unsigned long long lastrecv;
	short client;
	float ping;
	bool closed;
	bool disconnecting;

	void expirein(int millis);

#ifdef MATCHMAKER
	int svlistoff;	//offset in server list, sending a few at a time
	SendSvInfo svinfo;
#endif
	//void (*chcallback)(NetConn* nc, bool success);	//connection state change callback - did we connect successfully or time out?

	NetConn()
	{
		client = -1;
		handshook = false;
		nextsendack = 0;
		//important - reply ConnectPacket with ack=0 will be
		//ignored as copy (even though it is original) if new NetConn's lastrecvack=0.
		lastrecvack = USHRT_MAX;
		isclient = false;
		isourhost = false;
		ismatch = false;
		ishostinfo = false;
		//isunresponsive = false;
		lastrecv = GetTicks();
		lastsent = GetTicks();
		//chcallback = NULL;
#ifdef MATCHMAKER
		svlistoff = -1;
#endif
		ping = 1;
		closed = false;
	}
};

“nextsendack” is the outgoing sequence number that the next reliable packet will have. We increment it by one each time and it wraps around from 0 when it maxes out.

“lastrecvack” is the inbound sequence number of the last received reliable packet (the next one will be greater until it wraps around). Because we can send and receive independently, we keep two acks/sequences.

When we start a connection, “nextsendack” is 0 (for the first packet sent, the ConnectPacket), and “lastrecvack” is 65535, which is the maximum unsigned short value, before it wraps around to 0. Although nextsendack can be set to anything (and should) as long as we set the ack of the first ConnectPacket to that, and that is more secure, as it is harder to predict or crack and probably protects data better (but I haven't tried it).

“handshook” tells us whether the other side has acknowledged the ConnectPacket (and therefore created an accompanying NetConn connection instance for us). When we receive a ConnectPacket, we set “handshook” to true and acknowledge them, recording the inbound ack and setting “nextsendack” to 0. If “handshook” is true, it tells us that we can now send reliable packets on the connection to this address, as the sequence numbers are in place.

IPaddress is the SDL2_net IP address and port structure, the equivalent of sockaddr_in in regular Berkeley sockets.

Translating Packets

This is what TranslatePacket does. When we “translate” or process the packet we want to know if it's an old packet that we've already processed once or if it's ahead of the next expected packet, if it's meant to be reliable as defined previously. We also acknowledge any packets that are reliable. And then finally we execute them. The game host does an extra part in beginning of the function, checking if any client connections are unresponsive or have become responsive again and relays that information to other clients, so the blame can be pinned on the lagger.

The TranslatePacket() function has this basic outline:

1. Match address to a connection
2. Update last received timestamp if match found
3. Check packet type if we need to check sequence number or if we process it right away
4. Check sequence number for one of three cases (behind, current, or future)
5. Acknowledge packet if needed
6. If we don't recognize the connection and it's supposed to be reliable, tell the other side that we don't have a connection with them
7. Execute the packet
8. Execute any buffered packets after the current one (in order)
9. And update the last received sequence number to the last packet executed

Step by step, the function is:

void TranslatePacket(char* buffer, int bytes, bool checkprev, UDPsocket* sock, IPaddress* from)
{
	//1. Match address to a connection
	PacketHeader* header = (PacketHeader*)buffer;

	NetConn* nc = Match(from);

We pass an IPaddress struct pointer to Match which returns the matching connection or NULL on failure.

Then, if we got a match, we update the last received time for the connection. If the connection is associated with client in the game room, and that client was previously unresponsive, we can mark it as responsive again and tell the other clients.

	//If we recognize this connection...
	if(nc)
	{
		//2. Update the timestamp of the last received packet
		nc->lastrecv = GetTicks();

#ifndef MATCHMAKER
		//check if was previously unresponsive
		//and if (s)he was, tell others that (s)he
		//is now responsive.
		if(nc->client >= 0)
		{
			Client* c = &g_client[nc->client];

			//was this client unresponsive?
			if(c->unresp)
			{
				//it's now responsive again.
				c->unresp = false;

				//if we're the game host
				if(g_netmode == NETM_HOST)
				{
					//inform others
					ClStatePacket csp;
					csp.header.type = PACKET_CLSTATE;
					csp.chtype = CLCH_RESP;
					csp.client = nc->client;

					//send to all except the original client (nc->addr)
					SendAll((char*)&csp, sizeof(ClStatePacket), true, false, &nc->addr);
				}
			}
		}
#endif
	}

We know that certain packets are meant to be processed right away, without checking for them to be processed in sequence. For example, acknowedgement packets are non-reliable and don't need to be processed in a specific order. Connect packets are to be executed as soon as they are received, because no other packet is supposed to be sent with them. Same for disconnect packets, “no connection” packets, LAN call, and LAN answer.

	//3. Check packet type if we need to check sequence number or if we process it right away
	//control packets
	//don't check sequence for these ones and process them straight away
	//but acknowledge CONNECT and DISCONNECT
	switch(header->type)
	{
	case PACKET_ACKNOWLEDGMENT:
	case PACKET_CONNECT:	//need to send back ack
	case PACKET_DISCONNECT:	//need to send back ack
	case PACKET_NOCONN:
	case PACKET_NACK:
	case PACKET_LANCALL:
	case PACKET_LANANSWER:
		checkprev = false;
		break;
	default:
		break;
	}

If it's not one of those packet types, checkprev=true, and we check the sequence number. These are reliable packets that must be processed in the order they were sent in. If we're missing a packet in the sequence, we will buffer the packets after it while we wait for the missing packet to arrive.

“next” will be the next expected sequence number (the one after lastrecvack in NetConn). “last” will be updated each time we execute a packet, to update lastrecvack with the last one.

	unsigned short next;	//next expected packet ack
	unsigned short last = PrevAck(header->ack);	//last packet ack to be executed

	//4. Check sequence number for one of three cases (behind, current, or future)

	//If checkprev was set (directly above), we need to check the sequence.
	//It must be a recognized NetConn; otherwise we don't have any sequence numbers.
	if(checkprev && nc != NULL)
	{
		// ÃÃÃâÃÃÃæ check sequence number (check snippet further down) ...
	}

Then we acknowledge the packet if it's meant to be reliable. Acknowledgement packets don't need acknowledgements themselves. A “no connection” packet tells us the other side doesn't even have sequence numbers for us, so there's no point acknowledging it. Usually, if checkprev=false, we don't check the packet sequence so we don't care about acknowledging it, but for connect and disconnect packets we must acknowledge because the other side expects a success signal back.

	//5. Acknowledge packet if needed

procpack:

	//We might disconnect further down in PacketSwitch()
	//So acknowledge packets while we still have the sequence numbers
	nc = Match(from);
	//Don't acknowledge NoConn packets as they are non-reliable,
	//and ack'ing them would cause a non-ending ack loop.
	if(header->type != PACKET_ACKNOWLEDGMENT &&
		header->type != PACKET_NOCONN &&
		sock && nc)
	{
		Acknowledge(header->ack, nc, from, sock, buffer, bytes);
	}
	//Always acknowledge ConnectPacket's
	else if( header->type == PACKET_CONNECT &&
		sock )
	{
		Acknowledge(header->ack, NULL, from, sock, buffer, bytes);
 	}
	//And acknowledge DisconnectPacket's
	else if(header->type == PACKET_DISCONNECT && sock)
	{
		Acknowledge(header->ack, NULL, from, sock, buffer, bytes);
	}

If we got disconnected from the other side and for some reason they retained the connection, we'll get packets that we have to tell the other side we can't process. They can then show an error to the user or try to reconnect.

	//6. If we don't recognize the connection and it's supposed to be reliable, tell the other side that we don't have a connection with them

	//We're getting an anonymous packet.
	//Maybe we've timed out and they still have a connection.
	//Tell them we don't have a connection.
	//We check if sock is set to make sure this isn't a local
	//command packet being executed.
	if(!nc &&
	header->type != PACKET_CONNECT &&
	header->type != PACKET_NOCONN &&
	header->type != PACKET_LANCALL &&
	header->type != PACKET_LANANSWER &&
	sock)
	{
		NoConnectionPacket ncp;
		ncp.header.type = PACKET_NOCONN;
		SendData((char*)&ncp, sizeof(NoConnectionPacket), from, false, true, NULL, &g_sock, 0, NULL);
		return;
	}

Then we execute packets. First, any packets before the current received one. Then the one we just received. And then any that we buffered that come after it.

The reason we execute packets that came BEFORE is because we may have a case like this:

packet 1 received
packet 2 received
packet 5 received

We'll be able to execute packets 1 and 2 even though the current is 5.

updinack:

	//7. Execute the packet
	//8. Execute any buffered packets after the current one (in order)

	// Translate in order
	if(checkprev && nc)
	{
		last = PrevAck(header->ack);
		last = ParseRecieved(next, last, nc);
	}

	// Translate in order
	if(NextAck(last) == header->ack ||
		!checkprev)
	{
		PacketSwitch(header->type, buffer, bytes, nc, from, sock);
		last = header->ack;
	}

	// Translate in order
	if(checkprev && nc)
	{
		while(true)
		{
			if(!Recieved(last+1, last+1, nc))
				break;

			last++;
			ParseRecieved(last, last, nc);
		}
	}

Finally, we update the received sequence number. We have to match up the connection pointer with the address again, because the instance it was pointing to might have been erased, or it might have appeared when it wasn't previously, as a connection is erased or created respectively.

For non-reliable packets we don't update the sequence number. For connect or disconnect packets, we only set the sequence number inside the PacketSwitch read function call when we create a connection.

	//9. And update the last received sequence number to the last packet executed

	//have to do this again because PacketSwitch might
	//read a ConnectPacket, which adds new connections.
	//also connection might have
	//been Disconnected(); and erased.
	nc = Match(from);

	//ack Connect packets after new NetConn added...
	//Don't acknowledge NoConn packets as they are non-reliable
	if(header->type != PACKET_ACKNOWLEDGMENT &&
		header->type != PACKET_NOCONN &&
		sock && nc && checkprev)
	{
		if(header->type != PACKET_CONNECT &&
			header->type != PACKET_DISCONNECT)
			nc->lastrecvack = last;
	}
}

The PacketSwitch() at the end is what executes the packet. It might better be called ExecPacket().

The Match() function at the top compares the “addr” port and IP address integers to every known connection and returns the match, or NULL on failure.

NetConn* Match(IPaddress* addr)
{
	if(!addr)
		return NULL;

	for(auto ci=g_conn.begin(); ci!=g_conn.end(); ci++)
		if(Same(&ci->addr, addr))
			return &*ci;

	return NULL;
}

bool Same(IPaddress* a, IPaddress* b)
{
	if(a->host != b->host)
		return false;

	if(a->port != b->port)
		return false;

	return true;
}

The packet is “old” if we've already buffered it (but it's ahead of the next expected ack/sequence number that we processed), or if its ack/sequence is behind our connection class's “lastrecvack”. We use an unsigned short for the sequence number, which holds a maximum value of 65535. Because we might exceed this value after 36 minutes if we send 30 packets a second, we wrap around and thus, there's a “sliding window” of values that are considered to be in the past (don't confuse this with the “sliding window” packet range that might be being reliably resent at any given moment). We can check if an ack is in the past (behind what is already executed) using PastAck():

bool PastAck(unsigned short test, unsigned short current)
{
	return ((current >= test) && (current - test <= USHRT_MAX/2))
	       || ((test > current) && (test - current > USHRT_MAX/2));
}

Where PastAck tests whether “test” is behind or at “current”.

Let's look in more detail at the part in the middle of TranslatePacket that checks the sequence number.

We define some variables. “next” will hold the current expected ack (lastrecvack+1) inside the following code block. “last” will hold the last packet to have been executed. For now, it's set to something, but it doesn't matter, as we update it at the end.

	unsigned short next;	//next expected packet ack
	unsigned short last = PrevAck(header->ack);	//last packet ack to be executed

We only check the sequence numbers if it's a packet that makes checkprev=true and if it's from a recognized connection.

	if(checkprev && nc != NULL)
	{

We set the “next” expected packet number.

		next = NextAck(nc->lastrecvack);	//next expected packet ack
		last = next;	//last packet ack to be executed

Next, we check how the received packet's sequence number compares to the next expected one.

		//CASE #1: ÃÃÃâoldÃÃÃâ packet
		if(PastAck(header->ack, nc->lastrecvack) || Recieved(header->ack, header->ack, nc))
		{
			Acknowledge(header->ack, nc, from, sock, buffer, bytes);
			return;
		}

		//CASE #2: current packet (the next expected packet)
		if(header->ack == next) 
		{
			// Translate packet
			last = next;
		} 

		//CASE #3: an unbuffered, future packet
		else  // More than +1 after lastrecvack?
		{
			/*
			last will be updated to the last executed packet at the end.
			for now it will hold the last buffered packet to be executed.
			*/
			unsigned short checklast = PrevAck(header->ack);

			if(Recieved(next, checklast, nc))
			{
				// Translate in order
				last = checklast;
				goto procpack;
			}
			else
			{
				AddRecieved(buffer, bytes, nc);

				if(Recieved(next, checklast, nc))
				{
					// Translate in order
					last = checklast;
					goto procpack;
				}
				else
				{
					//TODO
					//how to find which ack was missed, have to go through all buffered
					//this is something somebody smart can do in the future
					//NAckPacket nap;
					//nap.header.type = PACKET_NACK;
					//nap.header.ack =
				}
			}
		}
	}

As can be seen, there are three possible cases for the inbound packet's sequence number: it is either, 1.) behind or buffered, 2.) current expected, or 3.) future unbuffered.

Case 1: behind and buffered received packets

If we've already dealt with (executed) the packet, we simply acknowledge it again and return from TranslatePacket() with no further action.

		if(PastAck(header->ack, nc->lastrecvack) || Recieved(header->ack, header->ack, nc))
		{
			Acknowledge(header->ack, nc, from, sock, buffer, bytes);
			return;
		}

In the the second testcase of the if statement (packet is buffered received), we check if we've already buffered it, using Recieved():

//check when we've recieved a packet range [first,last] inclusive
bool Recieved(unsigned short first, unsigned short last, NetConn* nc)
{
	OldPacket* p;
	PacketHeader* header;
	unsigned short current = first;
	unsigned short afterlast = NextAck(last);
	bool missed;

	//go through all the received packets and check if we have the complete range [first,last]
	do
	{
		//for each number in the sequence...
		missed = true;

		//look through each packet from that address
		for(auto i=g_recv.begin(); i!=g_recv.end(); i++)
		{
			p = &*i;
			header = (PacketHeader*)&p->buffer;

			//is this the sequence number we're looking for?
			if(header->ack != current)
				continue;

			//is this the correct address?
			if(!Same(&p->addr, &nc->addr))
				continue;

			//go to next number in the sequence now that we know we have the previous one
			current = NextAck(current);
			missed = false;
			break;
		}

		//if we finished the inner loop and ÃÃÃâmissedÃÃÃâ is still false, we missed a number in the sequence, so return false
		if(missed)
			return false;

	//continue looping until we've arrived at the number after the ÃÃÃâlastÃÃÃâ number
	} while(current != afterlast);

	//if we got here, we got all the numbers
	return true;
}

“g_recv” is a linked list of OldPacket's. We go through each sequence number between “first” and “last” and check if we have each and every one. Because we use the received packet's ack number for both parameters in Case 1, we only check if we've buffered that one packet. Because g_recv holds inbound packets from every address we're connected to, we have to check to match the address when comparing ack numbers. You can store g_recv in the NetConn's and this might be more efficient.

Buffered Packets

The OldPacket class holds the byte array for the packet and the address and port of the sender (or the outbound port and address for outgoing buffered packets).

class OldPacket
{
public:
	char* buffer;
	int len;
	unsigned long long last;	//last time resent
	unsigned long long first;	//first time sent
	bool expires;
	bool acked;	//used for outgoing packets

	//sender/reciever
	IPaddress addr;
	void (*onackfunc)(OldPacket* op, NetConn* nc);

	void freemem()
	{
		if(len <= 0)
			return;

		if(buffer != NULL)
			delete [] buffer;
		buffer = NULL;
	}

	OldPacket()
	{
		len = 0;
		buffer = NULL;
		onackfunc = NULL;
		acked = false;
	}
	~OldPacket()
	{
		freemem();
	}

	OldPacket(const OldPacket& original)
	{
		len = 0;
		buffer = NULL;
		*this = original;
	}

	OldPacket& operator=(const OldPacket &original)
	{
		freemem();

		if(original.buffer && original.len > 0)
		{
			len = original.len;
			if(len > 0)
			{
				buffer = new char[len];
				memcpy((void*)buffer, (void*)original.buffer, len);
			}
			last = original.last;
			first = original.first;
			expires = original.expires;
			acked = original.acked;
			addr = original.addr;
			onackfunc = original.onackfunc;
		}
		else
		{
			buffer = NULL;
			len = 0;
			onackfunc = NULL;
		}

		return *this;
	}
};

It has some extra fields for outbound packets.

Case 2: current expected received packets

The second case is when the received packet is the next expected one, which means we received it in the correct order without repeats. The next expected (current) packet is the one after the “last received” one (lastrecvack). The variable “next” here will hold that ack. It is equal to nc->lastrecvack + 1, so you can use that instead of the function “NextAck”.

		next = NextAck(nc->lastrecvack);	//next expected packet ack
		last = next;	//last packet ack to be executed
	
		//CASE #2: current packet (the next expected packet)
		if(header->ack == next) 
		{
			// Translate packet
			last = next;
		} 

If it matches “next” we will process the packet and acknowledge it further down. We record the “last” packet executed, to update the sequence number.

Case #3: future, unbuffered received packets

If we reach “else” it means we have an unbuffered, future packet.

		//CASE #3: an unbuffered, future packet
		else  // More than +1 after lastrecvack?
		{
			/*
			last will be updated to the last executed packet at the end.
			for now it will hold the last buffered packet to be executed.
			*/
			unsigned short checklast = PrevAck(header->ack);

			if(Recieved(next, checklast, nc))
			{
				// Translate in order
				last = checklast;
				goto procpack;
			}
			else
			{
				AddRecieved(buffer, bytes, nc);

				if(Recieved(next, checklast, nc))
				{
					// Translate in order
					last = checklast;
					goto procpack;
				}
				else
				{
					//TODO
					//how to find which ack was missed, have to go through all buffered
					//this is something somebody smart can do in the future
					//NAckPacket nap;
					//nap.header.type = PACKET_NACK;
					//nap.header.ack =
				}
			}
		}

We check if we have a range of buffered packets up to this one. If we have a complete range, starting from the current (expected next) packet, we can execute them (because we only run them in the order they're sent in) and increase lastrecvack to equal “last”. We move up lastrecvack at the end of TranslatePacket. We might have more buffered packets after the received one. That is why we check for any extra packets and store the last executed one's ack number in “last”.

If we don't have a complete set of packets up to the received one, we call AddRecieved (buffer it).

void AddRecieved(char* buffer, int len, NetConn* nc)
{
	OldPacket p;
	p.addr = nc->addr;
	p.buffer = new char[ len ];
	p.len = len;
	memcpy((void*)p.buffer, (void*)buffer, len);
	memcpy((void*)&p.addr, (void*)&nc->addr, sizeof(IPaddress));

	g_recv.push_back(p);
}

If we have to buffer it, it means it's ahead of the last executed packet, and there's one missing before it.

If we wanted to only send selective repeats every second or so (if that was the delay on the channel and we didn't want to send some three copies of it before we received back an ack, and we're sure that loss of packets is minimal, and we'd rather leave the “sliding window” huge), we could use NAck's (negative ack's) to tell us when we've missed a packet. But selective repeat works pretty well. (Using nacks is a different kind of reliable UDP implementation.)

Acknowledgements

Further on we send ack's.

void Acknowledge(unsigned short ack, NetConn* nc, IPaddress* addr, UDPsocket* sock, char* buffer, int bytes)
{
	AckPacket p;
	p.header.type = PACKET_ACKNOWLEDGMENT;
	p.header.ack = ack;

	SendData((char*)&p, sizeof(AckPacket), addr, false, true, nc, sock, 0, NULL);
}

We use a SendData function for our RUDP implementation, shown and explained further down.

Whenever we send data, we have to fill out a packet struct for that type of packet. At minimum, we have to set header.type so that the received end can know what packet type it is from reading the first 2 bytes of the packet.

Executing Packet and Updating Sequence Number

If we get to this point in TranslatePacket, we'll execute the packets in order. If we checked sequence numbers, and we have a connection, we'll execute the buffered previous packets, then the current received packets, then check for any future buffered packets. If we don't check the sequence, or don't have a connection, we just execute the one packet we received.

updinack:
	
	// Translate in order
	if(checkprev && nc)
	{
		last = header->ack;
		last = ParseRecieved(next, last, nc);
	}
	
	// Translate in order
	if(NextAck(last) == header->ack ||
		!checkprev)
	{
		PacketSwitch(header->type, buffer, bytes, nc, from, sock);
		last = header->ack;
	}
	
	// Translate in order
	if(checkprev && nc)
	{
		last = header->ack;

		while(true)
		{
			if(!Recieved(last+1, last+1, nc))
				break;

			last++;
			ParseRecieved(last, last, nc);
		}
	}

	//have to do this again because PacketSwitch might
	//read a ConnectPacket, which adds new connections.
	//but also the connection might have
	//been Disconnected(); and erased.
	nc = Match(from);

	//ack Connect packets after new NetConn added...
	//Don't acknowledge NoConn packets as they are non-reliable
	if(header->type != PACKET_ACKNOWLEDGMENT &&
		header->type != PACKET_NOCONN &&
		sock && nc && checkprev)
	{
		if(header->type != PACKET_CONNECT &&
			header->type != PACKET_DISCONNECT)
			nc->lastrecvack = last;
	}

At the end we update the connection's “lastrecvack” to “last” one executed. If it's a ConnectPacket, we set the lastrecvack when reading the packet.

Executing a buffered packet range

We need to execute a packet range when we know we've got a complete sequence up to a certain ack. We return the last executed packet number here, in case it's behind “last”.

unsigned short ParseRecieved(unsigned short first, unsigned short last, NetConn* nc)
{
	OldPacket* p;
	PacketHeader* header;
	unsigned short current = first;
	unsigned short afterlast = NextAck(last);

	do
	{
		bool execd = false;

		for(auto i=g_recv.begin(); i!=g_recv.end(); i++)
		{
			p = &*i;
			header = (PacketHeader*)&p->buffer;

			if(header->ack != current)
				continue;

			if(!Same(&p->addr, &nc->addr))
				continue;

			PacketSwitch(header->type, p->buffer, p->len, nc, &p->addr, &g_sock);
			execd = true;
			current = NextAck(current);

			i = g_recv.erase(i);
			break;
		}

		if(execd)
			continue;

		break;
	} while(current != afterlast);

	return PrevAck(current);
}

SendData

We send data like so, passing the data bytes, size, address, whether it is meant to be reliable, whether we want it to expire after a certain time of resending (like a ConnectPacket that needs to fail sooner than the default timeout), the NetConn connection (which musn't be NULL if we're sending a reliable packet), the socket, the millisecond delay if we want to queue it to send a few moments from now, and a callback function to be called when it's acknowledged so we can take further action (like setting “handshook” to true for ConnectPacket's, or destroying the NetConn when a DisconnectPacket is acknowledged).

void SendData(char* data, int size, IPaddress * paddr, bool reliable, bool expires, NetConn* nc, UDPsocket* sock, int msdelay, void (*onackfunc)(OldPacket* p, NetConn* nc))
{
	//is this packet supposed to be reliable?
	if(reliable)
	{
		//if so, set the ack number
		((PacketHeader*)data)->ack = nc->nextsendack;

		//and add an OldPacket to the g_outgo list
		OldPacket p;
		p.buffer = new char[ size ];
		p.len = size;
		memcpy(p.buffer, data, size);
		memcpy((void*)&p.addr, (void*)paddr, sizeof(IPaddress));
		//in msdelay milliseconds, p.last will be RESEND_DELAY millisecs behind GetTicks()

		//set last sent time
		p.last = GetTicks() + msdelay - RESEND_DELAY;
		p.first = p.last;
		p.expires = expires;
		p.onackfunc = onackfunc;
		g_outgo.push_back(p);

		//update outbound ack for this connection
		nc->nextsendack = NextAck(nc->nextsendack);
	}

	if(reliable && msdelay > 0)
		return;

	PacketHeader* ph = (PacketHeader*)data;

	if(reliable && 
		(!nc || !nc->handshook) && 
		(ph->type != PACKET_CONNECT && ph->type != PACKET_DISCONNECT && ph->type != PACKET_ACKNOWLEDGMENT && ph->type != PACKET_NOCONN) )
	{
		Connect(paddr, false, false, false, false);
		return;
	}

	memcpy(out->data, data, size);
	out->len = size;
	out->data[size] = 0;

	SDLNet_UDP_Unbind(*sock, 0);
	if(SDLNet_UDP_Bind(*sock, 0, (const IPaddress*)paddr) == -1)
	{
		char msg[1280];
		sprintf(msg, "SDLNet_UDP_Bind: %s\n",SDLNet_GetError());
		ErrMess("Error", msg);
		//printf("SDLNet_UDP_Bind: %s\n",SDLNet_GetError());
		//exit(7);
	}

	//sendto(g_socket, data, size, 0, (struct addr *)paddr, sizeof(struct sockaddr_in));
	SDLNet_UDP_Send(*sock, 0, out);

	g_transmitted += size;

	SDLNet_FreePacket(out);
}

If it's reliable, we add an entry to the outbound OldPacket list. We set the “last” member variable of the OldPacket entry such that it is resent in a certain amount of time depending on when we delayed it to and the usual resend delay.

If it's reliable and the delay is greater than 0, we don't take any action in this function after buffering it in the outbound list because we will send it after ResendPacks() is called.

If it's reliable and we don't have a connection specified, we call Connect() to connect first, and return. It is also called if the connection hasn't finished the handshake (in which case Connect() will check to make sure that we have an outgoing ConnectPacket). The only case in which we don't need a handshook connection and send reliably is if we're sending a ConnectPacket or DisconnectPacket.

The SendData function is called itself with “reliable” set to false when resending a reliable packet from a buffered outbound OldPacket container.

The SendData function automatically sets the outbound ack for the reliable packets.

Keeping Connections Alive

As mentioned, there are three more functions in the UpdNet loop function:

KeepAlive();
CheckConns();
ResendPacks();

The KeepAlive() function sends KeepAlive packets to connections that are expiring. It prevents the other side from closing the connection, and also triggers an ack packet back, preventing from the connection being closed locally. The default is to keep connections alive until the user decides to Disconnect them.

//keep expiring connections alive (try to)
void KeepAlive()
{
	unsigned long long nowt = GetTicks();
	auto ci = g_conn.begin();

	//loop while we still have more connections to process...
	while(g_conn.size() > 0 && ci != g_conn.end())
	{
		//if we haven't received a handshake back, or if it's closed, we don't need to be keep it alive
		if(!ci->handshook || ci->closed)
		{
			ci++;
			continue;
		}

		//otherwise, if it's reached a certain percent of the timeout period, send a KeepAlivePacket...
		if(nowt - ci->lastrecv > NETCONN_TIMEOUT/4)
		{
			//check if we're already trying to send a packet to get a reply
			bool outgoing = false;

			//check all outgoing packets for a packet to this address
			for(auto pi=g_outgo.begin(); pi!=g_outgo.end(); pi++)
			{
				//if(memcmp(&pi->addr, &ci->addr, sizeof(IPaddress)) != 0)
				if(!Same(&pi->addr, &ci->addr))
				{
					continue;
				}

				outgoing = true;
				break;
			}

			//if we have an outgoing packet, we don't have to send a KeepAlivePacket
			if(outgoing)
			{
				ci++;
				continue;
			}

			//otherwise, send a KeepAlivePacket...
			KeepAlivePacket kap;
			kap.header.type = PACKET_KEEPALIVE;
			SendData((char*)&kap, sizeof(KeepAlivePacket), &ci->addr, true, false, &*ci, &g_sock, 0, NULL);
		}

		//check next connection next
		ci++;
	}
}

GetTicks() is our 64-bit timestamp function in milliseconds:

unsigned long long GetTicks()
{
#ifdef PLATFORM_WIN
	SYSTEMTIME st;
	GetSystemTime (&st);
	_FILETIME ft;
	SystemTimeToFileTime(&st, &ft);
	//convert from 100-nanosecond intervals to milliseconds
	return (*(unsigned long long*)&ft)/(10*1000);
#else
	struct timeval tv;

	gettimeofday(&tv, NULL);

	return
    (unsigned long long)(tv.tv_sec) * 1000 +
    (unsigned long long)(tv.tv_usec) / 1000;
#endif
}

Checking and Pruning Connections

Two more functions in UpdNet:

CheckConns();
ResendPacks();

In CheckConns we do several things:

1. Send out periodic pings for all the players in the room for all the clients using Cl(ient)StatePacket's
2. Handle and close any connections that are not yet closed but have timed out because the last received message has been longer than NETCONN_TIMEOUT milliseconds ago
3. For closed connections, flush any buffered inbound or outbound OldPacket's, and erase the NetConn from the list
4. For unresponsive clients, inform other players of the lagger

void CheckConns()
{
	unsigned long long now = GetTicks();

	// If we're not compiling for the matchmaker (the game app itself)
#ifndef MATCHMAKER

	static unsigned long long pingsend = GetTicks();

	//send out client pings
	if(g_netmode == NETM_HOST &&
		now - pingsend > (NETCONN_UNRESP/2)
		)
	{
		pingsend = now;

		for(int i=0; i<CLIENTS; i++)
		{
			Client* c = &g_client[i];

			if(!c->on)
				continue;

			if(i == g_localC)
				continue;	//clients will have their own ping for the host

			NetConn* nc = c->nc;

			if(!nc)
				continue;

			ClStatePacket csp;
			csp.header.type = PACKET_CLSTATE;
			csp.chtype = CLCH_PING;
			csp.ping = nc->ping;
			csp.client = i;
			SendAll((char*)&csp, sizeof(ClStatePacket), true, false, NULL);
		}
	}
#endif
	
	auto ci = g_conn.begin();

	while(g_conn.size() > 0 && ci != g_conn.end())
	{
		//get rid of timed out connections
		if(!ci->closed && now - ci->lastrecv > NETCONN_TIMEOUT)
		{
			//TO DO any special condition handling, inform user about sv timeout, etc.

#ifndef MATCHMAKER
			if(ci->ismatch)
			{
				g_sentsvinfo = false;
			}
			else if(ci->isourhost)
			{
				EndSess();
				RichText mess = RichText("ERROR: Connection to host timed out.");
				Mess(&mess);
			}
			else if(ci->ishostinfo)
				;	//ErrMess("Error", "Connection to prospective game host timed out.");
			else if(ci->isclient)
			{
				//ErrMess("Error", "Connection to client timed out.");

				/*
				TODO
				combine ClDisconnectedPacket and ClientLeftPacket.
				use params to specify conditions of leaving:
				- of own accord
				- timed out
				- kicked by host
				*/

				//TODO inform other clients?
				ClDisconnectedPacket cdp;
				cdp.header.type = PACKET_CLDISCONNECTED;
				cdp.client = ci->client;
				cdp.timeout = true;
				SendAll((char*)&cdp, sizeof(ClDisconnectedPacket), true, false, &ci->addr);
				
				Client* c = &g_client[ci->client];
				RichText msg = c->name + RichText(" timed out.");
				AddChat(&msg);
			}
#else
			g_log<<DateTime()<<" timed out"<<std::endl;
			g_log.flush();
#endif

			ci->closed = true;	//Close it using code below
		}

		//get rid of closed connections
		if(ci->closed)
		{
			if(&*ci == g_mmconn)
			{
				g_sentsvinfo = false;
				g_mmconn = NULL;
			}
			if(&*ci == g_svconn)
				g_svconn = NULL;
#ifndef MATCHMAKER
			for(int cli=0; cli<CLIENTS; cli++)
			{
				Client* c = &g_client[cli];

				if(!c->on)
					continue;

				if(c->nc == &*ci)
				{
					if(g_netmode == NETM_HOST)
					{
					}

					if(c->player >= 0)
					{
						Player* py = &g_player[c->player];
						py->on = false;
						py->client = -1;
					}

					c->player = -1;
					c->on = false;
				}
			}
#endif

			//necessary to flush? already done in ReadDisconnectPacket();
			//might be needed if connection can become ->closed another way.
			FlushPrev(&ci->addr);
			ci = g_conn.erase(ci);
			continue;
		}

		//inform other clients of unresponsive clients
		//or inform local player or unresponsive host
		if(now - ci->lastrecv > NETCONN_UNRESP &&
			ci->isclient)	//make sure this is not us or a matchmaker
		{
#ifndef MATCHMAKER
			NetConn* nc = &*ci;

			Client* c = NULL;

			if(nc->client >= 0)
				c = &g_client[nc->client];

			if(g_netmode == NETM_CLIENT &&
				nc->isourhost)
			{
				//inform local player TODO
				c->unresp = true;
			}
			else if(g_netmode == NETM_HOST &&
				nc->isclient &&
				c)
			{
				//inform others
				if(c->unresp)
				{
					ci++;	
					continue; //already informed
				}

				c->unresp = true;

				ClStatePacket csp;
				csp.header.type = PACKET_CLSTATE;
				csp.chtype = CLCH_UNRESP;
				csp.client = c - g_client;
				SendAll((char*)&csp, sizeof(ClStatePacket), true, false, &nc->addr);
			}
#endif
		}

		ci++;
	}
}

Resending Packets

Finally, ResendPacks():

void ResendPacks()
{
	OldPacket* p;
	unsigned long long now = GetTicks();

	//remove expired ack'd packets
	auto i=g_outgo.begin();
	while(i!=g_outgo.end())
	{
		p = &*i;

		if(!p->acked)
		{
			i++;
			continue;
		}

        //p->last and first might be in the future due to delayed sends,
        //which would cause an overflow for unsigned long long.
        unsigned long long safelast = enmin(p->last, now);
        unsigned long long passed = now - safelast;
        unsigned long long safefirst = enmin(p->first, now);

		if(passed < RESEND_EXPIRE)
		{
			i++;
			continue;
		}

		i = g_outgo.erase(i);
	}

	//resend due packets within sliding window
	i=g_outgo.begin();
	while(i!=g_outgo.end())
	{
		p = &*i;

		//kept just in case it needs to be recalled by other side
		if(p->acked)
		{
			i++;
			continue;
		}
        
        unsigned long long safelast = enmin(p->last, now);
        unsigned long long passed = now - safelast;
        unsigned long long safefirst = enmin(p->first, now);

		NetConn* nc = Match(&p->addr);

		//increasing resend delay for the same outgoing packet

		unsigned int nextdelay = RESEND_DELAY;
        unsigned long long firstpassed = now - safefirst;

		if(nc && firstpassed >= RESEND_DELAY)
		{
			unsigned long long sincelast = safelast - safefirst;
			//30, 60, 90, 120, 150, 180, 210, 240, 270
			nextdelay = ((sincelast / RESEND_DELAY) + 1) * RESEND_DELAY;
		}

		if(passed < nextdelay)
		{
			i++;
			continue;
		}

		PacketHeader* ph = (PacketHeader*)p->buffer;

		/*
		If we don't have a connection to them
		and it's not a control packet, we
		need to connect to them to send reliably.
		Send it when we get a handshake back.
		*/
		if((!nc || !nc->handshook) &&
			ph->type != PACKET_CONNECT &&
			ph->type != PACKET_DISCONNECT &&
			ph->type != PACKET_ACKNOWLEDGMENT &&
			ph->type != PACKET_NOCONN)
		{
			Connect(&p->addr, false, false, false, false);
			i++;
			continue;
		}

//do we want a sliding window?
#if 1
		if(nc)
		{
			unsigned short lastack = nc->nextsendack + SLIDING_WIN - 1;

			if(PastAck(lastack, ph->ack) && ph->ack != lastack)
			{
				i++;
				continue;
				//don't resend more than SLIDING_WIN packets ahead
			}
		}
#endif

		if(p->expires && now - safefirst > RESEND_EXPIRE)
		{
			i = g_outgo.erase(i);
			continue;
		}

		SendData(p->buffer, p->len, &p->addr, false, p->expires, nc, &g_sock, 0, NULL);

		p->last = now;

		i++;
	}
}

We

1.) erase OldPacket's that have been acknowledged (acked = true),
2.) check if the OldPacket in question is within the sliding window, and if it is,
2.) resend those OldPacket's that have reached a certain delay,
3.) and erase OldPacket's that are set to expire.

“enmin” and “enmax” are just the min max macros:

#define enmax(a,b) (((a)>(b))?(a):(b))
#define enmin(a,b) (((a)<(b))?(a):(b))

We don't want the “firstpassed” value (the amount of time that has passed since the OldPacket was first sent) to be negative (which would be a giant positive number for an unsigned 64-bit long long), so we set “safefirst” used in its calculation to be no more than the time “now”, from which it is subtracted. If we didn't do this, we would get undefined behaviour, with some packets getting resent and some getting erased.

        unsigned long long safelast = enmin(p->last, now);
        unsigned long long passed = now - safelast;
        unsigned long long safefirst = enmin(p->first, now);

		NetConn* nc = Match(&p->addr);

		//increasing resend delay for the same outgoing packet

		unsigned int nextdelay = RESEND_DELAY;
        unsigned long long firstpassed = now - safefirst;

Reading Acknowledgements

Whenever we receive an AckPacket, we call ReadAckPacket on it in PacketSwitch:

void ReadAckPacket(AckPacket* ap, NetConn* nc, IPaddress* from, UDPsocket* sock)
{
	OldPacket* p;
	PacketHeader* header;

	for(auto i=g_outgo.begin(); i!=g_outgo.end(); i++)
	{
		p = &*i;
		header = (PacketHeader*)p->buffer;
		if(header->ack == ap->header.ack &&
				Same(&p->addr, from))
		{
			if(!nc)
				nc = Match(from);

			if(nc)
			{
				nc->ping = (float)(GetTicks() - i->first);
			}

			if(p->onackfunc)
				p->onackfunc(p, nc);

			i = g_outgo.erase(i);

			return;
		}
	}
}

In it, we will check for the matching buffered inbound OldPacket, and erase it from the list if found. But before that, we call a registered callback method that was set up when the packet was sent.

Using the “first” time the packet was sent, subtracting it from the current time, gives the round-trip latency for that connection, which we can record in the NetConn class.

Callbacks on Acknowledgement

Whenever we send a DisconnectPacket, we set the callback function to:

void OnAck_Disconnect(OldPacket* p, NetConn* nc)
{
	if(!nc)
		return;

	nc->closed = true;	//to be cleaned up this or next frame
}

Which will clean up the connection and stop resending the DisconnectPacket once it's acknowledged. It's best to encapsulate the needed functionality so we can safely Disconnect.

void Disconnect(NetConn* nc)
{
	nc->disconnecting = true;

	//check if we already called Disconnect on this connection
	//and have an outgoing DisconnectPacket
	bool out = false;

	for(auto pit=g_outgo.begin(); pit!=g_outgo.end(); pit++)
	{
		if(!Same(&pit->addr, &nc->addr))
			continue;

		PacketHeader* ph = (PacketHeader*)pit->buffer;

		if(ph->type != PACKET_DISCONNECT)
			continue;

		out = true;
		break;
	}

	if(!out)
	{
		DisconnectPacket dp;
		dp.header.type = PACKET_DISCONNECT;
		SendData((char*)&dp, sizeof(DisconnectPacket), &nc->addr, true, false, nc, &g_sock, 0, OnAck_Disconnect);
	}
}

When we receive an acknowledgement of a ConnectPacket that we sent out, we also need to set “handshook” to true. You can set user callbacks for certain special connections, like matchmakers or game hosts, to carry out certain functions, like immediately polling for servers, or getting server info, or joining the game room.

//on connect packed ack'd
void OnAck_Connect(OldPacket* p, NetConn* nc)
{
	if(!nc)
		nc = Match(&p->addr);

	if(!nc)
		return;

	nc->handshook = true;

	ConnectPacket* scp = (ConnectPacket*)p->buffer;

	//if(!scp->reconnect)
	{
#ifndef MATCHMAKER
		GUI* gui = &g_gui;

		if(nc->isourhost)
		{
			g_svconn = nc;

			//TO DO request data, get ping, whatever, server info

			JoinPacket jp;
			jp.header.type = PACKET_JOIN;
			std::string name = g_name.rawstr();
			if(name.length() >= PYNAME_LEN)
				name[PYNAME_LEN] = 0;
			strcpy(jp.name, name.c_str());
			jp.version = VERSION;
			SendData((char*)&jp, sizeof(JoinPacket), &nc->addr, true, false, nc, &g_sock, 0, NULL);
		}
#endif

		if(nc->ishostinfo)
		{
			//TO DO request data, get ping, whatever, server info
			GetSvInfoPacket gsip;
			gsip.header.type = PACKET_GETSVINFO;
			SendData((char*)&gsip, sizeof(GetSvInfoPacket), &nc->addr, true, false, nc, &g_sock, 0, NULL);
		}

#ifndef MATCHMAKER
		if(nc->ismatch)
		{
			g_mmconn = nc;
			g_sentsvinfo = false;

			if(g_reqsvlist && !g_reqdnexthost)
			{
				g_reqdnexthost = true;

				GetSvListPacket gslp;
				gslp.header.type = PACKET_GETSVLIST;
				SendData((char*)&gslp, sizeof(GetSvListPacket), &nc->addr, true, false, nc, &g_sock, 0, NULL);
			}
		}
#endif
	}
}

You can see there's a commented out function pointer called “chcallback” in the NetConn class, which might be given a function to call when the connection is handshook, instead of hard-coding several cases for the connection type (“ismatch”, “isourhost”, etc.)

Connecting

Before we host a server or connect to the matchmaker, we must open a socket.

void OpenSock()
{
	unsigned short startport = PORT;

	if(g_sock)
	{
		IPaddress* ip = SDLNet_UDP_GetPeerAddress(g_sock, -1);

		if(!ip)
			g_log<<"SDLNet_UDP_GetPeerAddress: "<<SDLNet_GetError()<<std::endl;
		else
			startport = SDL_SwapBE16(ip->port);

		SDLNet_UDP_Close(g_sock);
		g_sock = NULL;
	}

	if(g_sock = SDLNet_UDP_Open(startport))
		return;

	//try 10 ports
#ifndef MATCHMAKER
	for(int i=0; i<10; i++)
	{
		if(!(g_sock = SDLNet_UDP_Open(PORT+i)))
			continue;

		return;
	}
#endif

	char msg[1280];
	sprintf(msg, "SDLNet_UDP_Open: %s\n", SDLNet_GetError());
	g_log<<msg<<std::endl;
	ErrMess("Error", msg);
}

This OpenSock method will try 10 different port numbers if the first one fails. If it still doesn't work, it will log a message from SDLNet. After we open a port, we can send packets. The OpenSock method is encapsulated in the Connect() function. We can call the first Connect method, which takes an IP string or domain address, or the second, which accepts an IPaddress struct. They also accept some parameters to describe their use, like whether the connection is the matchmaker, the host being joined, a client of our room, or a random server we're getting info on.

NetConn* Connect(const char* addrstr, unsigned short port, bool ismatch, bool isourhost, bool isclient, bool ishostinfo)
{
	IPaddress ip;

	//translate the web address string to an IP and port number
	if(SDLNet_ResolveHost(&ip, addrstr, port) == -1)
	{
		return NULL;
	}

	//call the following function...
	return Connect(&ip, ismatch, isourhost, isclient, ishostinfo);
}

//Safe to call more than once, if connection already established, this will just
//update NetConn booleans.
NetConn* Connect(IPaddress* ip, bool ismatch, bool isourhost, bool isclient, bool ishostinfo)
{
	if(!g_sock)
		OpenSock();


	NetConn* nc = Match(ip);

	NetConn newnc;
	bool isnew = false;

	//if we don't recognize this address as having a connection to, make a new NetConn instance for the list
	if(!nc)
	{
		isnew = true;
		newnc.addr = *ip;
		newnc.handshook = false;
		newnc.lastrecv = GetTicks();
		newnc.lastsent = newnc.lastrecv;
		//important - reply ConnectPacket with ack=0 will be
		//ignored as copy (even though it is original) if new NetConn's lastrecvack=0.
		newnc.lastrecvack = USHRT_MAX;
		newnc.nextsendack = 0;
		newnc.closed = false;
		g_conn.push_back(newnc);

		nc = &*g_conn.rbegin();
	}
	else
	{
		//force reconnect (sending ConnectPacket).
		//also important for Click_SL_Join to know that we
		//can't send a JoinPacket immediately after this function,
		//but must wait for a reply ConnectPacket.
		if(nc->closed)
			nc->handshook = false;
	}

	bool disconnecting = false;

	//if we have an outgoing DisconnectPacket, set disconnecting=true
	for(auto pit=g_outgo.begin(); pit!=g_outgo.end(); pit++)
	{
		OldPacket* op = &*pit;

		if(!Same(&op->addr, &nc->addr))
			continue;

		PacketHeader* ph = (PacketHeader*)op->buffer;

		if(ph->type != PACKET_DISCONNECT)
			continue;

		disconnecting = true;
		break;
	}

	//if we're closing this connection, don't send any other reliable packets on it except DisconnectPacket and clear any outbound or inbound OldPacket's
	if(disconnecting)
	{
		nc->handshook = false;
		FlushPrev(&nc->addr);
	}

	//different connection purposes
	//only "true" it, or retain current state of nc->...
	nc->isclient = isclient ? true : nc->isclient;
	nc->isourhost = isourhost ? true : nc->isourhost;
	nc->ismatch = ismatch ? true : nc->ismatch;
	nc->ishostinfo = ishostinfo ? true : nc->ishostinfo;

	if(isourhost)
		g_svconn = nc;
	if(ismatch)
		g_mmconn = nc;

	//see if we need to connect for realsies (send a ConnectPacket).
	//i.e., send a connect packet and clean previous packets (OldPacket's list).
	if(!nc->handshook)
	{
		bool sending = false;	//sending ConnectPacket?
		unsigned short yourlastrecvack = PrevAck(nc->nextsendack);

		//check if we have an outgoing ConnectPacket
		for(auto pi=g_outgo.begin(); pi!=g_outgo.end(); pi++)
		{
			if(!Same(&pi->addr, &nc->addr))
				continue;

			PacketHeader* ph = (PacketHeader*)pi->buffer;

			if(PastAck(PrevAck(ph->ack), yourlastrecvack))
				yourlastrecvack = PrevAck(ph->ack);

			if(ph->type != PACKET_CONNECT)
				continue;

			sending = true;
			break;
		}

		if(!sending)
		{
			ConnectPacket cp;
			cp.header.type = PACKET_CONNECT;
			cp.reconnect = false;
			cp.yourlastrecvack = yourlastrecvack;
			cp.yournextrecvack = nc->nextsendack;
			cp.yourlastsendack = nc->lastrecvack;
			SendData((char*)&cp, sizeof(ConnectPacket), ip, isnew, false, nc, &g_sock, 0, OnAck_Connect);
		}
	}

	nc->closed = false;

	return nc;
}

When closing a connection, or connecting again after a connection had been disconnected, we flush any buffered in- or out-bound OldPacket's.

//flush all previous incoming and outgoing packets from this addr
void FlushPrev(IPaddress* from)
{
	auto it = g_outgo.begin();

	while(it!=g_outgo.end())
	{
		if(!Same(&it->addr, from))
		{
			it++;
			continue;
		}

		it = g_outgo.erase(it);
	}

	it = g_recv.begin();

	while(it!=g_recv.end())
	{
		if(!Same(&it->addr, from))
		{
			it++;
			continue;
		}

		it = g_recv.erase(it);
	}
}

Conclusion

That is all for this article. If you want to see the rest of the article covering parity bit checking, lockstep, and LAN networking, purchase the full article here: http://www.gamedev.net/files/file/223-reliable-udp-implementation-lockstep-lan-and-parity-bit-checking/

Article Update Log

6 Oct 2015: Initial release