My ActionScript Game Server with 15,000 Concurrent Connections
Goal
Create a game using ActionScript for both the server side game server and the client side game with the ability for a large number of concurrent (10,000+) socket connections. Make sure memory and performance of the application are up to the task.
What am I talking about?
This blog post will explain how I created a socket server running on an Amazon EC2 Ubuntu server with more than 15k+ concurrent connections. This socket server has all the game logic for a game called SpellTraction, for which both the server and client are written in ActionScript.
Server: Game built in ActionScript 3 and run as a shell process using a modified redtamarin build. The modified redtamarin has a libev based socket server.
Client: Client game built with ActionScript 3.
To check out a running version of the client go to either:
http://renaun.com/serveras/test/ – this version is driven by other Amazon EC2 instances that are running test scripts to create the 15k+ connections.
or
http://renaun.com/serveras/spelltraction/ – the game with any other player looking at it right now.
Here is a screenshot of the game:
The source code that I used for the server and client is located on my github repo here.
History of leveraging Tamarin to run ActionScript as a Process (i.e. ActionScript on the Server)
Adobe open sourced the virtual machine (VM) of the Flash Player and donated it to the Mozilla foundation as the project Tamarin. What gives Flash Player its rich set of features is all the c/cpp classes created around the ActionScript Tamarin VM. Making the Tamarin VM available to developers opened the door for different use cases and implementations. This is where people started to realize they could do ActionScript on the server.
Three main projects that created feature rich classes around Tamarin for the specific use of server side ActionScript were: Thane (part of Whirled SDK by ThreeRings), redtamarin (by zwetan and others), and PushButton Engines Network component by Ben Garney.
Most of these projects were built for specific use cases and are not necessarily in active development. But the main point is that creating ActionScript based specific game servers is very doable. For example Whirled is a whole ecosystem that lets developers create ActionScript based games and load them on their server. The server then uses their Tamarin modified shell process to run the ActionScript based game code.
For my use case I wanted something that was specific to games and a little more modern, e.g. make it Node.js like. So I pulled down the redtamarin source, which had documentation that I could somewhat understand, and built a c/c++ libev based socket server. I added as3signals to have a simplified callback api to create a game tick and socket connections.
The API could easily be change some more to make it even smaller but the hopefully you get the idea. Here is an application that would respond to any connections to the server with the text “You have connected”:
{
if (connect)
client.sendUTF("You have connected");
}
function loopHandler():void
{
if (!ss.listening)
ss.listen("192.168.10.20", 9898);
else
;// Game server tick
}
var ss:ServerSocket = new ServerSocket(connectHandler);
ss.loop.add(loopHandler);
ss.start(250);
The Fun of Solving the High Concurrent Connections Problem
There are a bunch of data and blog posts on the C10K problem, which nowadays is the C1M problem. Node.js is based on libev but just using the event model over the select, kqueue, epoll, etc.. socket models is half the battle. You still have to configure you server to handle that kind of load.
For example I am no expert in kernel configuration for linux, but for my current EC2 Ubuntu setup I have settings that look like this:
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.core.rmem_default = 1048576
net.core.wmem_default = 1048576
net.core.optmem_max = 33554432
net.ipv4.tcp_rmem = 4096 4096 33554432
net.ipv4.tcp_wmem = 4096 4096 33554432
net.ipv4.tcp_mem = 786432 1048576 26777216
net.ipv4.tcp_max_tw_buckets = 360000
net.core.netdev_max_backlog = 30000
net.ipv4.ip_local_port_range = 2048 65535
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_no_metrics_save = 1
net.core.somaxconn = 131072
fs.file-max = 131072
/usr/include/linux/limits.h
NR_OPEN = 65536
/etc/security/limits.conf
* soft nofile 65535
* hard nofile 65535
Another side thing I learned was how much memory the OS uses for connections. At first I was worried about my game taking up a bunch of memory but quickly found out the bigger concern is what the OS needs to handle sending and receiving data on that many connections. Look at the 1 million connection link above, he is running servers with 25GB+ memory.
Conclusions
It took me just 3 days to go from not understanding c/c++ socket programming to having a working libev socket server prototype integrated with redtamarin. If I knew what I was doing it would have been quicker and I could flesh out a bunch more game specific features. But for my test case it was sufficient. The tooling workflow around Tamarin and building the abc files that get run by the shell process are a bit lacking. But zwetan’s tools on the redtamarin Google code project help a lot (thats really for another blog post).
I have had the app running now for 5+ days with over 15k+ connections. I could try to push it past 15k+ connections but I didn’t want to do large ec2 instances and ramping up that many connections actually takes a bit of time. The memory is staying stable at a delta of 15Mb since it started at a total of 20Mb (current stats). And using “top” on the server the CPU usages is not much more than 9%, and that is all processes running. This means memory and CPU are not the bottleneck for running ActionScript on the server. Of course this could change based on your game but my game does do some logic calculations each frame and sends a lot of messages out each 250ms, which is representative of, a big set of use cases.
My final conclusion: Using Tamarin is a very promising approach to using ActionScript on the server, especially for gaming. The pieces have been around for a while but the know-how is not so prevalent in the ActionScript community. If you are interested in using Tamarin for a game server let me know on twitter, @renaun.
Pingback: renaun/ActionScriptGameServerExamples « marcusjpotter