Renaun Erickson

My ActionScript Game Server with 15,000 Concurrent Connections

Renaun Erickson — Wed, 25 Apr 2012 19:33:05 +0000

Goal

Create a game using ActionScript for both the server side game server and the client side game with the ability for a large number of concurrent (10,000+) socket connections. Make sure memory and performance of the application are up to the task.

What am I talking about?

This blog post will explain how I created a socket server running on an Amazon EC2 Ubuntu server with more than 15k+ concurrent connections. This socket server has all the game logic for a game called SpellTraction, for which both the server and client are written in ActionScript.

Server: Game built in ActionScript 3 and run as a shell process using a modified redtamarin build. The modified redtamarin has a libev based socket server.
Client: Client game built with ActionScript 3.

To check out a running version of the client go to either:

http://renaun.com/serveras/test/ - this version is driven by other Amazon EC2 instances that are running test scripts to create the 15k+ connections.

or

http://renaun.com/serveras/spelltraction/ - the game with any other player looking at it right now.

Here is a screenshot of the game:

The source code that I used for the server and client is located on my github repo here.

History of leveraging Tamarin to run ActionScript as a Process (i.e. ActionScript on the Server)

Adobe open sourced the virtual machine (VM) of the Flash Player and donated it to the Mozilla foundation as the project Tamarin. What gives Flash Player its rich set of features is all the c/cpp classes created around the ActionScript Tamarin VM. Making the Tamarin VM available to developers opened the door for different use cases and implementations. This is where people started to realize they could do ActionScript on the server.

Three main projects that created feature rich classes around Tamarin for the specific use of server side ActionScript were: Thane (part of Whirled SDK by ThreeRings), redtamarin (by zwetan and others), and PushButton Engines Network component by Ben Garney.

Most of these projects were built for specific use cases and are not necessarily in active development. But the main point is that creating ActionScript based specific game servers is very doable. For example Whirled is a whole ecosystem that lets developers create ActionScript based games and load them on their server. The server then uses their Tamarin modified shell process to run the ActionScript based game code.

For my use case I wanted something that was specific to games and a little more modern, e.g. make it Node.js like. So I pulled down the redtamarin source, which had documentation that I could somewhat understand, and built a c/c++ libev based socket server. I added as3signals to have a simplified callback api to create a game tick and socket connections.

The API could easily be change some more to make it even smaller but the hopefully you get the idea. Here is an application that would respond to any connections to the server with the text "You have connected":

PLAIN TEXT

ACTIONSCRIPT:

public function connectHandler(client:ClientSocket, connect:Boolean):void
{
if (connect)
client.sendUTF("You have connected");
}
function loopHandler():void
{
if (!ss.listening)
ss.listen("192.168.10.20", 9898);
else
;// Game server tick
}
var ss:ServerSocket = new ServerSocket(connectHandler);
ss.loop.add(loopHandler);
ss.start(250);

The Fun of Solving the High Concurrent Connections Problem

There are a bunch of data and blog posts on the C10K problem, which nowadays is the C1M problem. Node.js is based on libev but just using the event model over the select, kqueue, epoll, etc.. socket models is half the battle. You still have to configure you server to handle that kind of load.

For example I am no expert in kernel configuration for linux, but for my current EC2 Ubuntu setup I have settings that look like this:

PLAIN TEXT

CODE:

/etc/sysctl.conf
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.core.rmem_default = 1048576
net.core.wmem_default = 1048576
net.core.optmem_max = 33554432
net.ipv4.tcp_rmem = 4096 4096 33554432
net.ipv4.tcp_wmem = 4096 4096 33554432
net.ipv4.tcp_mem = 786432 1048576 26777216
net.ipv4.tcp_max_tw_buckets = 360000
net.core.netdev_max_backlog = 30000
net.ipv4.ip_local_port_range = 2048 65535
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_no_metrics_save = 1
net.core.somaxconn = 131072
fs.file-max = 131072

/usr/include/linux/limits.h
NR_OPEN = 65536

/etc/security/limits.conf
* soft nofile 65535
* hard nofile 65535

Another side thing I learned was how much memory the OS uses for connections. At first I was worried about my game taking up a bunch of memory but quickly found out the bigger concern is what the OS needs to handle sending and receiving data on that many connections. Look at the 1 million connection link above, he is running servers with 25GB+ memory.

Conclusions

It took me just 3 days to go from not understanding c/c++ socket programming to having a working libev socket server prototype integrated with redtamarin. If I knew what I was doing it would have been quicker and I could flesh out a bunch more game specific features. But for my test case it was sufficient. The tooling workflow around Tamarin and building the abc files that get run by the shell process are a bit lacking. But zwetan's tools on the redtamarin Google code project help a lot (thats really for another blog post).

I have had the app running now for 5+ days with over 15k+ connections. I could try to push it past 15k+ connections but I didn't want to do large ec2 instances and ramping up that many connections actually takes a bit of time. The memory is staying stable at a delta of 15Mb since it started at a total of 20Mb (current stats). And using "top" on the server the CPU usages is not much more than 9%, and that is all processes running. This means memory and CPU are not the bottleneck for running ActionScript on the server. Of course this could change based on your game but my game does do some logic calculations each frame and sends a lot of messages out each 250ms, which is representative of, a big set of use cases.

My final conclusion: Using Tamarin is a very promising approach to using ActionScript on the server, especially for gaming. The pieces have been around for a while but the know-how is not so prevalent in the ActionScript community. If you are interested in using Tamarin for a game server let me know on twitter, @renaun.

Using Fullscreen Interactive and Mouse Lock in Flash Player 11.3

Renaun Erickson — Tue, 24 Apr 2012 18:20:14 +0000

Fullscreen Interactive Feature

In Flash Player 11.3 (currently in beta on labs.adobe.com) there is a new feature that allows applications to use full keyboard input in fullscreen mode.

To use the new fullscreen interactive feature you must first set the "allowFullscreenInteractive=true" value in the HTML embed code. Then in ActionScript you set the display state correctly, stage.displayState = StageDisplayState.FULL_SCREEN_INTERACTIVE;, to launch the application into fullscreen interactive mode. This mode presents the user with a slightly different overlay then normal fullscreen mode. It presents the user with an overlay that has an "Allow" button which must be clicked before entering the fullscreen interactive mode. Here is what it looks like:

As a developer you can listen for the new FullScreenEvent.FULL_SCREEN_INTERACTIVE_ACCEPTED event to know when the user has clicked the allowed button.

Mouse Lock Feature

The mouse lock feature is supported in Flash Player 11.2 and beyond. You can only turn on mouse lock, stage.mouseLock = true, while in any fullscreen mode. There is a bug in 11.3 that will be fixed for 11.4 that I'll explain and show a workaround for 11.3. Since you can only use it in fullscreen mode, it makes sense to set stage.mouseLock = true in the function handler for the FullScreenEvent.FULL_SCREEN_INTERACTIVE_ACCEPTED event. This is where the bug is, if you do set stage.mouseLock = true in that function handler it will not get set correctly, it stays equal to false. That means you have to wait a frame or some amount of time before setting it. In my code example on github I add a flag and then listen on MOUSE_MOVE, which I am doing anyways, to set stage.mouseLock = true. Here is a code snippet:

PLAIN TEXT

ACTIONSCRIPT:

protected function mouseMoveHandler(event:MouseEvent):void
{
if (isFirstTime == true && stage.mouseLock == false)
{
isFirstTime = false;
errorMessage.text = "here: " + stage.mouseLock;
stage.mouseLock = true;
}
if (stage.mouseLock)
{
errorMessage.text = "here: " + event.movementX;
deltaX = event.movementX;
}
else
{
//deltaX = event.stageX - lastX;
//lastX = event.stageX;
}
}

protected function fullscreenHandler(event:FullScreenEvent):void
{
if (event.type == FullScreenEvent.FULL_SCREEN_INTERACTIVE_ACCEPTED)
{
isFirstTime = true;
//*****
// For Cyril (Flash Player 11.3) this is a bug that mouseLock does not get set to true,
// The workaround is using isFirstTime approach
// Flash Player 11.4 will have a fix for this behavior
//*****
stage.mouseLock = true;

trace("["+event.type+"]Set mouse lock: " + stage.mouseLock);
errorMessage.text = "["+event.type+"]Set mouse lock: " + stage.mouseLock;
}
else
{
isFirstTime = false;
}
}

The full source as a Flash Builder 4.6 project, with modified index.template.html, -swf-version, and latest playerglobal.swc for 11.3 is located on my github repo here.

Here is a running version of the example - http://renaun.com/flash/fullscreeninteractivemouselock/

How To Know That Your Flash Content Requires a Premium Feature License

Renaun Erickson — Thu, 29 Mar 2012 18:40:57 +0000

A license is needed for your SWF if it uses premium features. The current premium feature that requires a license is a SWF that makes use of Stage3D and ApplicationDomain.domainMemory APIs at the same time. If only one is used without the other a license is not required.

Still not sure if your application requires the license? Well there is a way to find out with using a debug version of Flash Player. Starting with Flash Player 11.2 you will get a message that is rendered on top of your SWF if you require a license. Here is what it looks like:

Premium Features Watermark

And here is the code to force the watermark:

PLAIN TEXT

ACTIONSCRIPT:

package
{
import flash.display.Sprite;
import flash.events.Event;
import flash.system.ApplicationDomain;
import flash.utils.ByteArray;
import flash.utils.Endian;

[SWF(width="320",height="160",backgroundColor="0x333333")]
public class PremiumFeatureCheck extends Sprite
{
public function PremiumFeatureCheck()
{
addEventListener(Event.ADDED_TO_STAGE, addedToStageHandler);
}

protected function addedToStageHandler(event:Event):void
{
trace("Stage3Ds Lenght: " + stage.stage3Ds.length + "");
stage.stage3Ds[0].addEventListener( Event.CONTEXT3D_CREATE, initStage3D );
stage.stage3Ds[0].requestContext3D();

var testData:ByteArray = new ByteArray();
testData.endian = Endian.LITTLE_ENDIAN;
testData.length=0xffff*4; //4bytes

ApplicationDomain.currentDomain.domainMemory=testData;
var testValue:int=123;
ApplicationDomain.currentDomain.domainMemory[0] = testValue;
var readValue:int = ApplicationDomain.currentDomain.domainMemory[0];
trace(readValue+"");//should print 123

}
protected function initStage3D(e:Event):void
{
var context3D:Object = stage.stage3Ds[0].context3D;
}
}
}

If you comment out "stage.stage3Ds[0].requestContext3D(); " or remove the domainMemory lines of code you will not see this watermark.

NOTE: If you get a context3D error you need to make sure you have swf-version=13 and wmode="direct" set up properly to run Stage3D content.

It's also important to note that the watermark doesn't show up until the features are used in your code. Take this code example:

PLAIN TEXT

ACTIONSCRIPT:

package
{
import flash.display.Sprite;
import flash.events.Event;
import flash.events.MouseEvent;
import flash.system.ApplicationDomain;
import flash.utils.ByteArray;
import flash.utils.Endian;

[SWF(width="400",height="160",backgroundColor="0x333333")]
public class PremiumFeatureCheck extends Sprite
{
public function PremiumFeatureCheck()
{
addEventListener(Event.ADDED_TO_STAGE, addedToStageHandler);
}

protected function addedToStageHandler(event:Event):void
{
trace("Stage3Ds Lenght: " + stage.stage3Ds.length + "");
stage.stage3Ds[0].addEventListener( Event.CONTEXT3D_CREATE, initStage3D );
stage.stage3Ds[0].requestContext3D();

stage.addEventListener(MouseEvent.CLICK, useDomainMemory);
}
protected function initStage3D(e:Event):void
{
var context3D:Object = stage.stage3Ds[0].context3D;
}
protected function useDomainMemory(event:Event):void
{

var testData:ByteArray = new ByteArray();
testData.endian = Endian.LITTLE_ENDIAN;
testData.length=0xffff*4; //4bytes

ApplicationDomain.currentDomain.domainMemory=testData;
var testValue:int=123;
ApplicationDomain.currentDomain.domainMemory[0] = testValue;
var readValue:int = ApplicationDomain.currentDomain.domainMemory[0];
trace(readValue+"");//should print 123
}
}
}

The watermark will only show up once the domainMemory chunk of code is ran after clicking on the stage.

Here is another scenario, SWFMain.swf loads both SWFStage3DAPI.swf and SWFDomainMemory.swf, do you need a license? Yes you do, the water shows up and stays (even if you try and unload) after you have run API calls for Stage3D or ApplicationDomain.domainMemory regardless if they are from the main SWF or loaded SWFs. If you want to try it out yourself here is the source code:

SWFMain.as

PLAIN TEXT

ACTIONSCRIPT:

package
{
import flash.display.DisplayObjectContainer;
import flash.display.Loader;
import flash.display.Sprite;
import flash.events.Event;
import flash.events.MouseEvent;
import flash.net.URLLoader;
import flash.net.URLRequest;
import flash.text.TextField;
import flash.text.TextFormat;

public class SWFMain extends Sprite
{
public function SWFMain()
{
addEventListener(Event.ADDED_TO_STAGE, addedToStageHandler);
format = new TextFormat();
format.size = 24;
format.color = 0xffffff;
}

public var loaderStage3D:Loader;
public var loaderDomainMemory:Loader;
public var format:TextFormat;

protected function addedToStageHandler(event:Event):void
{
// Create buttons
var s:Sprite = createButton("Load Stage3D SWF");
s.x = 10;
s.y = 10;
s = createButton("Load DomainMemory SWF");
s.x = 10;
s.y = 60;
}

protected function createButton(label:String):Sprite
{
var text:TextField = new TextField();
text.defaultTextFormat = format;
text.text = label;
text.width = 200;

var s:Sprite = new Sprite();
s.graphics.beginFill(0x222222, 0.8);
s.graphics.lineStyle(2,0x000000);
s.graphics.drawRect(0, 0, 240, 40);
s.graphics.endFill();
s.addEventListener(MouseEvent.CLICK, clickHandler);

s.addChild(text);
text.x = 10;
text.y = 8;

addChild(s);
return s;
}

protected function clickHandler(event:MouseEvent):void
{
var t:String = "";
if (event.currentTarget is Sprite)
t = ((event.currentTarget as DisplayObjectContainer).getChildAt(0) as TextField).text;

trace("Click handler:" + t);
if (t == "Load Stage3D SWF")
{
if (loaderStage3D)
{
loaderStage3D.unloadAndStop(true);
loaderStage3D = null;
}
else
{
loaderStage3D = new Loader();
loaderStage3D.load(new URLRequest("SWFStage3DAPI.swf"));
loaderStage3D.x = 40;
loaderStage3D.y = 160;
addChild(loaderStage3D);
}

}
if (t == "Load DomainMemory SWF")
{
if (loaderDomainMemory)
{
loaderDomainMemory.unloadAndStop(true);
loaderDomainMemory = null;
}
else
{
loaderDomainMemory = new Loader();
loaderDomainMemory.load(new URLRequest("SWFDomainMemory.swf"));
loaderDomainMemory.x = 40;
loaderDomainMemory.y = 200;
addChild(loaderDomainMemory);
}
}
}
}
}

Here is the SWFStage3DAPI.as

PLAIN TEXT

ACTIONSCRIPT:

package
{
import flash.display.Sprite;
import flash.events.Event;

public class SWFStage3DAPI extends Sprite
{
public function SWFStage3DAPI()
{
addEventListener(Event.ADDED_TO_STAGE, addedToStageHandler);
}

protected function addedToStageHandler(event:Event):void
{
graphics.beginFill(0x0000ff);
graphics.drawRect(0, 0, 10, 10);
graphics.endFill();

trace("SWFStage3DAPI - Stage3Ds Lenght: " + stage.stage3Ds.length + "");
stage.stage3Ds[0].addEventListener( Event.CONTEXT3D_CREATE, initStage3D );
stage.stage3Ds[0].requestContext3D();
}

protected function initStage3D(e:Event):void
{
if (stage)
var context3D:Object = stage.stage3Ds[0].context3D;
}
}
}

Here is the SWFDomainMemory.as

PLAIN TEXT

ACTIONSCRIPT:

package
{
import flash.display.Sprite;
import flash.events.Event;
import flash.system.ApplicationDomain;
import flash.utils.ByteArray;
import flash.utils.Endian;

public class SWFDomainMemory extends Sprite
{
public function SWFDomainMemory()
{
addEventListener(Event.ADDED_TO_STAGE, addedToStageHandler);
}

protected function addedToStageHandler(event:Event):void
{
graphics.beginFill(0xff0000);
graphics.drawRect(0, 0, 10, 10);
graphics.endFill();

var testData:ByteArray = new ByteArray();
testData.endian = Endian.LITTLE_ENDIAN;
testData.length=0xffff*4; //4bytes

ApplicationDomain.currentDomain.domainMemory=testData;
var testValue:int=123;
ApplicationDomain.currentDomain.domainMemory[0] = testValue;
var readValue:int = ApplicationDomain.currentDomain.domainMemory[0];
trace("SWFDomainMemroy " + readValue);//should print 123
}
}
}

Epic Games Unreal Engine in the Browser Try For Yourself

Renaun Erickson — Thu, 08 Mar 2012 06:37:45 +0000

You can now try out the Epic Games Unreal engine running in the browser for yourself. All you need is the Flash Player 11.2 RC installed and then go to the http://unrealengine.com/flash/ website.

Here is a screenshot of it running in Chrome on my MacBook Pro:

Mark Rein form Epic Games talks about the latest Unreal demo at GDC.

At the Adobe GDC booth you can also see us show off Dungeon Defenders by Trendy which also uses the Unreal engine.

What does all this mean? Check out http://gaming.adobe.com for some more ideas of how AIR and Flash enable game developers to create great content.

Flash Game Summit 2012

Renaun Erickson — Wed, 07 Mar 2012 03:05:40 +0000

Flash Game Summit is held before the big Game Developer Conference (GDC). The 2012 Flash Game Summit was a great turn out, with a wide variety of attendees. Because you get a crowd that usually comes in for GDC from around the world Flash Game Summit has people and companies from all over the world.

Adobe is a sponsor and this year they provided live streaming as well as recordings of the sessions. So if you were not able to attend in person you can catch some of the latest topics around gaming with the Flash platform. There were two rooms with with session through the day so here they are separate by room:

Title	Speaker	Recording
Robertson Auditorium
Welcome	Josh Larson	http://partners-ar.adobeconnect.com/p10j7raiidd/
Flash The Next Generation	Lee Brimelow / Thibault Imbert	http://partners-ar.adobeconnect.com/p84glz9113f/
SAS: Zombie Assault 3	Chris Harris / Jonathan Hopcroft	http://partners-ar.adobeconnect.com/p5g1mwjvvcu/
Mobile Gaming Goes Boarderless	Peter Relan / Suran Markosian / Ben Savage	http://partners-ar.adobeconnect.com/p8tzebm31ge/
Mochis Award Show	Sponsored By Armored Games	http://partners-ar.adobeconnect.com/p515odcc3xo/
Kingdom Rush	Alvaro Azofra / Gonzalo Sande / Pablo Realini	http://partners-ar.adobeconnect.com/p2am4xm3asx/
For Your Ice Onlye	Reece Millidge	http://partners-ar.adobeconnect.com/p3fdn7cvzy4/
It's The End of The World And We Know it.	Ben Garney	http://partners-ar.adobeconnect.com/p9dh8z22yjw/
It's A Long Way to The Top	David Scott / Paul Preece	http://partners-ar.adobeconnect.com/p20c2imvu74/
Chasing China	Chris Shen	http://partners-ar.adobeconnect.com/p5vzvma4a11/
Fisher Room
Panel: Flash Development Life 2012	Moderated by: Ethan Levy, Producer, BioWare San Francisco Panelists: Sean McGee, Developer, Diseased ProductionsAndrew Sega, Lead Engineer, Sojo StudiosDan Stradwick, Developer, Monstrum	http://partners-ar.adobeconnect.com/p9ewajrc1fq/
Two Dimensions of Awesome: Advanced ActionScript for Platform Games	Iain Lobb, Managing Creative Director, Dull Dude Ltd.	http://partners-ar.adobeconnect.com/p71gwods77j/
Getting Some Perspective: Away3D 4.0 & Friends	Rob Bateman, Co-Founder, Away 3D	http://partners-ar.adobeconnect.com/p3qpzsv9gw7/
Going Mobile	Nate Beck, Principal Architect, ZaaLabs, Ltd.	http://partners-ar.adobeconnect.com/p4k55h49x30/
Winning With Audio	Selcuk Bor, CEO & Founder, Symphony of Specters	http://partners-ar.adobeconnect.com/p3ja0j9hl6m/
Ponycorns & The Price of Popularity	Ryan Henson Creighton, President, Untold Entertainment Inc.	http://partners-ar.adobeconnect.com/p12ns4cb8l6/
Screenplay	Tom Vian, Freelance Game Developer, Super Flash Bros.	http://partners-ar.adobeconnect.com/p43hdz1tfs6/

Some take a ways for me after talking with people at the conference are there are still alot of students that are building Flash games. Although they say their teachers and schools could use some more love. If you have a large student base and doing senior gaming projects, and are thinking of using Flash as the platform feel free to reach out to me and I'll try and help as much as I can. Next thing I noticed is people are building a lot of Flash games, but people are still not quite using Stage3D in mainstream. This is for various reasons, dev cycles, Flash Player runtime releases, mobile AIR 3.2, etc... but uniformly people want to get their Flash games seen. So again if you have some great looking content, using the latest Flash Player features the better, send them my way and I'll take a look.

Good luck and on to a great future of Flash gaming. And if you are looking for some of the latest info check out Lee and Thibault session above. And the new http://gaming.adobe.com site to participate in the telling of the Flash for gaming story.