Experimenting with Web Game Development

I am pleased to announce the initial version of neash 0.1, that works with the latest version on NME, 0.3.2. Neash allows the same code to be compiled for both the flash virtual machine and the neko virtual machine. NME provides the advanced vector rendering routines to emulate the flash9 API. You can download each of these packages using “haxelib install nme” and “haxelib install neash”.

The image here shows the same code running the the flash player, as well as a native win32 window. The code is in the samples area, and is as follows:

import neash.display.Sprite;
import neash.display.Shape;
import neash.geom.Matrix;

import neash.Timer;

class Clock extends Sprite
{
   var mCx:Float;
   var mCy:Float;
   var mRad:Float;

   var mMinuteHand:Shape;
   var mSecondHand:Shape;
   var mHourHand:Shape;

   public function new(inCx:Float, inCy:Float, inRad:Float)
   {
      super();

      neash.Lib.current.addChild(this);

      mCx = inCx;
      mCy = inCy;
      mRad = inRad;

      var gfx = graphics;
      for(i in 0...12)
      {
          gfx.beginFill(0x603030);
          gfx.lineStyle(0x000000);
          var theta = i * Math.PI * 2.0 / 12.0;
          var cos1 = Math.cos(theta-0.01);
          var sin1 = Math.sin(theta-0.01);
          var cos2 = Math.cos(theta+0.01);
          var sin2 = Math.sin(theta+0.01);

          gfx.moveTo( mCx + cos1*mRad,     mCy + sin1*mRad );
          gfx.lineTo( mCx + cos2*mRad,     mCy + sin2*mRad );
          gfx.lineTo( mCx + cos2*mRad*1.1, mCy + sin2*mRad*1.1 );
          gfx.lineTo( mCx + cos1*mRad*1.1, mCy + sin1*mRad*1.1 );
          gfx.lineTo( mCx + cos1*mRad,     mCy + sin1*mRad );
          gfx.endFill();
      }

      mSecondHand = CreateHand(mRad*0.02,mRad);
      mMinuteHand = CreateHand(mRad*0.05,mRad*0.75);
      mHourHand = CreateHand(mRad*0.1,mRad*0.5);

      var timer = new Timer(1000);
      var me = this;
      timer.run = function() { me.SetHandPositions(); }
      SetHandPositions();
   }

   function SetHandPositions()
   {
      var date = Date.now();

      var hour = date.getHours() % 12;
      var minute = date.getMinutes();
      var seconds = date.getSeconds();

      mSecondHand.rotation = seconds *360/60;
      mMinuteHand.rotation = (minute + seconds/60.0)  * 360/60;
      mHourHand.rotation =   (hour + minute/60.0 + seconds/3600.0) * 360/12;
   }

   function CreateHand(inWidth:Float, inLength:Float) : Shape
   {
      var result = new Shape();
      addChild(result);

      var gfx = result.graphics;

      gfx.beginFill(0x303060);
      gfx.moveTo(0,0);
      gfx.lineTo(-inWidth/2,-inWidth);
      gfx.lineTo(0,-inLength);
      gfx.lineTo(inWidth/2,-inWidth);
      gfx.lineTo(0,0);

      result.x = mCx;
      result.y = mCy;

      return result;
   }

   static public function main()
   {
      neash.Lib.Init("Clock",640,480); 
      neash.Lib.SetBackgroundColour(0xffffff); 

      new Clock(320,240,200);

      neash.Lib.Run(); 
   }
}

Porting this from a pure flash program involves:

1. Changing the “import” commands to replace “flash” with “neash”
2. Also changing from the “haxe.Timer” to the “neash.Timer”, which supports additional options.
3. The “main” routine must call “neash.Lib.Init( )” very early and at some point must call “neash.Lib.Run()”.
4. That’s it!

I’m looking at ways to remove point 1 using some compiler options, but this may not happen. Also note that the command-line for compiling requires “-lib neash” and, on the native platform, also requires “-lib nme”.

The clock example is really just to show timers working. NME runs the game loop as fast as it can at the moment, so timers are a bit pointless to some extent. This may change in the future.

The flash9 api coverage is a little thin at the moment. I have concentrated on flash.display.Graphics, including linear and radial gradients, bitmaps and line styles. However, some supporting things such as events and TextFields, are not well covered. Also note that some things (so far using a colour-alpha as a 32-bit int, eg #ffffffff) are not supported under neko.

Neash is currently only for flash9/neko compiling. Flash8 and JavaScript will only be supported if there is big need for it (basically if someone else wants to do it).

Neash does not have a binary component, so it works on platforms currently supported by NME. Namely windows and linux. Mac support will follow when NME is ported.

Neash was designed initally for games, and so is quite graphics-oriented. Here is an example showing the neash port of the “plotex” haxe module (you can get it with “haxelib install plotex”) running on windows. I’m not sure if this necessarily useful, but it is fun in a dorky sort of a way. One of the porting problems was due to the fact that plotex already supports 3 out of the 4 haxe platforms, so it contains conditional compiles. As a rule of thumb, use neash where you would use flash9. So “#if flash9″ becomes “#if (flash9||neko)”. Another issue was parameters passed from the html to the swf object. These are emulated in neash using command-line arguements. The source code changes can be found here. Note:copyright retained by original author, see plotex website for more details.

Well, one year on and I’ve had some fun, but still no games! The technology side of things has certainly come into focus in the last 12 months. I’ve seen haXe double and redouble performance to become my preferred option for web game development. I’ve been actively working on the NME to get the downloadable side of things going, and I’m almost done (just add one more feature…). Once that is done, I will pull my finger out and actually try to release a game - even if it is a little one.

I’m really enjoying the haXe/flash development, and looking back at my original ideas about trying java, I think I will have to change my game-plan (or “mission statement”) to just look at haXe/as3 based programming. The technology is pretty good these days, so it just comes down to productivity, and having fun.

So, I decided to try porting NME to Linux. The first thing you do is find an empty hard drive, download an ISO, and off you go, right? Wrong. I did this - but I wanted to keep my windows partition, without risk of rooting it over, so I didn’t over write my MBR. Then, I could not actually boot my linux partition. Eventually I got around this by booting from a rescue CD (I do not, and never will again, own a working floppy driver) and specifying my /dev/sd partition in a bit of a round about sort of way. Anyhow, I got Linux booting (with a little bit of pain per boot - but I could live with this), but then the wireless network did not work. This I could not live with, so I though about resurrecting some crappy old hardware I have lying around, but in the end I sought, and found, an infinitely better solution.

The answer is virtualization! I can’t overstate how much easier this was than actually getting the hardware together. First, I downloaded the free version of vmware’s “player” product, and then the Ubuntu 7.1 virtual machine, that included a dev environment http://jars.de/linux/ubuntu-710-vmware-image-download-english. The reference to “jars” seems to be a java reference, but we wont hold that against them. It “just worked”. I edited the xorg.conf file to change the keyboard to US, (so I could use the ‘|’ key), and that was about it. With only mimimal clicking, the network was fully working, and I could switch to my windows emails etc. This was what I wanted - much better than dual boot. Compiling NME was just a matter of adding the “dev” version of packages (vim, subversion, SDL, SDL_miver, SDL_image, TTF) with the GUI “Synaptic Package Manager”, which also “just worked”.

The download from code.google, via svn, worked easily once I added the “subversion” package.

Then a little makefile to bring it all together. I had to make some code hacks. I don’t know if this is a gcc “bug” or “feature”, but when I tried to call a class member function, that was templated on an int, eg: int x = Class.Member<4>(10); the gcc compiler thought I was using “operator<”, which is a bit of a shame.

I also had a quick go at static linking, via the “.a”s, rather than the “.so”s, but the linker told me I needed a “version section” for one of the SDL symbols, so I gave up.

The image you see is in the top left is of a window, on my windows desktop, running a virtual machine, running ubuntu, running the neko virtual machine, running a neko script linking to the Linux version of NME. And running it very well, I might add.

The only problems at the moment, are text, when a font is not supplied, the “mod” music in the “blox” demo, and OpenGL on my setup. The PNG, JPG, WAV and OGGs stuff all worked first time.

I think I might be able to get the music going, if I get the right plugin dso.

Also, I’m not sure about what “.so” files need to be distributed with the program to resolve all the dependencies. Any SDL linux guru have any comments on this?

Neko NME now compiles into a single, statically linked ndll (actually, it still uses plugins for mp3 and tiff). This means you can’t use the standard development distributions for building it, so I’ve made some libraries to use. You can download them from nme-dev-0.3.tgz.

To compile, first checkout the svn version as described below, then extract these to create a “thirdparty” directory next to the nekonme-read-only directory. The “NME.sln” project should then refere to the libraries and includes in this thirdparty directory.

This has been build with Visual Studio Express 2005. Currently, this is a windows-only version.

I have had a bit of a think about where some of this cross-platform code should sit, and I’ve partnered with Lee McColl-Sylvester at DesignRealm to add the functionality to the existing “Neko Media Engine” (NME) project. This idea here is to provide the flash drawing API to the neko runtime using opengl or software - which ever is fastest at the time.

There have been some big changed to NME recently, and it’s pretty easy for a haxe developer to checkout the “bleeding edge” and have a look at the samples, if they have svn installed. First install the existing NME project using the haxelib tool, ie:

haxelib install nme

Now, at the time of writing, it is 0.2.0, which is now a bit old. To use the new stuff (this works on most haxelib modules), checkout the latest stuff from code.google.com. First make a directory - it’s a matter of taste where - to hold the code. For simplicity, I’m using one directory to hold all the google code checkouts, and I’m calling it “C:/code.google/”. From a shell in this directory, follow the instructions on project page about how to checkout the svn version:

svn checkout http://nekonme.googlecode.com/svn/trunk/ nekonme-read-only

(You can actually call the last bit of the directory anything you want). This will give you a whole lot of files, called something like “c:/code.google/nekonme-read-only/nme/Manager.hx” etc. Now you can point your neko at this new code using the haxelib command:

haxelib dev nme c:/code.google/nekonme-read-only

This should give you access to both the new “.hx” class files, and the new “Windows/nme.ndll” binary file. Then you can look at the samples by building them with “haxe Compile.hxml”, and running them with “neko *.n”.

There are examples showing the use of sound, a complete game and where things are going with the new flash-drawing api (not yet complete). The images here show circles, lines and text (solid and transparent backgrounds) and 2d-transformations, using both software and opengl rendering (no bilinear-sampling on software version yet).

I have renamed the “blink” project to “neash” (neko flash) and made a project page for it on code.google.com.

http://code.google.com/p/neash/

You will need the neko/dlls from the new NeashDemo.zip. This has “blink” renamed to “neash” and I think I’ve fixed a bug in the font stuff that caused a crash on vista.

You could also use the new stand-alone exe (with font fix), and checkout the same demos with the fix too.

neko-nme-static.exe

cardemo.exe

robotdemo.exe

Speaking of stand-alone neko executables, the first half of the tutorial below is already covered the project “xCross”, which is cross platform,integrated with haxelib and includes regexp. So that is worth knowing about.

Did you know you can bind a neko “.n” file to the neko exe to create a stand-along exe? Well you can, with the “nekotools.exe boot” command. This simply appends the “.n” file to the standard neko.exe, and adds a small footer, giving you an executable that you can run. This is all well and good, except that you will also need to distribute the “.ndll” files, and place them correctly (usually, next to your exe) so that the dynamic loader can find them. This is also not that hard, but is there is an even simpler, single file solution, with no chance of picking up a wrong or development version. The trick is to build a statically linked version of neko.

To do this, you are going to need to compile the source code yourself. I will be using here Visual Studio 2005, express edition because I am tight and it is free. Start by downloading the source from nekovm.org. Currently, neko is at version 1.6.0. Unpack it and your will get a directory like “neko-1.6.0”, which has sub-directories like “libs” and “vm” in it. I will call this directory “the neko directory”, NEKO. You will also need the garbage collector, “gc”. You can download it from hp.com. Unpack it, and you will get a directory called something like “gc6.7”. Now this is something I’ll be doing with all the extra packages we will be downloading: rename the directory to have no version number, and move it to somewhere in the NEKO/libs directory. ie “NEKO/libs/gc”.

Now start a new empty visual studio project. Set the location to NEKO, so the path names can be relative. I’ve called mine “neko-static”. Untick the “Create directory for solution” (because I hate this option). Exit visual studio and move the “.sln” and “.vc_proj” files out of the “neko-static” directory that VC created against your will into the NEKO directory. (my second most hated feature of VC). Now remove the neko-static directory, and launch the neko-static.sln file. You can leave it in the default location if you wish - I’m just old and stuck in my ways.

Expand out the neko-static folder and add existing items to the Source Files : all the “.c” files from the vm directory, except “gc.c”. (gc.c is not needed since we are using the gc lib).

At this stage if you try to compile you get a bunch dll linkage errors. To get around this, you need to add the -DNEKO_SOURCES deines, so right-click the project, set the configuration to “All configs” instead of “Active” (THE FEATURE I HATE THE MOST - one of the reasons I prefer makefiles) and in the c++/Preprocessor/Proprocessor Definitions add “NEKO_SOURCES”. This says we are building and exporting neko, not importing a dll.

If you compile now, you will get an error in alloc.c finding gc/gc.h, because it is in the “include” directory, not the base. I think the easiest way to fix this is to edit alloc.c to #include “libs/gc/include/gc.h”.

Now you get a bunch of “__imp__GC_ …” undefines. This tells me that we are using the dynamic import version for gc, so also change the #define of “GC_DLL” to “GC_NOT_DLL” in vm/alloc.c. Now we get the static versions undefined, which is what we want - fix this by creating a sub-folder in the project “Source Files” called “gc” and add the files: allchblk.c alloc.c blacklst.c checksums.c dbg_mlc.c dyn_load.c finalize.c headers.c mach_dep.c malloc.c mallocx.c mark.c mark_rts.c misc.c new_hblk.c obj_map.c os_dep.c ptr_chck.c reclaim.c stubborn.c typd_mlc.c win32_threads.c. Hint: instead of using the “Existing item” dialog from the project, you can keep an explorer window open and drag in the files you want.

I initally made the mistake of putting “gc_cpp.cpp” in the project. This overrode the “new” operator and sent eveything through gc. It worked, but was much slower (you could hear the hard drive grinding) (Edit:maybe it was the log file ?).

And add to the project the include directory “../libs/gc/include/” from the project properties (don’t forget my most hated feature) in c++/General/Additional Include Directory.

Now there are only a couple of errors. In builtin.c, comment out the “_ftol2” function. Add the defines :GC_NOT_DLL;GC_WIN32_THREADS, and add the “Linker/Input/Additional Dependencies” user32.lib. While you are there, add _CRT_SECURE_NO_DEPRECATE to the defines, and 4996 to the Advanced/”Disable Specific Warnings” for more reasonable output.

I now have a “Debug/neko-static.exe” that can be run. Not that it will do us much good, because it has no libraries, and crashes on:

class Test
   { public static function main() { neko.Lib.print("Hello static world.\n"); } }

because it finds the “std.ndll” from NEKOPATH. Because it is statically linked, it will not work properly with any extenal ndll files. We can get around this with a simple trick. In “load.c”, instead of using h = dlopen(val_string(pname),RTLD_LAZY); Use: h = GetModuleHandle(0); This is quite neat, because any “__dllexport” functions will be accessable will be visible from the exe module.

Now the program just fails because there is no std library. Fair enough - just add one: Create in “Source Files” filters “libs” and “libs/std”, and add all the .c files from libs/std. “neko.h” is not found, so add “vm” to the include path (DFMHF), and add “wsock32.lib” to dependencies. Also, comment out “_ftol2” from math.c.

Now build - hey presto a stand-alone neko exe!

One more trick - before adding the “.n” bootstrap onto the end, compress the exe using the rather satifying exe-packing program, upx. You need to do this first, because the script will be read from the original file on disk, not the decompressed image in memory.

So you get:neko-static-upx.exe that you can use as a base. To build an exe, you can use the “nekotools.exe boot -b neko-static-upx.exe Test.n”. If neko has trouble finding the exe, place it in the $NEKO_INSTALLPATH directory. This exe should run anything that uses only the “std” library (sadly not “regexp”). There is one bug with this build, it generates “gc.log” files. I will have to track down the flag to disable this. So witness my opus:

test.exe

Note: Now, I’m no lawyer (as all good legal opinions start), but all the code here will be using the LGPL license (or more liberal). My understanding is that the spirit and letter of the law means that if you compile with LGPL code, you should give everyone the right to RELINK your code with a different version of the LGPL code. Normally this is done with DLLs, since this is simple. In this case, it is being done with the “.n” file you are embedding. ie, there is no reason why someone could not relink your exe using the excact same procedure above. You do not need to give them your haxe source code, only your “.n”. However, it does mean you can’t do any tricky hidden DRM stuff in the exe, without surrendering the appropriate “.obj” file.

“Harr”, I hear you say, “that is satisfying”. But wait, there’s more.

Wouldn’t it be cool if you had a simple base you could use for all your gaming? Well, it can be done. To statically link against NME, you need to gird your loins and download the following packages:

• SDL-1.2.12.tar.gz
• SDL_mixer-1.2.8.tar.gz
• SDL_image-1.2.5.tar.gz
• SDL_ttf-2.0.9.tar.gz
• libogg-1.1.3.tar.gz
• libvorbis-1.0.0.tar.gz
• freetype-2.3.5.tar.gz
• pcre-7.4.tar.gz

I ran out of steam and just commented out the “sge” library bits, and stopped at ogg audio (untested) (sorry Lee).

If you are not interested in gaming , the regexp bit may still be appropriate. I extracted the pcre library to libs/regexp and renamed it “pcre”, and the “hand configured” by

cp  pcre.h.generic  pcre.h
cp  config.h.generic  config.h
cp pcre_chartables.c.dist pcre_chartables.c

Added ‘#include “config.h’ to top of pcre.h and removed “HAVE_DIRENT_H” define. I then added the c files (skipping the example files that define a “main” function) as well as “libs/regexp/regexp.c” to the project in a sub-folder.

I copied the NME source files from the “project” folder of the “haxelib” version of NME. I then added some extensions for blink, and commented out the sge stuff because there are only so many libraries you can add in an evening.

I created an “sdl” directory in the NME directory (should have probably stuck it next to it) and extracted all the above libraries and renamed them without their version numbers. I then painfully selected source files from these libraries for the project. Some things to note:

• #define OGG_MUSIC from the project properties.
• #define _WIN32_WINNT=0x400 to get TRACKMOUSEEVENT
• Added each of the packages somewhere in the include path, eg: libs/nme/sdl/SDL/include libs/nme/sdl/SDL_image libs/nme/sdl/SDL_mixer etc.
• Added SDK path the get ddraw.h (additional download required for VS express edition)
• #define FT2_BUILD_LIBRARY to say we are building it, not using it.
• When adding files from freefont, you only need to add one from each “driver”.
• I commented out the “GC_write” function to try to stop the log.

When compiling modules with “DEFINE_ENTRY_POINT”, you get a multiply defined symbols. So I commented these out (could probably have done something in the header), and added specific calls to these boot routines in vm/main.c.

There, if you are still awake, you have it. The final output is the base: neko-nme-upx.exe(615k) which will run the BlinkDemo “.n” files (and any standard program that uses “std” and “regexp”) without and additional help. And see the stand-alone programs:

robotdemo.exe (765k)

cardemo.exe (760k)

Plenty of stuff still to be done, such as “subsystem:windows”, more audio, sge etc, but I think this is an excellent way to deliver games (or utilities for that matter), without fear of DLL hell. It also sidesteps “manifest hell” that microsoft seems keen on inflicting on us by insisting that the vc8 dynamic runtime is installed in a central location. I think the logic goes “if .net apps have to have a separate runtime installer, every one has to too”. What a stupid solution to force when the simple “just put the dll in the application directory” has worked for decades. (sorry about the rant).

It runs faster, too!

So far, I’ve mostly looked at the flash/swf version but now I will return my attention to cross-platform development.

There are a number of existing libraries that can be used with haXe, but most of these are low level, but what I’m after is a higher level option. So the plan is to build a higher-level layer on top of an existing module. I have chosen to build on top of NME, which is SDL based. My decision was mainly to do with support for opengl, sound/music, font, input and screen management.

In the end, the design wrote itself, based on the simple rule “it should be easy to port something from existing flash code”. Initially I tried writing a substitute library called “flash”, but the haxe compiler rejected me. This is probably best, because, although the alternative requires slightly more porting for the flash case, I think it allows for greater possiblities of minor architectural changes. This has two big advantages - half the work is already done for me and there is an excellent design document for the rest.

The result is a library I have called “blink”. There is essentailly one blink class definition for each flash class. On the flash platform, a simple “typedef” is used to get exactly the same code as native flash. On the neko platform, there is a haxe implementation that ultimately falls through to an NME call.

The library is only at the demo stage, and only implements enough to get the APE demos off the ground, but I think is shows the possibility. The only changes required were to change “flash.” to “blink.”, modify the main-line boot function slightly and make sure to use cross-platform constructs (eg, no “as” casting).

The code here (BlinkDemo.zip) shows the same code compiled for flash and neko. It uses a slightly extended NME library, which is provided as a dll in the bin directory - to use the dll, make sure you run the neko.exe in the bin directory so it finds the right one.

The updated performace is (note:using “cast” not “as”):

	Car Demo	Robot Demo
Original	2.0ms	9.5ms
haXe	1.58ms	9.45ms
hx->as3	1.56ms	9.47ms
neko/nme	4.0ms	16.9ms

On first glance it would appear the numerical processing takes about twice as long under neko as it does under flash. However this code might not be the greatest test because we can see how the performance of the “cast” command can effect the results.

Also of note is that the graphics is quite capable of reaching 100fps, so I do not think the SDL code will be a bottleneck.

I am very pleased with this approach, and I think it might be the way forward for cross platform game development. In some ways (certain) games are easier because they use a generally smaller sub-set of graphics primitives - mostly image drawing.

It was pointed out to me that there was a better way to do a “cast” and a few simple changes to to porting script yielded some big improvements. So, the new bundle here now gives:

	Car Demo	Robot Demo
Original	2.0ms	9.5ms
haXe	1.58ms	9.45ms
hx->as3	1.56ms	9.47ms

So now you can add speed as a reason to use haXe.

I see that APE http://www.cove.org/ape has moved on to version 0.45 alpha, and has an extremely beautiful “robot” demo. So, with the faster version of haXe, and improved knowledge, I though it was time to try porting it again. This time, I took a different approach - I wrote a program to do the porting for me. This has a few advantages. It allows for easy porting of future versions. It provides a list of things required, and it allows for modifications (such as the FPS counter) to be done only once (to the as3 code) and ported automatically to the haXe code.

The full project can be found here. It contains source, conversion program and demos.

The timings for the calculations are as follows:

	Car Demo	Robot Demo
Original	2.0ms	9.5ms
haXe	2.04ms	12.1ms
hx->as3	2.3ms	24.1ms

Which I think is pretty good - except for the last entry - not sure what happened there. Note that the haXe speed required a hack to avoid the “as” and “is” cast/query operators - and used a virtual function to achieve the same result in a neat way.

The conversion program is not a complex parser, rather a bunch of regular-expressions that relied on coding style as much as syntax. However, it worked pretty well in the end, once I got the “properties” sorted out - APE uses these quite a bit - and you must have “strong” types to use them in haXe. This program may be reusable to a small extent, but it pretty much tied to APE.

An outline of the porting tasks is as follows:

• Convert “int”, “void”, “Number” etc.
• Convert “package xxx {” to “package xxx;”.
• Expand out “import xxx.*” imports.
• Remove “private”, “final”, “internal” etc.
• Scan the class for “get” and “set” functions and insert “var prop(get,set):type” where appropriate. This was complicated by the fact that some of these were “override” properties and should not have this extra insertion. (I should have looked for the “override” keyword to make this easier).
• Add return statements to set functions.
• Fix POSITIVE_INFINITY.
• Make sure arrays are strongly typed - need this for properties.
• Change in-line array declarations when array is not of type Dynamic.
• Convert “indexOf” function in array.
• Convert “for(a ;b ;c )” to “a; while(b) { … c }”.
• Fix scoping of variables resulting from variables declared inside for statements.
• Add semi-columns to lines that needed them.
• Change constructors to “new”.
• Add static main function to main class, and “addChild” it.
• Call “super()” where required.
• Convert default-arguments to optional-arguments.
• Remove “break” from switch statements.
• Change “is” and “as” operators.

AS3 and haXe are reasonably close and with a consistent coding style, I think the automatic porting is a very viable option. If I had control over both sources, I would have done a few little things to the AS3 to make it slightly easier - ensure “;” on all lines, explicit call to super(), don’t double up on variable names inside for loops and other minor stuff. But the reg-ex engine makes most of these things pretty easy to work around.