Report and presentation

Hi everyone!

We are nearly done with our bachelor's thesis, yeeha!

You can download out thesis from here:

A Generic Game Server - Bachelor's Thesis (PDF)

And today we gave a 20 minutes presentation, here are the presentation slides.

JavaScript Virtual Machine in GGS

In short: it works bitches!

Oh yeah, thanks to Mattias work this week we are able to use JavaScript as a programming language for our games now. There remains still a little bit of work but it does not hang when we try to get something into of from the database, check this out:

2> GameVm = ggs_gamevm:start_link("test_table").
<0.85.0>
2> ggs_gamevm:call_js(GameVm, "GGS.jeena.setItem('a','foo')").
undefined
3> ggs_gamevm:call_js(GameVm, "GGS.jeena.getItem('a')").     
"foo"

This is so awesome, I won't be able to sleep tonight :D I think I need beer or something :D

What needs to be done now is to adapt the code so the first client, who will act as kind of a host, is able to upload the server source code (JavaScript) at the beginning and then other clients shall be able to connect to this one table.

Second testing session

Today we had an online testing session. Jeena started an instance of the GGS on his server in Germany. I(Niklas), Mattias and Jeena connected pong clients written in Erlang to the server in order to test how much data/how many client the GGS could handle. In the last testing session we tested the server by how many clients it could handle, however this seems like a bad way of testing a system like the GGS. Jeena found out that a good way of testing servers is how many messages they can handle per second and in today’s session we focused on that instead of how many clients that could be connected to the server. To be able to measure the number of messages per second did we add some minor things in the GGS.

When the GGS was operational we started to connect bots, the messages per second reached 5000 when about 200 bots where connected. However the number of messages per second did soon decrease to about 3500 even though we added more bots. We continued the testing for a while and the server did never proceed more than 5000 messages per second, most of the time it was pending between 3000 and 4000 messages per second. The conclusion we can draw from this is that the GGS currently is limited to around 5000 messages per second and when this number is exceeded the server will queue messages. A short message queue is acceptable but when it becomes to long the clients will get delays and the game will be impossible to play if to many clients are added to the same server.

The blue line is the amount of clients, the yellow one is how many messages per second the clients send and the red one how many messages per second the server ist sending to the clients.

Stress testing

Here is a little video showing us stress testing the GGS prototype.

Here are the conclusions we reached today, the stuff we need to work on, and
the thoughts that came up when we tested the application.

*Can ping be used to measure the network saturation?*
Quite possibly yes; ping measures the round-trip time of one ICMP packet (as
far as I know), and "if all goes well", the RTT will be the same for all
packages. It is possible, though, that some harwdare (read: the switch)
prioritizes ICMP packages through QoS (I think it's called that). It's also
possible that the OS treats pings differently than other traffic. This is
the main "problem" I have with using ping to measure saturation. We should
look these things up, if pings are not specially treated, they are a good
measurement of saturation.

*Optimize network traffic by sending less data*
This is obvious. Less traffic means less saturation. We should/could/may
send less data, and therefore increase performance both by removing the
(potential) network bottleneck, and remove some processing load from the
network cards / network stacks of the OS:es.

*Are screen print-outs a performance issue?*
We can possibly lower the IO bottleneck by removing printouts, we did this.
There is a information loss here for us when debugging, so a tradeoff has to
be made. A logging system which is customizable would be ideal.

*Load average*
On Niklas' machine the load was *really* low, on my machine it never went
below 1, and never above 1.3. On Richard's Mac it was like 4.. Is this
really a good measurement? Are there others which we can use? Erlang has a
notion of "reductions" - maybe we can use that? Maybe we can measure the
processor time consumed?

*# sockets limit*
Common machines seem to have limits on open files, and this includes
sockets. We should look at how web servers are configured here. Is it common
to raise this limit.. Are there penalties / alternatives to this? On Linux,
change /etc/security/security.conf to increase the limit.

*Warning; Mnesia is overloaded!*
Can we use the DB less? Exactly what does this mean? Can we use a different
machine as DB server?

*Can we get super-accurate CPU stats?*
Can we get stats from /proc/ instead of top and the likes? Can we compare
different CPUs easily? Remember that the important thing is how the system *
scales*. How much more load do we have when there are 20 players compared to
10, for example.

These are the notes I kept when we did the tests. I think we can discuss
these topics in the practice section of the report, once we've figured them
all out!

First attempt to get some statistics

Today was a great day, we have been able to fix the bugs which didn't allow us to run a bigger number of clients. After that we have tested to run more and more clients. At the end of the day we were able to run ca. 3000 clients and they all played pong with each other, we had too one normal player who played against a bot.

We had a look at the load of the computer which run the server and it was not really stressed but only up at a load of ca. 0.31. The problem seemed to be the network we have been working on, it was a ad-hock WLAN network between all our laptops which isn't the fastest. And since we use TCP which resends all the data packages which got lost we got quite a big amount of traffic and therefore a high round trip time ("lag" in game speek).

For the future we want to try to get a GB-network-switch and attach our computers via cable. It's a pitty that we will not have the time to implement UDP to get it working with lower lag.

Another bottleneck was running all the bots, which are written in ruby (and each bot is a real process), on our limited amount of laptops (we had only 3). Jonatan was able to start ca. 2500 on his linux machine but I, on OS X, was only able to start ca. 200, after that it said I was not allowed to start more processes. Therefore, before we will be able to do some real statistic stuff, we will implement the bots in erlang, processes there are cheap.

But it was a great feeling to see 1500 games of pong, it is indeed a real time game, running on our GGS-prototype without having performance problems other then the network-bottleneck.

Bots playing Pong

We got quite long with our GGS prototype now, I'm not sure if we'll be able to add the erlang<->js stuff because it seems not to work like we intend. Anyways, we have to focus on some statistics for now, so I just implemented a bot in ruby which can play the Pong game we already have written in erlang. So now we can run bots who play against each other Pong on our GGS prototype.

We have to do some more work before we can run thousends of games simultanously, but I think it will work next week or something. Then we will produce some nice statistics for our report.

You can take a look at the code, it is implemented in ruby and I devided it into different "modules" so we can reuse the GGSNetwork in other games too: https://github.com/jeena/GGS/tree/master/games/Pong-bots

Writing the report is working quite well too, we already have 15 of the minimum 30 pages with actual relevant content. We'll have to work on the language though.

Some rewrites, report, status

I'm working on some rewrites now, I am rewriting the ggs_player to be a gen_server because it is just much easier to handle and I am rewriting the ggs_protocol module because it didn't take in account that TCP is something like a stream and gen_tcp does not the work to seperate between different messages a client sends but only sends a callback when a tcp package arrives. Therefore sometimes only half of the data arrives and stuff like that. I had to add something like a accumulator to handle all of that.

Perhaps the more important thing is that we changed the direction of our thesis from "just" implementing a game server to researching about scalability and fault tolerance of game servers at a academic level. Currently we are reading many scientific reports and books about scalability and fault tolerance and try to be as scientific as possible when we write our report. You can actually take a look at it: https://github.com/jeena/GGS-report

We still want a working Generic Game Server but it won't be a real world product but more some type of a prototype of such one where we test our hypothesis on performance, fault tolerance and scalability if we need to get our hands dirty.

To be able to test such stuff we need to write some bots which will play games via our server. And we need to write code which collects data during the games.

To get the code more stable we are trying to replace our unit tests with quickCheck tests.

Hope the project is still interesting for you who have been following it until now :-)

Texts and presentations

It was a little bit quiet here on the blog for the last two weeks, but this doesn't mean that we had vacation or something. We have actually been writing a lot and I'd like just to load up the stuff here so you can read it if you want. This are links to PDF:s

• Planning report
• Usecases
• Half time presentations, (video 1, video 2 )
• (Brainstorming) Tools for developers

More fault tolerance & protocol improvements

Today I and Niklas sat down and took a look at improving the fault tolerance of our dispatcher process. Currently it (for some reason) crashes all of our clients when it goes down, and this is bad. We are looking at fault-proofing every subsystem of GGS, so in a worst case scenario we still wont lose any data, but all the clients will be disconnected when the dispatcher goes down.

The problem seems to have to do with the way sockets are implemented in Erlang, and I'm sure there is a way around this, we just need to find it. Other than looking at the dispatcher, I also changed all outgoing communication to also use our protocol, previously it did not. I wrote a very rudimentary protocol "message builder" for this purpose, and also a very very simple parser for python.

The chat client now uses the protocol both for receiving messages and also for sending them. As a step in the direction of game loops not written in Erlang, I redid the "identification system" for player and table processes. Previously we handled all communication using process IDs, which is really handy when doing game loops in Erlang, since you get a unique identifier, and at the same time a way to communicate with the process. When moving outside Erlang and into JavaScript for example, this is not as convenient.

I wrote a post about why we need UUIDs for describing players, and this is what I have implemented, both for tables and for players. Now all identification of tables and processes are in the form of UUIDs and not process IDs.

Fault tolerance & chat

Today we have seen some improvements in two different parts of GGS. I have extended the GGSChat program a bit, it now features nicknames stored in GGS via Mnesia (ggs_db), a list to show the nicks, and some other cosmetics.

This is interesting because this is the first usage of ggs_db, and I can conclude that it works nicely.

I have also added some fault tolerance to the ggs_coordinator. Now clients are unaffected by crashes in ggs_coordinator. This was mostly accomplished by adding a backup server which ggs_coordinator can restore it's old state from when it crashes, but also by changing the startup routine of new tables.

Previously we created new tables using gen_server:start_link, but this didn't work very well. ggs_coordinator becomes the parent process of all tables, and when ggs_coordinator goes down, it brought all the tables down with it. I solved this by using gen:server:start instead, and also I enable trap_exit to catch any exit messages.

Many features of GGS have now been tested using GGSChat, and next in line is the room creation feature. Currently all clients join the same room when GGSChat starts, I want to be able to specify a room. This makes the chat a bit more IRC-like.

Oh, here's how the chat looks now:

Brain storming

I sat down this morning and thought a bit more about which features we need to provide to game developers using GGS. I came up with some stuff that we hadn't written down previously, and also described our current things a bit more in detail.

One thing that we had discusses, but never really formalized was the idea of a rule enforcer, who checks if a rule is broken when a move is made, or rather, if a move is legal or not. Our supervisor had previously asked us if it would be possible to change the rules of the game while it was commencing, and this is a great way to do this, we think. The enforcer will have a list of all the rules, and checks each move against the rules, if a rule is broken, the enforcer will stop the player from performing the bad action. A bit like a policeman, or a judge in soccer.

The problem then becomes; how do we specify the rules of the game in such a way that they can be changed, and easily formulated? Mathias found a specification [1] for the GDL language, which in part does just this (and much more). Using a GDL dialect, or possibly a fully fledged GDL system, we could allow developers to form rules, and players to change some of them, on the fly.

The idea is that a developer writes the rules he/she wants players to be able to change (possibly all the rules) in GDL, passes them to the enforcer, and the player can then change them as she wishes.

On a side note, I will stop time-logging on the blog and do this on ping-pong instead.

Bulk update

A lot has happened, and this post should really be several posts; unfortunately it is not.

We now have an 'emulator' for our game virtual machine, which is in the future written in a DSL, but for now written in Erlang, lets call this emulator gamevm_e.

Gamevm_e implements the functions and protocol that will later be written and uploaded by game developers, things like placing chess pieces on a board. The purpose of gamevm_e is to be able to test our backing architecture before we have a proper vm in place, so in the future, gamevm_e will probably be removed, since we really want to use other languages, such as Javascript, or Lua for this.

We have during the end of the past week and this weekend managed to connect several Player processes to one Table process, thereby sharing one gamevm_e process. This has allowed us to communicate between players, and thereby also client. To test this functionality I developed a small chat client with its server side logic written in gamevm_e, in erlang.

We have also shifted more of our focus towards investigating which support libraries and helpers we need to provide to game developers. We have so far come up with things related to ranking systems, content distribution (for sending graphics, maps, etc to clients), perhaps micro-payment systems to allow charging players. More time will be spent investigating these things.

Time report:

From th 18/2 - 20/2 : Approximately 10 hours

Board game classifications

Strategic play: Choice is based on multiple turns. Example: Openings in chess. Tactical play: Choice is based on current turn only. Example: Having the knight perform check while simultaneously threatening the queen.

informal games: Undirected play. As in a playground.
formal games: Games with means and ends. where Means consists of are predetermined set of rules and equipment. Ends are a set goals to accomplish through contest.

Game board:
* A flat surface. Example: Chess
* Tile laying. Tokens stacked on top of each other.Example: Scrabble(Afapet in swedish)
* Boardless. Boards which isn't flat like 3D boards or no board at all. Example: Yatzy

Combinatorial games
Discrete: No simultaneous moves
Deterministic: No luck
Perfect information: No hidden information
Finite: Well defined outcome after first nr of moves

Race games: Players trying to be the first to reach from start to end. Example: Parcheesi
Space games: Arrange pieces into patterns. Developing and disrupting the enemy. Example: Three in a row
Chase games: Hunter and pray. Capturing and entrapping. Example: Cops & robbers.
Displace games: Capture a fixed nr of opponents pieces. Example: Fox games.

Variable geometry:
Fixed: Nr in a row, Target pattern. Example: 3 in a row
Variable: Pathmaking. Territorial. Reach a goal. China chess.

Board game decomposition

I tried figuring out the elements of boardgames. Hopefully it will be useful in future design decisions.
Source of inspation: Decoupling Aspects in Board Games Modeling

Logic:
This is the core of the game.
Agents:
Users of the game. May be humans or computers.
Representation:
How the game is presented to the user. Possibly with figures cards and tokens. Usually expresses the game logic, making it reasonable for the user.

States:
The conditions of the game at each turn.
Behaviors:
Transitions from state to state.
Rules:
Static laws producing the behaviors in the game.

Administrive agents:
Upholder of the rules of a game. Like the bank in monopoly.
Players:
Players playing the game.

Game Semantics

Used to study the behavior of simple programming constructs and create an abstract model. This model is then used to express game rules, strategies, terminologies etc.

Example: An expression written using a newly defined model compared to a lambda expression.

The reason is to have the simplest most well fitting syntax for the domain being studied. In this way it will be easy to understand expressions even as they become complex.

How could GGS benefit of game semantics? There probably exists optimal models to express all sorts of game behaviours. I believe you could have an application reading the history of gamestates and parses them with respect to such a model and then represent them to a user. These sorts of applications is not our main concern but could probably very easy be used in conjunction with GGS. Or maybe even built into the javascript game.

Another idea would be able to send such expressions to the game in runtime. Therefore enabling players to modify the rules of the game. Unfortunately GGS isn't powerful enough being able to support that idea. * You would need to translate whatever the game expressions are into java script. * The ability to reload the gamefunctions inside spidermonkey in runtime. * Alternatively having multiple virtual machines running simultaneously, cross communicating. This would be even harder as we would need to write our own vm and having them communicate with each other.

To summarize: Adding this kind of support to GGS would direct the focus from our current goals of not having to implement network support by oneself, to creating a game design tool with networking and communication support.

Semantics

I found some links about game semantics, game design languages and structures in board games which I've been looking into. The purpose is to get in touch with these subjects so that we will be able to find out what of this should be supported with GGS, what we will implement or what would easily be implemented by others. If not then give reasons and use the material while writing standards. Together with this I wrote a brief summary of the word Semantics for everyone to enjoy.

Semantics. The study of meaning according to wikipedia.

It is learning about the meaning of different words and symbols and how they together build up sentences which then has new meanings.

Applied to computer languages,this is learning about how they work and compare their syntaxes. Make abstractions of them and create mathematical models.

A bit of a rewrite

Today we realized that our old structure of GGS wasn't very good, so we decided to redo it! We have learned a lot developing the "old" structure of GGS, such as how to use supervisor structures properly and which the big parts of the application are.

Anyway.. Using this knowledge, we sat down and really thought through which interfaces we need internally. We also asked ourselves "what happens when this part of the application fails?", and provided solutions for this. In the end, I think we came up with a pretty nice solution.

We now represent all clients as separate processes, and we have a coordinator process which connects different clients with different games. The idea is that this coordinator process should be able to rebuild the entire server in a different location, and thereby easily provide replication.

We all sat down and created stubs for the new design and documented the design in this document, and I later sat down and implemented the basic supervisor structure and socket communication, just so we have something runnable.

I predict we will soon be up and running with the same amount of functionality we had with the previous design, much of the code was written in such a way that we can simply pop it in to the new design.

Time report:

Today: 8 hours

Yesterday: 5 hours

Joe Armstrong on Erlang lecture

Today we all have been on a lecture held by John Armstrong - who is one of the inventors of Erlang - on Erlang. The lecture seemed to have been made exactly for us with our problems.

Is Erlang the answer?
In that case what was the question?

Joe Armstrong
Programming languages are used to solve problems.
But what problems? Erlang solves a few interesting problems.
What problem does it solve, and why are these problems interesting?

What big problems are left in computer science and how might we solve them?

What's gone wrong with computer architectures and what can we do about it?

This talks looks back at the history of Erlang and forward to a future where one day software might actually work and be useful. Right now software is in a bit of mess, but there are ways to fix it ...

I'll tell you more next Tuesday, ...

/Joe

At first he talked a little bit about the history of Erlang, which was interesting but most of the stuff you already knew from books and so on.

Message passing and processes

He wrote a interesting paper about OOP a while ago: Why OO Sucks

One of his tutors said, after reading this "Why OO Sucks" paper, You got it all wrong. You know, Erlang is probably the only really object oriented language because the main thing is not at all the classes and inheritance and stuff, but message passing and code encapsulation. (I paraphrased here, can't remember the exact words he used).

Just yesterday I tried to explain to the other guys why we really should use asynchronous message passing instead of synchronous, I never could wrap what I was thinking and feeling about it into words. But lucky me, Joe did exactly that today for me, so I think we now agree on that we want asynchronous message passing for our server.

Another anecdote was that he once gave a talk in Germany about Erlang and some server programmer said that their server would serve data for about 10000 users he wouldn't need Erlang to acomplish that. Joe asked him: If something goes wrong in your server and it crashes, how many users does this affect, all the 10000? Our server only serves one user, but we start 10000 server processes to serve data to every user, when it crashes it only affects that one user. I must admit, it got me thinking.

Protocols

He also talked about protocols and that he never liked the fact that there are sooo many of them, he counted about 4900 or something like that. We felt like he was talking directly to us, because during the last two weeks we have been doing exactly that, designing a new protocol :-/.

In 2002 he proposed UBF, but as he states in his blog in february 2009, This scheme was never widely adopted - perhaps it was just to strange.... During the whole lecture he never mentioned a real world alternative to designing a own protocol, so during the Q&A session I asked him directly what we should use today instead of the protocol designed by ourselves.

He had to think about this question for some time and said That is a good question :D, but then he mentioned Google Protocol Buffers and Facebooks Thift. He said we should have a look at both and then just pick one, and we should really do that.

Realizing that this would be the right thing to do crossed my heart a little bit because I was a little bit proud that we had developed a simple to understand but flexible and powerful protocol.

So tomorrow we will have a closer look at Protocol Buffers and Thift and try to decide if one of them suits us good enough. I assume that we will rethink our structure and if we should use more processes or something. After all it was the perfect lecture in the perfect time for us.

Internal interfaces

Today we found out that there is no way to continue programming together without first defining some internal interfaces for our erlang modules. Every time someone added some functionality it broke the entire application because of the changed method calls.

So we decided to set up a new Wiki page: Interfaces

Untill now there are only three interfaces, the ones we are working on right now, but we want to add a interface for each of the modules we use. They will help us to structure the application and to easy understand what each module is supposed to do.

We tried to do that by the "Single responsibility principle", where each module only has one responsibility. If its doing more than one thing then obviously it needs refactoring.

Time: 10 hours

Improved protocol & test app

Today we had a meeting where we discussed the structure of the new protocol module. We ended up with a resursive solution which looks much better than the old one. This new module allows us to write more powerful handlers in ggs_server, so that's very good.

I've also written a little GTK application to demonstrate GGS, just a simple calculator. The important features of the calculator is the calculate button, and the connect button, which interact with GGS.

Time report
Approx. 8 hours today

Usage of javascript

JavaScript has gained a lot of popularity lately, it is used in large projects such as Riak[1], CouchDB[2], PhoneGap[3]. On the popular social coding site GitHub.com, 18% of all code is written in JavaScript.

When starting a new project, it is nice to have a low learning curve, to attract new users. Using a language that many people already know about means you get a lot of potetial users right away.

The popularity of JavaScript in the programming community, in combination with the availability of several[4] different JavaScript virtual machines was an important influence in choosing JavaScript as the main control language for GGS.

In addition to JavaScript being a popular language, both CouchDB and Riak mentioned above use Erlang.

References:
1. http://wiki.basho.com/An-Introduction-to-Riak.html
2. http://couchdb.apache.org/
3. http://www.phonegap.com/about
4. http://www.mozilla.org/js/spidermonkey/, https://wiki.mozilla.org/JaegerMonkey, http://code.google.com/p/v8/

Literature Quest

Here comes another time report. Today I haven't done much at all, but yesterday we did some good stuff. I assisted Jeena in trying to communicate with Erlang from C (and thus JS), and prior to that I read Joe Armstrong's thesis on Erlang, it's a really really good read for anyone wanting to know more about Erlang and OTP. I believe it was written some time around 2003, but it is according to my analysis still current.

I also read a report which was mainly focused on postmortem analysis of crashed Erlang applications / systems, and included a new tool to aid in this. This tool was cool and all, but what was really good about that report was that it also covered a good deal of how to properly write fault tolerant Erlang applications.

In total I spent around 6 hours

Exposing C-functions to Spidermonkey

Today I implemented my very first additional function into Spidermonkey, it was just a test function which just responds with a static string but anyhow, this is hot shit! We will use it to call Erlang functions from JavaScript to implement something like webstorage. here is some code, the js_erlang() function which will be exposed to JavaScript:

JSBool js_erlang(JSContext *cx, uintN argc, jsval *vp) {
  const char *s = "text comes from C function";
  JSString *str = JS_NewStringCopyN(cx, s, sizeof(s));
  JS_SET_RVAL(cx, vp, STRING_TO_JSVAL(str));
  return JSVAL_TRUE;
}

And here is how you add it to all the other native functions in Spidermonkey:

JSNative *js_erlptr = (JSNative *) *js_erlang;
JS_DefineFunction(vm->context, JS_GetGlobalObject(vm->context), "callErlang", js_erlptr, 0, JSFUN_FAST_NATIVE);

Because we wanted to add this, we had to fork the erlang_js project, I hope we can write code which is general enough so it will be back ported to the original project. Here is our version.

The next step is to get the argument from the function, I suppose it'll look something like that:

jsval *argv = JS_ARGV(cx, vp);
jsval js_arg = argv[0];
char *argument = JS_GetStringBytes(JS_ValueToString(cx, js_arg));

And after that we have to call a erlang function. Jonathan found this neat erlang helper called erl_call. With its help you can call functions in a node:

me@Zepto$ echo "[2+2, node(), \"Hello world\"]." | erl_call -sname ggs -e
{ok, [4, ggs@Zepto, "Hello world"]}

And you get an answer from the erlang node in plaintext. This looks like the easiest way to talk to erlang yet. You probably saw the argument "-sname ggs" and wandered what that is. It is the name of the node you would like to talk to, here "ggs".

Time report

Today: 7 hours

Fault tolerant GGS

Now we have some fault tolerance in GGS. Currently the server 'crashes' when a client exits, and this causes a new socket to be opened, which is good, because the old one was closed by the client.

The {ReferenceID, JSVM} tuple stored in the ggs_server state was previously lost when the ggs_server crashes, but this is now stored in ggs_backup as well, and this is where the fault tolerance takes place.

ggs_server may now crash at pretty much any time, and resume its state from ggs_backup. This works according to the following:

1. ggs_sup starts
2. mnesia_ctrl starts
3. ggs_server_sup starts
4. ggs_backup starts
5. ggs_server starts

In 5. when ggs_server starts, it always asks ggs_backup for any previous state, even if this is the initial start of the whole supervisor hierarchy. When the application is first run, there is no state, and therefore ggs_backup returns an atom signaling that it has no state to back up from, and that ggs_server should initialize a new state and back it up.

If ggs_backup is to crash, the ggs_server_sup will restart it, just as it restarts ggs_server, and the state is reloaded from ggs_server, unless both ggs_server and ggs_backup crash at the exact same time.

For some reason though, the JSVMs stored in the ggs_server state do not live through the migration to ggs_backup and back, most likely due to them being spawned from inside ggs_server and not being supervised properly. This problem should be easily solved by supervising erlang_js on a higher level.

UPDATE: Moving erlang_js to a higher hierachy is not the solution to the JSVM death issue. The JSVMs disappearing when ggs_server crashes is a design choice coming down to the new/3 function in js_driver.erl in erlang_js.

new/3 creates a connection to the erlang_js driver using an open_port command, and this command registers the calling process (ggs_server) as its port owner. Looking at the documentation for ports we can see that when the port owner dies, the port is supposed to go down as well. I see two possible solutions to this:

1. Change the owner of the port to ggs_backup when ggs_server crashes, and then back again
2. Set the owner of the port to a process which should not ever go down (thanks)

Now, according to the SO link above, option 2 is hacky, so option 1 is to be preferred. This means we start yet another gen_server process somewhere in the hierachy, which acts somewhat like a wrapper for erlang_js. This is probably the future of js_runner.

Update 2: A different approach is to incorporate the "wrapper" inside erlang_js so we don't need to see it, we just do our js:call/? calls and those are handed of to a "persistent" process inside erlang_js which registers its JSVMs to that process.


Time report
Today: 5 hours
Updating day: 1 hour

Protocol changes

Now I have implemented much of the new protocol Jeena proposed. GGS has gotten to the point where it is possible to define a function with an arbitrary name, and run the same function. A session is maintained via a token that is passed back and forth between client and server.
We still have not implemented unique IDs for the users, so if more than one user connected at once, there'd be a 0.1% chance that they get the same ID!
The javascript running code is really simple at the moment, and Jeena is working on improving it.
The supervisor tree graphic that was shown in the last post is now implemented. We have an mnesia controller which we can use to communicate with mnesia, we still only have one ggs_server though.
I and Niklas also went to the library to learn more about their databases, it was very informative and will certainly prove to be a great resource.

Time report
Yesterday: 3 hours (including meeting w/ supervisor)
Today: 4 hours, working on the protocol, 2 hours in library = 6 hours total

Supervisor

Now the supervisor is active and working in our little GGS. Currently there is one root supervisor, which supervises the ggs_server process. Once the ggs_server process dies (which can be forced by a __crash command), the supervisor restarts it.

The problem is that the ggs_server process holds valuable state, which we need to propagate to other ggs_server processes for reliability. The next step is figuring out how we can run two (or more) completely mirrored ggs_server processes, and bringing one down doesn't wring the other down.

The network part of the ggs_server module should also be moved out to ggs_network, similar to how the application level protocol is placed in ggs_protocol. The actions taken by ggs_server when a message is received are currently not implemented in an "OTP fashion" and should be rewritten as such (should be simple).

Yesterday, we spent some time discussing and implementing the protocol which we use to communicate to the clients and back to the server again. This ended up as the ggs_protocol module. Also, yesterday night (this morning..) I implemented {client, JS-VM} mappings in an OTP fashion in ggs_server.

The picture below shows the supervisor setup we are currently working towards. The tree depicts the GGS system running on one machine, when run on several machines, we need to link several of these trees together

Time report Yesterday (day): 4 hours Yesterday (night): 2 hours Today: 3 Hours

Randomness and Identifiers

We have a somewhat new problem to consider in our project. Picture GGS as a 'cloud' of computers, where we can add and remove computers as we wish. Now consider adding a game client (player) on node A of the cloud. At the same time, add a new player on node B. These two players need to be distinctly identifiable from each other. This means, we can not simply give each player a pseudorandom number from 1-1000, or even a sequential number from 1-1000, because we will have clashes after a while. To solve this, we first considered using Erlang's make_ref() function, however, we encountered several issues with this:

• The refs are not really intended to be sent over network in binary form (or so it seems at least)
• When converted to binary, they are quite large
• According to [1], they are unique to "approximately 2^82 calls" - which is a bit vague.

So, looking beyond make_ref(), we found UUIDs, which have more appealing properties:

• It is designed to be used over a network [2]
• It is designed to be used for objects with a long life [2]

We will proceed the project with UUIDs as out identifiers, both internally in the system, and externally towards the clients. If needed, we could even salt the UUIDs a bit, but I personally don't think we will need to. References

1. http://www.erlang.org/doc/man/erlang.html#make_ref-0
2. http://www.opengroup.org/dce/info/draft-leach-uuids-guids-01.txt
• Section 3.0

GGS simulator in JS

First I wanted to code a counterpart to Mathias Python-Tic Tac Toe in JavaScript, so we could test our Server and work with two different platforms for the game, but then I ended up writing a GGS-simular in JavaScript.

Doing so I found many WTF?:s for example having just a array with users and no getUserForId() method is retarded, every time you do something you need to iterate thru all users to find the right one.

I tried to work with the world object, the localStorage object and the users array, in order to not pollute the space I bundled them all in one GGS object.

One very odd thing came up, if you're used to programming object oriented, you use many instance variables and save there objects, etc. You can't do that if you want your game to go on after your spidermonkey instance crashed. I'm not sure how people will react to such a shortcomming.

Here is how a game on our Generic Game Server could look like: TicTacToe.js.

You can take a look at the server-simulator too, It doesn't work in IE GGS simulator

There are two instances of the game in different windows (frames), the do not talk to each other directly but via the GGS API and the simulator. In the future I want to add some stuff like random game-crashes and random latency.

Time report

Today: 14 hours

Webstorage

Today me and Mathias spend the whole day trying to figure out how to fix the problems with the JavaScript API I talked about the other day. node-overwrite doesn't work for us, because it doesn't have to do much with the way we do things and only works for node.js

world-object

The main problem we had was that we wanted someway to map a erlang dictionary to a JavaScript object but couldn't figure out how to expose a host object to the JS-VM. Then Mathias said that it can't be possible to do it in a easy way because there are no variables in erlang, erlang is a functional language and you alway just get copies. Put it that way, even I got it why no one has done that before.

So we had to do it the erlang way, get a copy of the data, modify it and send a new copy back to erlang. I recalled a W3C specification called HTML5 Webstorage which tries to deal with exactly the same problem, to let the JS-Programmer to store data locally. This is their interface:

interface Storage {
  readonly attribute unsigned long length;
  getter DOMString key(in unsigned long index);
  getter any getItem(in DOMString key);
  setter creator void setItem(in DOMString key, in any value);
  deleter void removeItem(in DOMString key);
  void clear();
};

As you can see, it is not possible to store hierarchal data like Arrays or Hashes. So if you't like to do that, you'd have to serialize and deserialize the Array/Hash. At first we thought that this was a very dumb idea because it would take way too much time to do that. But then we had a look at the erlang_js implementation, because we wanted to know how they communicate between erlang and the JS-VM. Astonishingly they just use plain JavaScript stings via a socket to talk to a C-application which holds a instance of Spidermonkey and which just executes the string in Spidermonkey and if there is a return, returns this as a string back to erlang via this open socket connection.

Ok, now there was really no need to try to make it more complicated as it had to be and we decided that if you would like to store structured data like arrays or hashes in our key-value-localStore you'd have to use JSON.stringify() and JSON.parse().

That way we can offer a really easy to use API which people know from the JavaScript-Web-world. There is a good specification of that, example code en mass and there are many people you can ask. And we like the idea of using as many already available standards as possible.

Public and Private data

Obviously every time you use the setter or deleter method we will notify all clients about that. But what if you just want to save something which should not be distributed to all clients like some counter or other private data which no client should know about? Then we will, like the W3C proposes, offer a localStorage object which too saves the data persistently, but does not distribute it to all clients.

Last but not least there will be a Users array with all of the users who are logged in into this game. Every user-object will have at least an id, a nickname, and methods to talk to just to his client.

Time report

13 hours

Gomoku

I just realized that the way I designed Tic-Tac-Toe makes it possible to change the screen size and the grid size from 3x3 to 19x19. Now you have Gomoku. Both games could share the algorithm that evaluates winning conditions later on.

Tic-Tac-Toe

Our first test game is almost done. It has been commited to the project just before this writing. It should be easy to seperate parts of the game to make it a network game instead of hotseat. I've taken care of some safety aspects as of making all turns final and not letting the same player do the same turns twice. I've also added a tiny "generic" inputmanager for flexible implementation of all kinds of events. I'm thinking about leaving the game behind for now and instead focus on the important parts.

Trying out Google V8

Jeena pointed out today that Google V8 may be a good alternative to SpiderMonkey, and it indeed seems so. In short, here is how we can implement callbacks in JS->Erl using erlv8 I have forked the repo Jeena hosts on GitHub to one I host, while playing with V8.

Time Report
Jan 26, we had a meeting between 13-15, so 2 hours
I worked with V8 between 00.30-02.00, so 1.5 hours
In total: 3.5 hours.

Problems with key-value store via JavaScript API

Problems with key-value store via JavaScript API

For security, concurrency and high scalability reasons we wanted to expose a key-value store, which would be implemented by us in Erlang and use Mnesia, as a convenient API for the game programmers in JavaScript.

I wanted for example have a "world" object, like for example the "document" object in browsers. You should be able to use this object like every vanilla JavaScript object, except that it would not store functions, only primitive objects {}, arrays [], strings "foo" and numbers 1.63.

You would set and get objects like this:

world.foo = "abc";
console.log(world.foo) // returns "abc"
world.bar = { "myArray": [99,33,1] };
console.log(world.bar.myArray[1]) // returns 33

and so on. Every time you save something in this object, we would need to call a setter method which would call a erlang function which would save the data in our erlang-database. The problem is, that this seems not possible and easy as I thought in todays JS-VMs.

At first I thought about getters and setters in JavaScript, but if you have a look at the code, then you see that you already have to know the name of your properties.

What we would like to have are something like "default getters and setters" which would be called even if the property isn't there yet. There is the non standard __noSuchMethod__(id, args) in Mozillas spidermonkey, but it only is called on methods world.undefinedFun(), not on properties like world.undefinedProperty123 = "test".

I talked to the guys in the irc://irc.epd-me.org/#selfhtml, they often know a lot about JavaScript, and got a link to something very promissing, but sadly not implemented yet, harmony proxies (bugtracker) in Googles V8 JavaScript engine. In the bugtracker comments you find a link to node-overload which is node.js addon that provides an Watchable class that has getters/setters without property names. And node.js is a server running in V8. I don't know if we'll be able to use it, first need to have a deeper look at it, but it looks like it would do what we want.

Time report

16 hours

Hello Mnesia!

The first lines of code concerning Mnesia was added today. So far it's not much more than a stub consisting of a three files. One erlang header to store the structure of a player table. One erlang source file for creating a player table, inserting an example player into the table and ask the table for the play element for printing. One textfile to help initialize Mnesia with the proper terminal and erlang commands together with this newly added peaces of code. All of which can be found in the newly added Mnesia directory inside the master branch of the GGS project. Mnesia seems very straight forward but a little low level as all calls to access the data in the database need to be implemented manually. This in other hand turns out to be an advantage for us as we intend not to follow the general rules of database management but to have games store information in runtime. Next step in the process is to design a generic database structure that suit our needs. This should be further discussed within the group later on but a recursive hashmap has already been suggested. How the Mnesia part will be integrated with the rest of the project isn't all too clear either. As pieces of code already has been added by different group members, it is time to agree on a code convention.

Preliminary thoughts about the planning report!

In todays meeting we talked about how we should design the planning report and which things to focus on when we write it.

The main motivation behind the project are to facilitate the development of network based games. In order to achieve this we need high scalability and high reliability(one reason to use Erlang). We also want to see how Javascript(JS) works in combination with Erlang, will it be extensive enough and provide the required stability? Another thing that we need to investigate are the connection JS->Erlang, this is currently not supported by erlang_js and before we choose what to use we will need to look at other alternatives.

With the project we want to deepen our understanding of Erlang, JS and how we can make links between different programming languages. We aim to build a reliable server that will make use of the possibilites that Erlang together with OTP gives us. It will also be interesting to see if it is possible to run a distributet game server in reality?

We want to give game developers the possbility to develop small multiplayer games easy and quick without the need to have knowledge about networks and network programming. The project will have a clear link to web applications/web games. We will strive to make it as easy as possible for the game developers, we hope to create an API that is easy to use.

Meeting with tutor, and first code session

Today was very productive! Our tutor suggested we got started with the coding, something we had not planned to do just yet, but turned out very productive. The general idea was to work out the rough part of the application to easier be able to plan the project. In an earlier post I supplied some sample code for connecting Erlang to Spidermonkey using erlang_js, today we integrated this code with an OTP server. This means we can execute javascript on our OTP server and return the result to the callee. The earlier sample code evolved into two functions residing in js_runner.erl which will be responsible for the execution of javascript. When executing the javascript, the js:call function expects all parameters to be in binary form, which became a bit of a headache. The client sends data in UTF8 form to the server, and the server must then somehow convert this UTF8 data to a binary string, "abc" becomes <<"abc>> in other words. The problem we had with this was that term_to_binary/1 does not support unicode or UTF8. Later we discovered the unicode library, and also that list_to_binary/1 does support unicode. Currently the server uses TCP for communication, mainly because it is easier to test. We use telnet, and a custom client developed by Jeena in Ruby. This client is basically a mock game for our game server, and in the future, games will use UDP to talk to the server. We have also identified that we should use Mnesia for our data store, and not as previously stated ETS. ETS is not suitable in a distributed environment while Mnesia is. This work has not yet begun, however Mathias has taken it upon himself to investigate it and produce some sample code. Work continues on the SpiderMonkey side regarding the JS -> Erlang bindings, we are investigating whether we should implement this in erlang_js or work around it some other way.

Time Report
We spent today, from 11 - 16.30 (5.5h) working on this.

About erlang callbacks from within SpiderMonkey

Now, about those callbacks.. We are planning to use ETS to store data. This data is to be manipulated from the JSVM. There is no support (according to Sean Cribbs, again, great guy, very helpful) for calling Erlang code from inside the JSVM (SpiderMonkey). The proposed way to do this is to implement a C(++) function (example code), which will interface with our Erlang code (I believe I have seen this been done - I'll drop an anchor if I find it, EDIT: Found it. See link.) and then call this C function from inside our JS. That is: JS -> C -> Erlang From a speed point of view, I don't really see this as really slow if it's done right. All we would need is a small C application which somehow interacts with our Erlang VM. The implementation of this C code is dependent on SpiderMonkey, and has nothing to do with our erlang_js bindings. The erlang_js fellows seem to be defining their own native functions for use within the JSVM. There might be a possibility to extend erlang_js with our native functions as well, and thereby not need additional C/++ code, but route everything through erlang_js. See here for example.

Using erlang_js

This post concerns installing and using erlang_js, together with some sample code to demonstrate how to run and extract data from SpiderMonkey using erlang_js. When installing erlang_js, you need a current version of Erlang. R14b04 (PPA) according to Sean Cribbs (#riak @ Freenode), this is the version I used, previous versions from 2009 do not seem to work. Initially, I used a mercurial repo to fetch the erlang_js code, but I was later informed that I should use the GitHub repo instead, perhaps this is more up to date. In order to execute javascript, we need (according to erlang_js docs):

1. Make sure the sasl application is running.
2. Start erlang_js.
3. Create a Javascript VM via js_driver:new/0.
4. Use the js module to execute Javascript.

For (1), this means we should supply -boot start_sasl to the erl commandline.(2) means we should add the path to the erlang_js binary to the erl path with -pa, and then execute erlang_js:start(). This amounts to: erl -boot start_sasl -pa ebin -eval 'erlang_js:start().' Now we are ready to run javascript! To run this code, start erlang with: erl -boot start_sasl -pa ebin, compile the example code with c() and simply run js_test:run(). Oh, by the way. I have not found the erlang_js docs online, you should build them with build_docs.sh in the erlang_js source directory.

Fourth meeting

meeting time: 12:30 - 15:30 _Server interface_ - Easy to use - Powerful - Slow communication(Low proiority) - Fast communication(High priority) - Send key:values (nick, pms) - Register callback (serverside) ball is moving, update graphics - Inlogg med cookies or passwords - Master Cookie, optional - loggout /shutdown same cookie as loggin - upload methods - spidermonkey _Internal_Interfaces_ - Javascript as pure text in memory - access the world through a global object - Store the world as ETS(erlang) JS -> ERL -> ETS ->JS -> ... _World_API_ API usage example: users = [users] world.p1.px = 2 world.foo.__set(2) user.send(2) world.foo.__setter = function (a) { erl_set("foo", 2) } Safety issues: - Bad javascript - No IO - ETS Communication requirements: Communication between everyone, single or multiple clients. _Server_interface_ runtime flow: 1. _Initialization_phase_ 2. Message_phase_ -dictionary(JSON,PList, ...) 3. _User_login/logout_

Which interfaces and API:s do we need? (meeting 3)

Server interface 1. Initiation phase (init the whole game, etc.) 2. User login/logout 3. Messaging phase, events from client to server and from server to client World API (JavaScript) In Spidermonkey we don't have the objects a browser exposes like document or window. But we'll have our own like "user" and "world". We will implement some functions we think are important but not present in JSAPI like setTimeout(). The user object will only be a readable array of all users with their data. All the data is saved in erlang as a dictionary. The world object will be read and writable, here too all the data will be stored in erlang. Spidermonkey allows you to use your own setters and getters, that way it will be no problem for us to use them to store the data in erlang like for example: world.a = 1; which would call a setter we implement which talks to erlang and stores the key value pair. That way we'd have a really nice and clean JavaScript-like API. Internal interfaces This needs to be evaluated further, but we want to use our own spidermonkey shell for security reasons and pipe between it and our erlang server. We will save all the worlds and users and stuff in a ETS instance One of the main goals is to get everything non blocking so we can serve huge amounts of clients and games.

Time report

9 hours

We need an API

Me (Jeena), Mathias and Niklas talked about how an API could look, we aren't really ready but this is what we talked about: There are two JS-Objects we introduce: "Users" which is a array of User objects "World" which is a key-value store and introduces two methods: getVal(key) which only gets the value for a key from erlang and returns it setVal(key, val) which does a little bit more, it sets the value for the key in our erlang key-value store and it saves the key in a erlang list with keys which have been updated. and there is a notifyAllClients() method which loops thru the updated list in erlang, packs all the changes into a one notification package and sends this package to all clients. This is only to not to have update all clients for every setVal() call but buffer it if needed. There is one more thing, our protocol needs two different action types. One when a client wants to instantiate a game; it sends the world in JSON format and sends all functions/methods as a (JavaScript) String which we evaluate and add to our spidermonkey. Then the other would be when they want to use the functions to change the world. Cloning and Spidermonkey I was just talking to the spidermonkey developers at irc.mozilla.org#jsapi They say there is no way to clone a spidermonkey instance for various reasons like that there is no clone method for user defined objects, etc. And they said that we should not use their js-shell because they use it on their self for debugging and it is really unsafe. One of them suggested we could just delete a bunch of unsafe functions from js/src/shell/js.cpp Another one suggested we write our own shell with just the functions we really need and he said it'd be easy to do if we'd have a look at their shell while doing it. But I think if we use a API for saving the data in erlang then we wouldn't need to clone spidermonkey, just start up a new one and send in the js-code we got at the beginning.

Time report

8 hours

Our second meeting

We have decided that SpiderMonkey is a good way for us to run javascript. erlang_js seems to be a good way to interact with javascript. We also found out that SpiderMonkey will not supply us with the browser features it usually has, and this is good, because we do not want them. In addition to SpiderMonkey, we should consider NodeJS, it may have other features, such as communication via sockets which we can use. Socket communication may be better suited than pipes. The communication between client and server (and the other way around) needs to be fast. We have decided to use UDP as our underlying protocol, but the protocol on top of UDP should be swappable via a protocol module. We have a few candidates for these "application level protocols"

• JSON
• Binary PLIST
• BSON

JSON is good because

• It is easy to construct
• easy to read
• readily parsable in javascript

It is bad because

• There will be a lot of overhead

Now, on the other hand, the two other protocols are quite small, but they are difficult to construct, and there's no built in support for them in javascript. We decided to make the server protocol independent, and to create at first a module for JSON allowing us to send messages back and forth in an easily debuggable manner. The lower levels of networking will be handled outside the JSON/other-protocols module. We need to find a way to replicate SpiderMonkey runtimes. We plan to keep at least two runtimes alive, and if one crashes, we should replicate it and produce an exact copy. We have not yet decided how to do this.

First meeting

Protocol: Stateful protocol JSON UDP Store session id (sockets) Game node hierarchy: Game → Group → Player (perhaps a lobby would be needed) Different game categories: Chess Farmville Quake We are about to focus to support more games like Chess and Farmville and less on Quake. Project is going towards a generic SpiderMonkey Server than a Generic game server. Is the server going to have one SpiderMonkey instance or multiple? More means safety. We strive for hundreds but not more. Libraries: OTP (Security) SpiderMonkey (Mozilla's C implementation of JavaScript)

Time report

8 hours

About this blog

This is "The Generic Game server development blog". We are a group of students at the Gothenburg University in Sweden and this is our Bachelor-Thesis blog. We will write here along mostly for ourselves so we don't forget what we were doing all the time. Background Currently it is very difficult to roll your own game server. Using our system a game developer should be allowed to focus more of his or her efforts on the actual game, and leave the server part to our software. Project description The general idea is to model your state, in an appropriate format (e.g. JSON, Plist, custom binary, etc.), supply modifying functions (such as game time, or gravity) and push this model to our server, which will keep track of the world. The server will be implemented in Erlang, for concurrency purposes and use JavaScript (or similar) as a DSL to let the client modify the world state. The server will allow clients to push data along to other participating clients (much like a chat), and to modify the game state. Using this model, of pushing either state or pushing messages allows for a very versatile server capable of many things. One could build a chat server (IRC), or a world server (World of Warcraft), etc.