Sunday, January 30, 2011

GGS simulator in JS

First I wanted to code a counterpart to Mathias Python-Tic Tac Toe in JavaScript, so we could test our Server and work with two different platforms for the game, but then I ended up writing a GGS-simular in JavaScript.

Doing so I found many WTF?:s for example having just a array with users and no getUserForId() method is retarded, every time you do something you need to iterate thru all users to find the right one.

I tried to work with the world object, the localStorage object and the users array, in order to not pollute the space I bundled them all in one GGS object.

One very odd thing came up, if you're used to programming object oriented, you use many instance variables and save there objects, etc. You can't do that if you want your game to go on after your spidermonkey instance crashed. I'm not sure how people will react to such a shortcomming.

Here is how a game on our Generic Game Server could look like: TicTacToe.js.

You can take a look at the server-simulator too, It doesn't work in IE GGS simulator

There are two instances of the game in different windows (frames), the do not talk to each other directly but via the GGS API and the simulator. In the future I want to add some stuff like random game-crashes and random latency.

Time report

Today: 14 hours

Friday, January 28, 2011

Webstorage

Today me and Mathias spend the whole day trying to figure out how to fix the problems with the JavaScript API I talked about the other day. node-overwrite doesn't work for us, because it doesn't have to do much with the way we do things and only works for node.js

world-object

The main problem we had was that we wanted someway to map a erlang dictionary to a JavaScript object but couldn't figure out how to expose a host object to the JS-VM. Then Mathias said that it can't be possible to do it in a easy way because there are no variables in erlang, erlang is a functional language and you alway just get copies. Put it that way, even I got it why no one has done that before.

So we had to do it the erlang way, get a copy of the data, modify it and send a new copy back to erlang. I recalled a W3C specification called HTML5 Webstorage which tries to deal with exactly the same problem, to let the JS-Programmer to store data locally. This is their interface:

interface Storage {
  readonly attribute unsigned long length;
  getter DOMString key(in unsigned long index);
  getter any getItem(in DOMString key);
  setter creator void setItem(in DOMString key, in any value);
  deleter void removeItem(in DOMString key);
  void clear();
};

As you can see, it is not possible to store hierarchal data like Arrays or Hashes. So if you't like to do that, you'd have to serialize and deserialize the Array/Hash. At first we thought that this was a very dumb idea because it would take way too much time to do that. But then we had a look at the erlang_js implementation, because we wanted to know how they communicate between erlang and the JS-VM. Astonishingly they just use plain JavaScript stings via a socket to talk to a C-application which holds a instance of Spidermonkey and which just executes the string in Spidermonkey and if there is a return, returns this as a string back to erlang via this open socket connection.

Ok, now there was really no need to try to make it more complicated as it had to be and we decided that if you would like to store structured data like arrays or hashes in our key-value-localStore you'd have to use JSON.stringify() and JSON.parse().

That way we can offer a really easy to use API which people know from the JavaScript-Web-world. There is a good specification of that, example code en mass and there are many people you can ask. And we like the idea of using as many already available standards as possible.

Public and Private data

Obviously every time you use the setter or deleter method we will notify all clients about that. But what if you just want to save something which should not be distributed to all clients like some counter or other private data which no client should know about? Then we will, like the W3C proposes, offer a localStorage object which too saves the data persistently, but does not distribute it to all clients.

Last but not least there will be a Users array with all of the users who are logged in into this game. Every user-object will have at least an id, a nickname, and methods to talk to just to his client.

Time report

13 hours

Gomoku

I just realized that the way I designed Tic-Tac-Toe makes it possible to change the screen size and the grid size from 3x3 to 19x19. Now you have Gomoku. Both games could share the algorithm that evaluates winning conditions later on.

Tic-Tac-Toe

Our first test game is almost done. It has been commited to the project just before this writing. It should be easy to seperate parts of the game to make it a network game instead of hotseat. I've taken care of some safety aspects as of making all turns final and not letting the same player do the same turns twice. I've also added a tiny "generic" inputmanager for flexible implementation of all kinds of events. I'm thinking about leaving the game behind for now and instead focus on the important parts.

Thursday, January 27, 2011

Trying out Google V8

Jeena pointed out today that Google V8 may be a good alternative to SpiderMonkey, and it indeed seems so. In short, here is how we can implement callbacks in JS->Erl using erlv8 I have forked the repo Jeena hosts on GitHub to one I host, while playing with V8.

Time Report
Jan 26, we had a meeting between 13-15, so 2 hours
I worked with V8 between 00.30-02.00, so 1.5 hours
In total: 3.5 hours.

Wednesday, January 26, 2011

Problems with key-value store via JavaScript API

Problems with key-value store via JavaScript API

For security, concurrency and high scalability reasons we wanted to expose a key-value store, which would be implemented by us in Erlang and use Mnesia, as a convenient API for the game programmers in JavaScript.

I wanted for example have a "world" object, like for example the "document" object in browsers. You should be able to use this object like every vanilla JavaScript object, except that it would not store functions, only primitive objects {}, arrays [], strings "foo" and numbers 1.63.

You would set and get objects like this:

world.foo = "abc";
console.log(world.foo) // returns "abc"
world.bar = { "myArray": [99,33,1] };
console.log(world.bar.myArray[1]) // returns 33

and so on. Every time you save something in this object, we would need to call a setter method which would call a erlang function which would save the data in our erlang-database. The problem is, that this seems not possible and easy as I thought in todays JS-VMs.

At first I thought about getters and setters in JavaScript, but if you have a look at the code, then you see that you already have to know the name of your properties.

What we would like to have are something like "default getters and setters" which would be called even if the property isn't there yet. There is the non standard __noSuchMethod__(id, args) in Mozillas spidermonkey, but it only is called on methods world.undefinedFun(), not on properties like world.undefinedProperty123 = "test".

I talked to the guys in the irc://irc.epd-me.org/#selfhtml, they often know a lot about JavaScript, and got a link to something very promissing, but sadly not implemented yet, harmony proxies (bugtracker) in Googles V8 JavaScript engine. In the bugtracker comments you find a link to node-overload which is node.js addon that provides an Watchable class that has getters/setters without property names. And node.js is a server running in V8. I don't know if we'll be able to use it, first need to have a deeper look at it, but it looks like it would do what we want.

Time report

16 hours

Hello Mnesia!

The first lines of code concerning Mnesia was added today. So far it's not much more than a stub consisting of a three files. One erlang header to store the structure of a player table. One erlang source file for creating a player table, inserting an example player into the table and ask the table for the play element for printing. One textfile to help initialize Mnesia with the proper terminal and erlang commands together with this newly added peaces of code. All of which can be found in the newly added Mnesia directory inside the master branch of the GGS project. Mnesia seems very straight forward but a little low level as all calls to access the data in the database need to be implemented manually. This in other hand turns out to be an advantage for us as we intend not to follow the general rules of database management but to have games store information in runtime. Next step in the process is to design a generic database structure that suit our needs. This should be further discussed within the group later on but a recursive hashmap has already been suggested. How the Mnesia part will be integrated with the rest of the project isn't all too clear either. As pieces of code already has been added by different group members, it is time to agree on a code convention.

Preliminary thoughts about the planning report!

In todays meeting we talked about how we should design the planning report and which things to focus on when we write it.

The main motivation behind the project are to facilitate the development of network based games. In order to achieve this we need high scalability and high reliability(one reason to use Erlang). We also want to see how Javascript(JS) works in combination with Erlang, will it be extensive enough and provide the required stability? Another thing that we need to investigate are the connection JS->Erlang, this is currently not supported by erlang_js and before we choose what to use we will need to look at other alternatives.

With the project we want to deepen our understanding of Erlang, JS and how we can make links between different programming languages. We aim to build a reliable server that will make use of the possibilites that Erlang together with OTP gives us. It will also be interesting to see if it is possible to run a distributet game server in reality?

We want to give game developers the possbility to develop small multiplayer games easy and quick without the need to have knowledge about networks and network programming. The project will have a clear link to web applications/web games. We will strive to make it as easy as possible for the game developers, we hope to create an API that is easy to use.

Tuesday, January 25, 2011

Meeting with tutor, and first code session

Today was very productive! Our tutor suggested we got started with the coding, something we had not planned to do just yet, but turned out very productive. The general idea was to work out the rough part of the application to easier be able to plan the project. In an earlier post I supplied some sample code for connecting Erlang to Spidermonkey using erlang_js, today we integrated this code with an OTP server. This means we can execute javascript on our OTP server and return the result to the callee. The earlier sample code evolved into two functions residing in js_runner.erl which will be responsible for the execution of javascript. When executing the javascript, the js:call function expects all parameters to be in binary form, which became a bit of a headache. The client sends data in UTF8 form to the server, and the server must then somehow convert this UTF8 data to a binary string, "abc" becomes <<"abc>> in other words. The problem we had with this was that term_to_binary/1 does not support unicode or UTF8. Later we discovered the unicode library, and also that list_to_binary/1 does support unicode. Currently the server uses TCP for communication, mainly because it is easier to test. We use telnet, and a custom client developed by Jeena in Ruby. This client is basically a mock game for our game server, and in the future, games will use UDP to talk to the server. We have also identified that we should use Mnesia for our data store, and not as previously stated ETS. ETS is not suitable in a distributed environment while Mnesia is. This work has not yet begun, however Mathias has taken it upon himself to investigate it and produce some sample code. Work continues on the SpiderMonkey side regarding the JS -> Erlang bindings, we are investigating whether we should implement this in erlang_js or work around it some other way.

Time Report
We spent today, from 11 - 16.30 (5.5h) working on this.

Friday, January 21, 2011

About erlang callbacks from within SpiderMonkey

Now, about those callbacks.. We are planning to use ETS to store data. This data is to be manipulated from the JSVM. There is no support (according to Sean Cribbs, again, great guy, very helpful) for calling Erlang code from inside the JSVM (SpiderMonkey). The proposed way to do this is to implement a C(++) function (example code), which will interface with our Erlang code (I believe I have seen this been done - I'll drop an anchor if I find it, EDIT: Found it. See link.) and then call this C function from inside our JS. That is: JS -> C -> Erlang From a speed point of view, I don't really see this as really slow if it's done right. All we would need is a small C application which somehow interacts with our Erlang VM. The implementation of this C code is dependent on SpiderMonkey, and has nothing to do with our erlang_js bindings. The erlang_js fellows seem to be defining their own native functions for use within the JSVM. There might be a possibility to extend erlang_js with our native functions as well, and thereby not need additional C/++ code, but route everything through erlang_js. See here for example.

Using erlang_js

This post concerns installing and using erlang_js, together with some sample code to demonstrate how to run and extract data from SpiderMonkey using erlang_js. When installing erlang_js, you need a current version of Erlang. R14b04 (PPA) according to Sean Cribbs (#riak @ Freenode), this is the version I used, previous versions from 2009 do not seem to work. Initially, I used a mercurial repo to fetch the erlang_js code, but I was later informed that I should use the GitHub repo instead, perhaps this is more up to date. In order to execute javascript, we need (according to erlang_js docs):
  1. Make sure the sasl application is running.
  2. Start erlang_js.
  3. Create a Javascript VM via js_driver:new/0.
  4. Use the js module to execute Javascript.
For (1), this means we should supply -boot start_sasl to the erl commandline.(2) means we should add the path to the erlang_js binary to the erl path with -pa, and then execute erlang_js:start(). This amounts to: erl -boot start_sasl -pa ebin -eval 'erlang_js:start().' Now we are ready to run javascript! To run this code, start erlang with: erl -boot start_sasl -pa ebin, compile the example code with c() and simply run js_test:run(). Oh, by the way. I have not found the erlang_js docs online, you should build them with build_docs.sh in the erlang_js source directory.

Fourth meeting

meeting time: 12:30 - 15:30 _Server interface_ - Easy to use - Powerful - Slow communication(Low proiority) - Fast communication(High priority) - Send key:values (nick, pms) - Register callback (serverside) ball is moving, update graphics - Inlogg med cookies or passwords - Master Cookie, optional - loggout /shutdown same cookie as loggin - upload methods - spidermonkey _Internal_Interfaces_ - Javascript as pure text in memory - access the world through a global object - Store the world as ETS(erlang) JS -> ERL -> ETS ->JS -> ... _World_API_ API usage example: users = [users] world.p1.px = 2 world.foo.__set(2) user.send(2) world.foo.__setter = function (a) { erl_set("foo", 2) } Safety issues: - Bad javascript - No IO - ETS Communication requirements: Communication between everyone, single or multiple clients. _Server_interface_ runtime flow: 1. _Initialization_phase_ 2. Message_phase_ -dictionary(JSON,PList, ...) 3. _User_login/logout_

Thursday, January 20, 2011

Which interfaces and API:s do we need? (meeting 3)

Server interface 1. Initiation phase (init the whole game, etc.) 2. User login/logout 3. Messaging phase, events from client to server and from server to client World API (JavaScript) In Spidermonkey we don't have the objects a browser exposes like document or window. But we'll have our own like "user" and "world". We will implement some functions we think are important but not present in JSAPI like setTimeout(). The user object will only be a readable array of all users with their data. All the data is saved in erlang as a dictionary. The world object will be read and writable, here too all the data will be stored in erlang. Spidermonkey allows you to use your own setters and getters, that way it will be no problem for us to use them to store the data in erlang like for example: world.a = 1; which would call a setter we implement which talks to erlang and stores the key value pair. That way we'd have a really nice and clean JavaScript-like API. Internal interfaces This needs to be evaluated further, but we want to use our own spidermonkey shell for security reasons and pipe between it and our erlang server. We will save all the worlds and users and stuff in a ETS instance One of the main goals is to get everything non blocking so we can serve huge amounts of clients and games.

Time report

9 hours

Wednesday, January 19, 2011

We need an API

Me (Jeena), Mathias and Niklas talked about how an API could look, we aren't really ready but this is what we talked about: There are two JS-Objects we introduce: "Users" which is a array of User objects "World" which is a key-value store and introduces two methods: getVal(key) which only gets the value for a key from erlang and returns it setVal(key, val) which does a little bit more, it sets the value for the key in our erlang key-value store and it saves the key in a erlang list with keys which have been updated. and there is a notifyAllClients() method which loops thru the updated list in erlang, packs all the changes into a one notification package and sends this package to all clients. This is only to not to have update all clients for every setVal() call but buffer it if needed. There is one more thing, our protocol needs two different action types. One when a client wants to instantiate a game; it sends the world in JSON format and sends all functions/methods as a (JavaScript) String which we evaluate and add to our spidermonkey. Then the other would be when they want to use the functions to change the world. Cloning and Spidermonkey I was just talking to the spidermonkey developers at irc.mozilla.org#jsapi They say there is no way to clone a spidermonkey instance for various reasons like that there is no clone method for user defined objects, etc. And they said that we should not use their js-shell because they use it on their self for debugging and it is really unsafe. One of them suggested we could just delete a bunch of unsafe functions from js/src/shell/js.cpp Another one suggested we write our own shell with just the functions we really need and he said it'd be easy to do if we'd have a look at their shell while doing it. But I think if we use a API for saving the data in erlang then we wouldn't need to clone spidermonkey, just start up a new one and send in the js-code we got at the beginning.

Time report

8 hours

Our second meeting

We have decided that SpiderMonkey is a good way for us to run javascript. erlang_js seems to be a good way to interact with javascript. We also found out that SpiderMonkey will not supply us with the browser features it usually has, and this is good, because we do not want them. In addition to SpiderMonkey, we should consider NodeJS, it may have other features, such as communication via sockets which we can use. Socket communication may be better suited than pipes. The communication between client and server (and the other way around) needs to be fast. We have decided to use UDP as our underlying protocol, but the protocol on top of UDP should be swappable via a protocol module. We have a few candidates for these "application level protocols"
  • JSON
  • Binary PLIST
  • BSON
JSON is good because
  • It is easy to construct
  • easy to read
  • readily parsable in javascript
It is bad because
  • There will be a lot of overhead
Now, on the other hand, the two other protocols are quite small, but they are difficult to construct, and there's no built in support for them in javascript. We decided to make the server protocol independent, and to create at first a module for JSON allowing us to send messages back and forth in an easily debuggable manner. The lower levels of networking will be handled outside the JSON/other-protocols module. We need to find a way to replicate SpiderMonkey runtimes. We plan to keep at least two runtimes alive, and if one crashes, we should replicate it and produce an exact copy. We have not yet decided how to do this.

Tuesday, January 18, 2011

First meeting

Protocol: Stateful protocol JSON UDP Store session id (sockets) Game node hierarchy: Game → Group → Player (perhaps a lobby would be needed) Different game categories: Chess Farmville Quake We are about to focus to support more games like Chess and Farmville and less on Quake. Project is going towards a generic SpiderMonkey Server than a Generic game server. Is the server going to have one SpiderMonkey instance or multiple? More means safety. We strive for hundreds but not more. Libraries: OTP (Security) SpiderMonkey (Mozilla's C implementation of JavaScript)

Time report

8 hours

Monday, January 17, 2011

About this blog

This is "The Generic Game server development blog". We are a group of students at the Gothenburg University in Sweden and this is our Bachelor-Thesis blog. We will write here along mostly for ourselves so we don't forget what we were doing all the time. Background Currently it is very difficult to roll your own game server. Using our system a game developer should be allowed to focus more of his or her efforts on the actual game, and leave the server part to our software. Project description The general idea is to model your state, in an appropriate format (e.g. JSON, Plist, custom binary, etc.), supply modifying functions (such as game time, or gravity) and push this model to our server, which will keep track of the world. The server will be implemented in Erlang, for concurrency purposes and use JavaScript (or similar) as a DSL to let the client modify the world state. The server will allow clients to push data along to other participating clients (much like a chat), and to modify the game state. Using this model, of pushing either state or pushing messages allows for a very versatile server capable of many things. One could build a chat server (IRC), or a world server (World of Warcraft), etc.