After releasing our last game (Pavel Piezo - Trip to the Kite Festival) we began work to finish MusiGuess. MusiGuess is a simple game where the player is presented with four cover arts, hears a preview of one track from one of the albums and has to pick the right cover to which the track belongs. It is a very addictive mechanic and loads of fun for short sessions in-between. The idea has been around our office for some time now and we had built various prototypes for us to play around with.
In this article I won't go into details about the game, basically everything about the core mechanic is said in the preceding paragraph. I am going to share an easy to use technology that improved performance with client-server communication from "sometimes stalling for a minute" to "lightning fast".
"But, Carsten, just program and set up your servers accordingly?" Yes, I hear you and am very aware of that, but the servers we get a large chunk of our games data from are not ours to tinker with...
MusiGuess uses several Top-Albums-Lists for the game. You can have your riddles assembled from "Top 50", "Rap / Hip Hop", "Rock", "Electronic" and so on. These lists are queried from a public interface on iTunes® which provides nifty RSS feeds for all kinds of product information from the iTunes Store® in XML or JSON (https://rss.itunes.apple.com/). If you mark the promotional resources correctly as being from iTunes® and provide your users with links to buy the displayed media in the iTunes Store®, use of these promotional resources is covered within the affiliate guidelines.
This API for RSS feeds is, at times, slow if it receives multiple queries from the same client/IP within a short timespan. Of course I do not have detailed technical information, but throttling makes perfect sense as protection against misuse and DOS-attacks. Though response times never interfered with the gameplay of the actual riddles in MusiGuess, it could make activating or updating many lists for some users (I can haz all, nau?!) tedious.
Plus, MusiGuess just needs a specific small subset of the information provided in the RSS.
Plus, we don't want MusiGuess (our players) to stress the API unduly.
Plus, every individual device needs the JSON as individual JSONP (JSON with padding), which the API is completely capable of but would mean extra stress on its internal caching methods.
The solution: Forward-caching the JSON objects with NGINX on our dedicated game-server.
I am using Nginx as forward-cache / -proxy. You can utilize a host of different systems to implement the discussed techniques (including "Apache HTTP Server", SQUID, Varnish and systems you code yourself) but to date I find Nginx the most powerful and easy to set up working horse that needs the least resources. Learn more via their wikipedia article (http://en.wikipedia.org/wiki/Nginx).
Delivering data over the internet via HTTP(S) is a common way to go nowadays. All kinds of data, not only in the format HTML, but for instance XML, RSS or JSON, and cache-servers were adapted to deal with these kinds of objects very well. The wikipedia page about web caches (http://en.wikipedia.org/wiki/Web_cache) explains the basics quite nicely.
If you have a server/software that delivers the same data-object to a large number of clients, you don't need to optimize that server to gain response time. You can simply put a forward-proxy "in front" of that server. Requests hit the proxy and the proxy checks if it has a "recent version" of the requested object (the definition of "recent" can be configured). Only if the proxy does not have an actual version of the requested object does it ask your source server/software for one, stores it in its "cache" then hands it on to the original requester. After that the forward proxy will hand the object to every requester without bothering your source server until the configured definition of "recent / actual" is no longer valid for that specific chunk of data.
JSON is a convenient data format for transmission via HTTP(S) and processing with JavaScript (http://en.wikipedia.org/wiki/JSON). Now, "JSONP or 'JSON with padding' is a communication technique used in JavaScript programs ... to request data from a server in a different domain, something prohibited by typical web browsers because of the "same-origin policy." (http://en.wikipedia.org/wiki/JSONP).
Short story: The answer to a JSONP-request has to be wrapped in a unique callback for every individual requester and for every request. This fact makes data in JSONP inconvenient to cache, as it can never be guaranteed to be the same for any two requests. Although the raw data that is transmitted might very well be the same (perfect for caching) the unique wrapping with a callback is most likely not!
How do we manage to still use full forward-caching? The forward-proxy requests the data from the original source in plain JSON, caches it and takes care of the JSONP-wrapping for every individual client and request itself.
For trying this yourself with the configuration I'm showing, you will need a recent version of free Nginx (http://nginx.org/) with the optional HttpEchoModule (http://wiki.nginx.org/HttpEchoModule). I'm not going to discuss the general configuration of Nginx in detail, I will explain the actual JSON/P parts in depth. I also simplified and abstracted this from our specific installation to generalize the concept for easier use in different applications.
You may have noticed that the original data source (php-apache.my-gameserver.com) uses all parameters fully embedded in the URL and the forward-cache (nginx.my-gameserver.com) gets the parameters as HTTP-GET query string. This has no deeper meaning besides it was quicker and easier for me to work with the $arg_* in the configuration. It is well possible to match APIs "query string" / "ULR embedded" any way you see fit within the configuration of your forward-proxy. Design of HTTP-APIs, RESTful, RESTlike etc. are well discussed topics. My advice would be to read up on all that and decide what fits best for your project.
There are some things you should be aware of when using Nginx as forward proxy and when implementing this technique in general. I'll continue to discuss based on Nginx, version 04.14.2014, things may have changed since then or are different in other systems.
If you base your configuration of the "location /.../ { ... }" in Nginx on e.g. an existing boilerplate (Copy&Paste), be aware of "proxy_redirect default;" within this locations block. "proxy_redirect default" cannot be used with a "proxy_pass" directive that uses $arg_* as in the example above.
Simply remove it or comment it out.
If you still want to see the IP-Address of the original request from the client in your source server's logfiles, you'll need to add a specific header that is processed by most major software for log-analysis.
> proxy_set_header X-Real-IP $remote_addr;
HTTP-headers coming from the requesting client can mess with your caching. For instance, when you use "shift-reload" in your browser a header is sent along with the request to specifically instruct the server to not use cached data as an answer. In our case, we want to suppress that, if only to allow "maximum efficiency caching".
> proxy_ignore_headers X-Accel-Expires Expires Cache-Control Set-Cookie;
Similarly, your source server may send a bunch of headers (many headers are sent automatically by web servers) that have no use for your project. If it's not needed, it eats up bandwidth and you may want to suppress these in the answer to the client. With Nginx you can do that by adding the directive "proxy_hide_header header-i-want-to-suppress" to that location's block. You can find all headers that are sent if you request the data with an ordinary browser from your forward-proxy and dig around in the "page information", "developer tools" or similar. Modern browsers have some function that show these.
Additionally, you may want to adjust how long your cached data is valid, how to deal with stale cache answers that can't be refreshed from the source server in time, etc. These are topics that are covered within the documentation of the cache systems and very often the defaults will suffice without much adjustment.
Do you have a game server which is queried for the recent version of the map of your MMSG by thousands of clients every few seconds and your code can't cope with that amount of requests? Forward cache (reverse cache) the JSON or XML object with Nginx and your game servers logic only has to deliver a single object every few seconds.
Got problems with utilizing a third party API because it does not provide JSONP itself? Set up Nginx as a forward proxy and pad the JSON yourself. As long as you have an eye on JSONP's security concerns (http://en.wikipedia.org/wiki/JSONP#Security_concerns) a lot of problems can be tackled this way.
I hope to have helped spread the word and that fellow devs may remember "I have read something about that..." when encountering challenges of that kind.
Article Update Log
06.03.2014: Initial release
About MusiGuess
As of 06.03.2014 there is no promotional website or similar yet.
The game will be released alongside accompanying material in 2014 for iOS.
In this article I won't go into details about the game, basically everything about the core mechanic is said in the preceding paragraph. I am going to share an easy to use technology that improved performance with client-server communication from "sometimes stalling for a minute" to "lightning fast".
"But, Carsten, just program and set up your servers accordingly?" Yes, I hear you and am very aware of that, but the servers we get a large chunk of our games data from are not ours to tinker with...
MusiGuess uses several Top-Albums-Lists for the game. You can have your riddles assembled from "Top 50", "Rap / Hip Hop", "Rock", "Electronic" and so on. These lists are queried from a public interface on iTunes® which provides nifty RSS feeds for all kinds of product information from the iTunes Store® in XML or JSON (https://rss.itunes.apple.com/). If you mark the promotional resources correctly as being from iTunes® and provide your users with links to buy the displayed media in the iTunes Store®, use of these promotional resources is covered within the affiliate guidelines.
This API for RSS feeds is, at times, slow if it receives multiple queries from the same client/IP within a short timespan. Of course I do not have detailed technical information, but throttling makes perfect sense as protection against misuse and DOS-attacks. Though response times never interfered with the gameplay of the actual riddles in MusiGuess, it could make activating or updating many lists for some users (I can haz all, nau?!) tedious.
Plus, MusiGuess just needs a specific small subset of the information provided in the RSS.
Plus, we don't want MusiGuess (our players) to stress the API unduly.
Plus, every individual device needs the JSON as individual JSONP (JSON with padding), which the API is completely capable of but would mean extra stress on its internal caching methods.
The solution: Forward-caching the JSON objects with NGINX on our dedicated game-server.
The server system
I am using Nginx as forward-cache / -proxy. You can utilize a host of different systems to implement the discussed techniques (including "Apache HTTP Server", SQUID, Varnish and systems you code yourself) but to date I find Nginx the most powerful and easy to set up working horse that needs the least resources. Learn more via their wikipedia article (http://en.wikipedia.org/wiki/Nginx).
Why forward caching?
Delivering data over the internet via HTTP(S) is a common way to go nowadays. All kinds of data, not only in the format HTML, but for instance XML, RSS or JSON, and cache-servers were adapted to deal with these kinds of objects very well. The wikipedia page about web caches (http://en.wikipedia.org/wiki/Web_cache) explains the basics quite nicely.
If you have a server/software that delivers the same data-object to a large number of clients, you don't need to optimize that server to gain response time. You can simply put a forward-proxy "in front" of that server. Requests hit the proxy and the proxy checks if it has a "recent version" of the requested object (the definition of "recent" can be configured). Only if the proxy does not have an actual version of the requested object does it ask your source server/software for one, stores it in its "cache" then hands it on to the original requester. After that the forward proxy will hand the object to every requester without bothering your source server until the configured definition of "recent / actual" is no longer valid for that specific chunk of data.
What's the deal with JSONP?
JSON is a convenient data format for transmission via HTTP(S) and processing with JavaScript (http://en.wikipedia.org/wiki/JSON). Now, "JSONP or 'JSON with padding' is a communication technique used in JavaScript programs ... to request data from a server in a different domain, something prohibited by typical web browsers because of the "same-origin policy." (http://en.wikipedia.org/wiki/JSONP).
Short story: The answer to a JSONP-request has to be wrapped in a unique callback for every individual requester and for every request. This fact makes data in JSONP inconvenient to cache, as it can never be guaranteed to be the same for any two requests. Although the raw data that is transmitted might very well be the same (perfect for caching) the unique wrapping with a callback is most likely not!
How do we manage to still use full forward-caching? The forward-proxy requests the data from the original source in plain JSON, caches it and takes care of the JSONP-wrapping for every individual client and request itself.
The Nginx configuration
For trying this yourself with the configuration I'm showing, you will need a recent version of free Nginx (http://nginx.org/) with the optional HttpEchoModule (http://wiki.nginx.org/HttpEchoModule). I'm not going to discuss the general configuration of Nginx in detail, I will explain the actual JSON/P parts in depth. I also simplified and abstracted this from our specific installation to generalize the concept for easier use in different applications.
# Proxy my original JSON data server
location /api_1_0/mapdata {
# This is running on the Nginx cache-server so it will called by something like
# http://nginx.my-gameserver.com/api_1_0/mapdata?area=5&viewwidth=8
# Deactivating GZIP compression between Nginx and original source.
# The setup can't handle GZIP chunks with echo_before (a glitch in nginx or echo as of 04.14.2014)
proxy_set_header Accept-Encoding "deflate";
# Injecting the current callback and wrapping into the answer.
# This is the magic part, that takes care of the JSONP wrapping for the individual clients and requests!
if ($arg_callback) {
echo_before_body -n '$arg_callback(';
# The actual data from the cache or the original data source will automatically be inserted at this point
echo_after_body ');';
}
# Yes, that's all there is to do :-)
# Specifying the key under which the JSON data from the original source is cached.
# This has to be done to ignore $arg_callback in the default cache_key. We utilize only params we specifically need.
proxy_cache_key $scheme$proxy_host$uri$arg_area$arg_viewwidth;
# I advise to always include "$scheme$proxy_host$uri" in a cache_key to be able to add other API versions (e.g. api_2_3)
# and other data points (e.g. /leaderboard) later, without having to rely on additional parameters.
# Telling Nginx were to request the actual source data (JSON) from.
proxy_pass http://php-apache.my-gameserver.com/map/$arg_area/$arg_viewidth/json;
}
You may have noticed that the original data source (php-apache.my-gameserver.com) uses all parameters fully embedded in the URL and the forward-cache (nginx.my-gameserver.com) gets the parameters as HTTP-GET query string. This has no deeper meaning besides it was quicker and easier for me to work with the $arg_* in the configuration. It is well possible to match APIs "query string" / "ULR embedded" any way you see fit within the configuration of your forward-proxy. Design of HTTP-APIs, RESTful, RESTlike etc. are well discussed topics. My advice would be to read up on all that and decide what fits best for your project.
Any more gritty details?
There are some things you should be aware of when using Nginx as forward proxy and when implementing this technique in general. I'll continue to discuss based on Nginx, version 04.14.2014, things may have changed since then or are different in other systems.
If you base your configuration of the "location /.../ { ... }" in Nginx on e.g. an existing boilerplate (Copy&Paste), be aware of "proxy_redirect default;" within this locations block. "proxy_redirect default" cannot be used with a "proxy_pass" directive that uses $arg_* as in the example above.
Simply remove it or comment it out.
If you still want to see the IP-Address of the original request from the client in your source server's logfiles, you'll need to add a specific header that is processed by most major software for log-analysis.
> proxy_set_header X-Real-IP $remote_addr;
HTTP-headers coming from the requesting client can mess with your caching. For instance, when you use "shift-reload" in your browser a header is sent along with the request to specifically instruct the server to not use cached data as an answer. In our case, we want to suppress that, if only to allow "maximum efficiency caching".
> proxy_ignore_headers X-Accel-Expires Expires Cache-Control Set-Cookie;
Similarly, your source server may send a bunch of headers (many headers are sent automatically by web servers) that have no use for your project. If it's not needed, it eats up bandwidth and you may want to suppress these in the answer to the client. With Nginx you can do that by adding the directive "proxy_hide_header header-i-want-to-suppress" to that location's block. You can find all headers that are sent if you request the data with an ordinary browser from your forward-proxy and dig around in the "page information", "developer tools" or similar. Modern browsers have some function that show these.
Additionally, you may want to adjust how long your cached data is valid, how to deal with stale cache answers that can't be refreshed from the source server in time, etc. These are topics that are covered within the documentation of the cache systems and very often the defaults will suffice without much adjustment.
In conclusion
Do you have a game server which is queried for the recent version of the map of your MMSG by thousands of clients every few seconds and your code can't cope with that amount of requests? Forward cache (reverse cache) the JSON or XML object with Nginx and your game servers logic only has to deliver a single object every few seconds.
Got problems with utilizing a third party API because it does not provide JSONP itself? Set up Nginx as a forward proxy and pad the JSON yourself. As long as you have an eye on JSONP's security concerns (http://en.wikipedia.org/wiki/JSONP#Security_concerns) a lot of problems can be tackled this way.
I hope to have helped spread the word and that fellow devs may remember "I have read something about that..." when encountering challenges of that kind.
Article Update Log
06.03.2014: Initial release
About MusiGuess
As of 06.03.2014 there is no promotional website or similar yet.
The game will be released alongside accompanying material in 2014 for iOS.