How to – cross domain ajax to a Ruby app

In some cases, you might have a bunch of apps running on different domains/subdomains and/or ports and you would like to make ajax requests between these services. The problem is that browsers wouldn’t let you make such requests because of the Same Origin Policy which only allowed them to make request to resources within the same domain.

However, most browsers (IE 8+, Firefox 3.5+, Safari 4+, Chrome) implement a simple way to allow cross domain requests as defined in this w3C document.

Of course, if your users have an old version of their browser, you  might have to look into jsonp or something else such as cheating by using iframes & setting document.domain. Let’s pretend for a minute that 100% of your users are on Chrome. The only thing you need to do is set a response header listing the accepted domains or “*” for all. A simple Rack middleware to do that would look like that.

 

class XOriginEnabler
  ORIGIN_HEADER = "Access-Control-Allow-Origin"
 
  def initialize(app, accepted_domain="*")
    @app = app
    @accepted_domain = accepted_domain
  end
 
  def call(env)
    status, header, body = @app.call(env)
    header[ORIGIN_HEADER] = @accepted_domain
    [status, header, body]
  end
end

And to use the middleware you would need to set it for use:

use XOriginEnabler

To enable all requests from whatever origin, or pass the white listed domain(s) as shown below.

use XOriginEnabler, "demo.mysite.com demo.mysite.fr demo.techcrunch.com"

For a full featured middleware, see this project.

6 Comments

Ruby optimization example and explanation

Recently I wrote a small DSL that allows the user to define some code that then gets executed later on and in different contexts. Imagine something like Sinatra where each route action is defined in a block and then executed in context of an incoming request.

The challenge is that blocks come with their context and you can’t execute a block in the context of another one.

Here is a reduction of the challenge I was trying to solve:

class SolutionZero
  def initialize(origin, &block)
    @origin = origin
    @block = block
  end
 
  def dispatch
    @block.call
  end
end
 
SolutionZero.new(42){ @origin + 1 }.dispatch
# undefined method `+' for nil:NilClass (NoMethodError)

The problem is that the block refers to the @origin instance variable which is not available in its context.
My first workaround was to use instance_eval:

class SolutionOne
  def initialize(origin, &block)
    @origin = origin
    @block = block
  end
 
  def dispatch
    self.instance_eval &@block
  end
end
 
SolutionOne.new(40){ @origin + 2}.dispatch
# 42

My workaround worked fine, since the block was evaluated in the context of the instance and therefore the @origin ivar is made available to block context. Technically, I was good to go, but I wasn’t really pleased with this solution. First using instance_eval often an indication that you are trying to take a shortcut. Then having to convert my block stored as a block back into a proc every single dispatch makes me sad. Finally, I think that this code is probably not performing as well as it could, mainly due to unnecessary object allocations and code evaluation.
I did some benchmarks replacing instance_eval by instance_exec since looking at the C code, instance_exec should be slightly faster. Turns out, it is not so I probably missed something when reading the implementation code.

I wrote some more benchmarks and profiled a loop of 2 million dispatches (only the #disptach method call on the same object). The GC profiler report showed that the GC was invoked 287 times and each invocation was blocking the execution for about 0.15ms.
Using Ruby’s ObjectSpace and disabling the GC during the benchmark, I could see that each loop allocates an object of type T_NODE which is more than likely our @block ivar converted back into a block. This is quite a waste. Furthermore, having to evaluate our block in a different context every single call surely isn’t good for performance.

So instead of doing the work at run time, why not doing it at load time? By that I mean that we can optimize the #dispatch method if we could “precompile” the method body instead of “proxying” the dispatch to an instance_eval call. Here is the code:

class SolutionTwo
  def initialize(origin, &block)
    @origin = origin
    implementation(block)
  end
 
  private
 
  def implementation(block)
    mod = Module.new
    mod.send(:define_method, :dispatch, block)
    self.extend mod
  end
end
 
SolutionTwo.new(40){ @origin + 2}.dispatch
# 42

This optimization is based on the fact that the benchmark (and the real life usage) creates the instance once and then calls #dispatch many times. So by making the initialization of our instance a bit slower, we can drastically improve the performance of the method call. We also still need to execute our block in the right context. And finally, each instance might have a different way to dispatch since it is defined dynamically at initialization. To work around all these issues, we create a new module on which we define a new method called dispatch and the body of this method is the passed block. Then we simply our instance using our new module.

Now every time we call #dispatch, a real method is dispatched which is much faster than doing an eval and no objects are allocated. Running the profiler and the benchmarks script used earlier, we can confirm that the GC doesn’t run a single time and that the optimized code runs 2X faster!

 

Once again, it’s yet another example showing that you should care about object allocation when dealing with code in the critical path. It also shows how to work around the block bindings. Now, it doesn’t mean that you have to obsess about object allocation and performance, even if my last implementation is 2X faster than the previous, we are only talking about a few microseconds per dispatch. That said microseconds do add up and creating too many objects will slow down even your faster code since the GC will stop-the-world as its cleaning up your memory. In real life, you probably don’t have to worry too much about low level details like that, unless you are working on a framework or sharing your code with others. But at least you can learn and understand why one approach is faster than the other, it might not be useful to you right away, but if you take programming as a craft, it’s good to understand how things work under the hood so you can make educated decisions.
 

Update:

@apeiros in the comments suggested a solution that works & performs the same as my solution, but is much cleaner:

class SolutionTwo
  def initialize(origin, &block)
    @origin = origin
    define_singleton_method(:dispatch, block) if block_given?
  end
end

, ,

12 Comments

Deploying a Rails 3.1 app – gotchas

Recently I had to build a new app as part of my research & development job at LivingSocial. My goal was to get the app up and running in just a few weeks, solid application architecture and graphic design included.
When you need to build an app quickly and you want it to have some solid foundations, Rails is quite useful.
I used Rails 3.1RCx so if we would to keep my app and push it to production, the engineering team wouldn’t have to update it and the transition should be seamless. I also quite like CoffeeScript and the app being quite heavy on JavaScript, the choice was easy. Furthermore, my coworker Bruce Williams is a fan of SCSS and he’s writing a PragProg book called “The Rails View” with other LivingSocialist: John Athayde. So you got the point, I’m using Rails3.1, but this post is about the challenges I faced when it was time to deploy and the solutions I found.

I’ll skip the intro to Rails 3.1 and how to use the new asset pipeline, refer to the Rails guide or one of the mainly posts referenced in this post (if I had properly read the guide, it would have saved me some valuable time, trust me, read it carefuly).

At that point last night, I had my app working great locally, Bruce created some awesome scss code using mixins and nested rules, the HTML was clean and working great, my CoffeeScript was brewing nicely, all was great until I tried to deploy to our QA environment.

JavaScript runtime dependency

The first thing you will notice is that you need the proper JavaScript runtime so the asset pipeline works properly. Not a big deal, you’ll find a lot of documentation about that. The problem is that you need to update your production environment or use depend on gem that will compile the required runtime (sounds dirty to me). So if you are deploying to many machines and you are using an image solution (EC2 AMI or other), you will need to update your image or spin new instances via updated chef/puppet recipes. In this case, the awesome team at LivingSocial had an image ready for me, so that wasn’t a big deal, but still, you need to take that in consideration as you are planning to update.

So the asset pipeline optimizes your asset management by processing/compiling asset files for you and optimizing their delivery. Instead of serving static files directly via public/images or public/javascripts you know serve them via the asset pipeline which will take care of compiling your CoffeeScript, grouping and minifying your JS and preprocessing all sorts of format. It also optimizes the caching process by giving a unique filename to each file based on the file metadata and gziping files. This is great, but you really, really, really don’t want to have your apps take care of that in production. Why wasting precious resources to serve assets when they can be prepared ahead of time. (by making Rails serve static assets, you are seriously reducing the throughput of  your app, please think of the children (or the dolphins/trees if you don’t like children))

Capistrano

Rails obviously has a preprocessor available as a rake task and you should update your deployment recipe to use that new feature. Here is my Capistrano code:

after 'deploy:update_code', 'deploy:compile_assets'
namespace :deploy do
  task :compile_assets do
    run "cd #{release_path}; RAILS_ENV=production rake assets:precompile"
  end
end

Well, my real code doesn’t hardcode the RAILS_ENV constant value, it’s in fact set in each env file, but I simplified it since most people only use 1 env outside of dev & test.

What that will do is compile all the files and dump them in public/assets/. But the file I had called bubble.png now becomes bubble-27543c671a3ab45141ee0d3216085009.png which means that my app is totally broken because images use in Bruce SCSS don’t load, my js files don’t load and the app is totally broken. Now this is least fun part, that I wish I had known before. This is where you go back and change your code so it uses magic to get the right file names.

Images

Fixing images was actually quite simple, in all my views, I just had to make sure I was using the image_tag helper everywhere.

CSS

SCSS files were a bit more tricky, I had to use the new scss preprocessor helpers you will find in the Rails guide (image_path and image_url). I first looked into using erb, but turned out it wasn’t needed and the end result is much cleaner.

Javascript/CoffeeScript

For the CoffeeScript files, I was referring to image assets in the code and of course all the links were broken. So I had to use ERB in my coffee which looked funky but it worked.

But to get that to work, you need to rename your coffee script and append erb at the end. For instance my feature.js.coffee script had to be renamed feature.js.coffee.erb. That made me cry a little inside, but oh well, at least its not a XML config file. Maybe soon we will start seeing code in filenames or filenames called my_feature.js.compressed.minified.coffee.erb.from_rails.mattetti.org
Also, be careful about the order of the file extensions, otherwise it won’t work. I thought I was done, ready to deploy my apps and this time the assets will show up properly. Turns out I was wrong :(

Rails asset precompilation env specific configuration.

My css looked good, the precompiling task had run fine but I was missing some js files. I scratched my head as I could only see some of my js files. I then realized that all my JS files were there but some of my CoffeeScript files were missing. The answer was given to me by Bruce who asked me if I had updated my “config.assets.precompile” setting. Sometimes I feel that Rails is trying to compete with Struts and here I was really surprised that by default Rails, in production mode only precompiles all static JS and application.js files, but none of the other dynamic js files. Now it does precompile all the scss files, but for a reason I just don’t understand, it’s not the case for the JS files. So, you have to go edit production.rb in the config/environments folder and add the other js files you would like Rails to precompile for you.

After making all these changes, I was able to redeploy my app and everything was working again. (you might want to tweak your apache/nginx config as explained in the Rails guide)

 

Conclusion

Don’t be fooled like me and expect that because you have an app running locally, deployment will work right away. Make sure to read about the new features and what’s needed. Overall, I think that the asset pipeline is a nice addon to Rails and if you don’t feel like using it, just can put/leave all your files in the public folder and everything will work just like before. I do have to say that I was surprised to see that even in a brand new Rails 3.1 project, Rails isn’t running in threaded mode by default. But that’s a different (old) story and I guess people still get more excited about asset management than framework raw performance ;)

 

 

 

7 Comments

First step in scaling a web site: HTTP caching

Today my friend Patrick Crowley and I were talking about scaling his website: http://cinematreasures.org since an article covering his work will soon be published in a very popular newspaper. Patrick’s site is hosted on Heroku which comes by default with Varnish caching enabled.

The challenge is that a lot of people using the Rails framework are used to doing page caching instead of relying on HTTP caching, even though this feature was added a long time ago. The major problem with page caching is that it doesn’t scale that well as soon as you run more than one server. Indeed you would need to store the page content to a shared drive between your servers or use memcached and do some work to avoid hitting your app every single time. On the other hand, HTTP caching is extremely easy to handle at the application level and it will dramatically reduce the amount of requests hitting your app. Let me explain a little more about HTTP caching.

Ryan Tomako wrote an excellent post about the details of caching, I strongly recommend you read it. In a nutshell, the HTTP caching layer (usually) seats before your application layer and allows you, the developer to store some responses that can be send back to the users based on optional conditions. That might still seem vague, let’s take a concrete example. If you look at http://cinematreasures.org‘s home page you can see that it’s an agglomerate of various information:

CinemaTreasures homepage

And the bottom of the page contains even more dynamic data such as the popular movie theater photos, latest movie theater videos and latest tweets. One might look at that and say that this page can’t really be cached and that the caching should be done at the model layer (i.e. cache the data coming from the database). I would certainly agree that caching the data layer is probably a good idea, but you shouldn’t start by that. In fact without caching, this page renders fast enough. The problem is when someone like Roger Ebert tweets about CinemaTreasures the load on the app peaks significantly. At the point, the amount of concurrent connections your app can handle gets put to the challenge. Even though your page load is “fast enough”, requests will queue up and some will eventually time out. That’s actually a perfect case of HTTP caching.

What we want to do in that case is to cache a version of the home page in Varnish for 60 seconds. During that time, all requests coming to the site, will be served by Varnish and will all get the same cached content. That allows our servers to handle the non cached requests and therefore increase our throughput. What’s even better, is that if a user refreshes the home page in his/her browser during the first 60 seconds the requests won’t even make it all the way to our servers. All of that thanks to conditions set on the response. The first user hitting the HTTP cache layer (Varnish in this case) won’t find a fresh cached response, so varnish will forward the request to our application layer which will send back the homepage to varnish and tell Varnish that this content is good for a full minute so please don’t ask for it again until a minute from now. Varnish serves this response to the users’ browser and let the browser know that the server said that the response was good enough for a minute so don’t bother asking for it again. But now, if during these 60 seconds another user comes in, he will hit Varnish and Varnish will have the cached response from the first user and because the cache is still fresh (it’s not been 60 seconds since the first request) and the cache is public, then the same response will be sent to the second user.

As you can see, the real strength of HTTP caching is the fact that it’s a conditional caching. It’s based on the request’s URL and some “flags” set in the request/response headers.

Setting these conditions in your app is actually very simple since you just need to set the response’s headers. If you are using a Ruby framework you will more than likely have access to the request object via the “request” method and you can set the headers directly like that: “response.headers['Cache-Control'] = ‘public, max-age=60′”.
In Rails, you can actually use a helper method instead: expires_in 1.minute, :public => true.

You might have a case where you HAVE TO serve fresh content if available and can’t serve stale cached content even for a few seconds. In this case, you can rely on the Etag header value. The Etag is meant to validate the freshness of a cached response. Think of it as a signature (unique ID) that is set on the response and used by the client (or cache layer) to see if the server response has changed or not. The way it works is that the client keeps track of the Etag received for each request (attached to the cached response) and then sends it with the next requests. The HTTP layer or application sees the Etag in the request and can check if it is still valid and the content didn’t change. If that’s the case, an empty response can be sent with a special HTTP status code (304) to let know the client that the old cached value is still good to be used.  Rails has a helper called “stale?” that helps you do the Etag/last modified check and allows you to not fetch all the objects from the database by doing a cheap check on an attribute (For instance you can check the updated_at value and use that as a condition to pull an object and its relationships).

So I explain HTTP caching, I often hear people telling me: “that’s great Matt, but you know what, that won’t work for us because we have custom content that we display specifically to our users”. So in that case, you can always set the Cache-Control header to private which will only cache the response in the client’s browser and not the cache layer. That’s good to some extent, but it can definitely be improved by rethinking a bit your view layer. In most web apps, the page content is rendered by server side code (Rails, Django, node.js, PHP..) and sent to the user all prepared for him. There are a few challenges with this approach, the biggest one is that the server has to wait until everything is ready (all data fetched, view rendered etc…) before sending back a response and before the client’s browser can start rendering (there are ways to chunk the response but that’s besides the scope of this post). The other is that the same expensive content has to be calculated/rendered for two different users because you might be inserting the username of the current user at the top of the page for instance. A classic way to deal with that is often to use fragment caching, where the expensive rendering is cached and reused by different requests. That’s good but if the only reason to do that is because we are displaying some user specific data, there is a simpler way: async page rendering. The concept is extremely simple: remove all user specific content from the rendered page and then inject the user content in a second step once the page is displayed. The advantage is that now the full page can be cached in Varnish (or Squid or whatever you use for HTTP caching). To inject the user content, the easiest way is to use JavaScript.

Let’s stay on CinemaTreasures, when you’re logged in, the username is shown on the top of each page:

Once logged in, the username is displayed on all pages

The only things that differs from the page rendered when the user is not logged in and when he is, are these 2 links and an avatar. So let’s write some code to inject that after rendering the page.

In Rails, in the sessions controller or whatever code logs you in, you need to create a new cookie containing the username:

cookies[:username] = {
         :value => session[:username],
         :expires => 2.days.from_now,
         :domain => ".cinematreasures.org"
       }

As you can see, we don’t store the data in the session cookie and the data won’t be encrypted. You need to be careful that someone changing his cookie value can’t access data he/should shouldn’t. But that’s a different discussion. Now that the cookie is set, we can read it from JavaScript when the page is loaded.

document.observe("dom:loaded", function() {
  displayLoggedinUserLinks();
});
 
function readCookie(name) {
     var nameEQ = name + "=";
     var ca = document.cookie.split(';');
     for(var i=0;i < ca.length;i++) {
          var c = ca[i];
          while (c.charAt(0)==' ') c = c.substring(1,c.length);
          if (c.indexOf(nameEQ) == 0) return c.substring(nameEQ.length,c.length);
     }
     return null;
}
 
function displayLoggedinUserLinks() {
  var username            = readCookie('username');
  var loginLink           = $('login');
  var logout              = $('logout');
  if (username == null){
    loginLink.show();
    logout.hide();
  }else{
    // user is logged in and we have his/her username
    loginLink.hide();
    if(userGreetings){ userGreetings.update("<span id='username'>username</span>"); }
    logout.show();
    showAvatar(username);
  };
  return true;
}

The code above doesn’t do much, once the DOM is loaded, the displayLoggedinUserLinks() function gets trigger. This function reads the cookie via the readCookie() function and if a username is found, the login link is hidden, the user name is displayed, as well as the logout link and the avatar. (You can also use a jQuery cookie plugin to handle the cookie, but this is an old example using Prototype, replace the code accordingly)
When the user logs out, we just need to delete the username cookie and the cached page will be rendered properly. In Rails, you would do delete the cookie like that: cookies.delete(‘username’).
Quite often you might even want to make an Ajax call to get some information such as the number of user messages or notifications. Using jQuery or whatever JS framework you fancy you can do that once the page is rendered. Here is an example, on this page, you can see the learderboards for MLB The Show. The leaderboards don’t change that often, especially the overall leaderboards so they can be cached for a little while, however the player’s presence can change anytime. The smart way to deal with that, would be to cache the  leaderboards for a few seconds/minutes and make an ajax call to a presence service passing it a list of user ids collected from the DOM. The service called via Ajax could also be cached  depending on the requirements.

Now there is one more problem that people using might encouter: flash notices. For those of you not familiar with Rails, flash notices are messages set in the controller and passed to the view via the session (at least last time I checked). The problem happens if I’m the home page isn’t cached anymore and I logged in which redirects me to the home page with a flash message like so:

The problem is that the message is part of the rendered page and now for 60 seconds, all people hitting the home page will get the same message. This is why you would want to write a helper that would put this message in a custom cookie that you’d pull JS and then delete once displayed. You could use a helper like that to set the cookie:

def flash_notice_cookie(msg, expiration=nil)
  cookies[:flash_notice] = {
    :value => msg,
    :expires => expiration || 1.minutes.from_now,
    :domain => ".cinematreasures.com"
   }
end

And then add a function called when the DOM is ready which loads the message and injects it in the DOM. Once the cookie read, delete it so the message isn’t displayed again.

 

So there you have it, if you follow these few steps, you should be able to handle easily 10x more traffic without increasing hardware or making any type of crazy code change. Before you start looking into memcached, redis, cdns or whatever, consider HTTP caching and async DOM manipulation. Finally, note that if you can’t use Varnish or Squid, you can very easily setup Rack-Cache locally and share the cache via memcached. It’s also a great way to test locally.


Update: CinemaTreasures was updated to use HTTP caching as described above. The hosting cost is now half of what it used to be and the throughput is actually higher which offers a better protection against peak traffic.


 

External resources:

, , , , ,

6 Comments

Go’s reflection example

The Go Programming language is really cool language by Google. According to the sales pitch, it’s a “fast, statically typed, compiled language that feels like a dynamically typed, interpreted language”. Well, if you are like me, you don’t trust sales pitches because you know that people writing them dont’ care about you, they care about their product. However cynical you are, you still have to check the facts. So here is a quick demonstration showing how to use Go’s reflection feature.

Installing Go is actually really straight forward on a Mac, and slightly harder on Linux, check this guide to see how to build Go in a few minutes.

Once all setup, you might want to read the documentation to see how to code in Go. Go is actually a kind of nice version of C with a simplified syntax, no header files, really fast compilation time, a garbage collector and a simple way to approach object inheritance without turning in the complicated mess C++ is. The language is designed around the concept of goroutines, a very nice way to handle concurrency. It also has some features that Rubyists, Pythonistas and Javascripters wouldn’t want to live without such as closures and some they probably wish they had such as defer. But of the things we are used to with dynamic languages is the concept of reflection. In a nutshell, at runtime, your code can reflect on the type of a given object and let the developer act accordingly. Depending on your programming background that might be obvious or you might not see the value. To be honest, that’s not the question here. What I’m interested in showing you is how it works.

For the sake of this demo, let’s pretend we want to have a “Dish” data model, each instance of the “Dish” type will have a few attributes, an id, a name, an origin and a custom query which really is a function that we store as an attribute. Here is how we would represent that model in Go:

// Data Model
type Dish struct {
  Id  int
  Name string
  Origin string
  Query func()
}

This is more or less the equivalent of the following Ruby code:

class Dish
  attr_accessor :id, :name, :origin, :query
end

Ruby works slightly differently in the sense that defining attribute accessors create getters and setter methods but doesn’t technically create instance variables until they are used. Here is what I mean:

shabushabu = Dish.new
shabushabu.instance_variables # => []
shabushabu.name = "Shabu-Shabu"
shabushabu.instance_variables # => ["@name"]
shabushabu.origin = "Japan"
shabushabu.instance_variables # => ["@name", "@origin"]

Another way of checking on the accessors is to check the methods defined on the object:

shabushabu.methods - Object.new.methods
=> ["name", "name=", "origin", "origin=", "id=", "query", "query="]

But anyway, this post isn’t about Ruby, it’s about Go and what we would like is to reflect on an object of “Dish” type and see its attributes. The good news is that the Go language ships with a package to do just that. Here is the full implementation:

package main
 
import(
  "fmt"
  "reflect"
)
 
func main(){
  // iterate through the attributes of a Data Model instance
  for name, mtype := range attributes(&Dish{}) {
    fmt.Printf("Name: %s, Type %s\n", name, mtype.Name())
  }
}
 
// Data Model
type Dish struct {
  Id  int
  Name string
  Origin string
  Query func()
}
 
// Example of how to use Go's reflection
// Print the attributes of a Data Model
func attributes(m interface{}) (map[string]reflect.Type) {
  typ := reflect.TypeOf(m)
  // if a pointer to a struct is passed, get the type of the dereferenced object
  if typ.Kind() == reflect.Ptr{
    typ = typ.Elem()
  }
 
  // create an attribute data structure as a map of types keyed by a string.
  attrs := make(map[string]reflect.Type)
  // Only structs are supported so return an empty result if the passed object
  // isn't a struct
  if typ.Kind() != reflect.Struct {
    fmt.Printf("%v type can't have attributes inspected\n", typ.Kind())
    return attrs
  }
 
  // loop through the struct's fields and set the map
  for i := 0; i < typ.NumField(); i++ {
    p := typ.Field(i)
      if !p.Anonymous {
        attrs[p.Name] = p.Type
      }
     }
 
  return attrs
}

Unfortunately, my code highlighter doesn’t support the Go syntax, but GitHub does, so here is a pretty version.

There are ways of running Go source code like Ruby or Python scripts but in this case, we’ll use the compiler & linker provided with Go. I named my source file “example.go”, and here is how I compiled, linked and run it:

$ 6g example.go && 6l example.6 && ./6.out
Name: Origin, Type string
Name: Id, Type int
Name: Query, Type 
Name: Name, Type string

As you can see each attribute is printed out with its name and type. The code might seem a bit odd if you never looked at Go before.
Here is a quick rundown of the code:

In our main function, we create a new instance of type Dish on which we call attributes on. The call returns a map on which we iterate through and print the attribute name (key) and type (value).
The attributes function is defined a bit below and and it takes any type of objects (empty interface) and returns a map, which is like a Hash or a Dictionary. The map has keys of String type and values of “Type” type. The “Type” type is defined in the reflect package. Inside the function, 23 then use the previously mentioned reflect package to check on the type and the name of each attribute and assign it to a map object. (note that I’m explicitly returning the map, but I could have done it in a more implicit way)

So there you go, that’s how you use reflection in Go. Pretty nifty and simple.

 

No Comments

Sayonara Sony

It’s now official, I have resigned from Sony Computer Entertainment America.

I was planning on posting this a bit later but since I was politely escorted out of the building by HR/security, I have more free time to let you know of my decision. Before you ask: No, my decision isn’t directly related to the recent PSN/Sony security breach events and no, I don’t have your credit card number. More seriously, my decision boils down to something much simpler & concrete: drive and passion.

The concept of drive is very well explained in this talk by Dan Pink and illustrated by RSA:

As explained, it is proven that when doing cognitive tasks, there are 3 factors that lead to better performance & personal satisfaction:

  • Autonomy: engagement vs compliance
  • Mastery: get better at stuff
  • Purpose: be disruptive but make the world a better place

 

The challenge is that when you work for a big corporation, you rarely see these factors applied. The amount of red tape, management overhead, lack of recognition and accountability result in a low drive by most employees. I think it was the first time in my entire career that I was told by a manager to care less about the quality and end result of our products.

The second important concept that I think is critical when looking at your career is passion. This topic is very well covered in Chad Fowler‘s book: The Passionate Programmer.

Here is a quote from Chad’s book:

Fulfillment and happiness don’t (often) come by chance. They require thought, intention, action, and a willingness to change course when you’ve made mistakes. [...] It might be a technology or business domain that gets you excited. Or, on the other hand, it might be a specific technology or business domain that drags you down. Or a type of organization. Maybe you’re meant for small teams or big teams. Or rigid processes. Or agile processes. Whatever the mix, take some time to find yours. You can fake it for a while, but a lack of passion will catch up with you and your work.

 

It’s hard to summarize Chad’s book into just a few sentences, but what I got from his book is that if, for whatever reason, you lose your passion for your job, you should move on to another place where you can be passionate and excel. In my case, I’m still very much passionate about video game development but I find my passion seriously affected by an unhealthy work environment, bad communication and a lack of desire to change things in concrete ways. As the saying goes, “People don’t leave jobs…”

 

What’s next for me?

If you are in the software industry you know that everybody is hiring and that there is a real shortage of talent out there. You probably also receive half a dozen emails per week from recruiters offering you “the best job ever” with an obscene salary. Well, I receive them too. But in this market, you and I can allow ourselves to be picky and to choose the right job for ourselves. By choosing the right job, we are going to be more passionate, more driven, more efficient and bring a lot more value to our employers, who, in return, will hopefully do everything they can to make sure we grow within the company with a strong desire to do even better. As I was considering what I would want to do if I were to leave Sony, I came up with a few ideas about a job:

  • I don’t want to live to work, but rather, work to live. (kind of an European cliché sentence)
  • It’s not about the job title.
  • It’s not about the pay check.
  • It’s not about how glamourous an industry  is.
  • What matters is :
    • the company culture.
    • the passion coming from the leaders.
    • the company’s ambitions.
    • how the company rewards and respects its employees.
    • the autonomy/trust given to the employees.
    • the purpose of the company and its potential to disrupt a market/change the world.
    • personal growth within the company.

 

Writing this list helped me realize that I was definitely not in the right place and helped me create a list of criteria to define what kind of job I should be doing instead. As I was thinking about all that, I was reminded of this quote:

“One should not pursue goals that are easily achieved. One must develop an instinct for what one can just barely achieve through one’s greatest efforts.” —Albert Einstein

This is probably a personality trait, but I like/need to be at the edge of my confort zone. I need to learn new things, experience new challenges. I need to try to solve unsolved problems. So part of me needs a company with a rich culture, a great a vision and leadership, but another part also needs to be allowed to think creatively, to push the existing boundaries, to challenge myself and to try to achieve things through my greatest efforts.

The challenge is that over the last 5-7 years I have become a Ruby specialist. I have learned to understand and master the language, learning its pros/cons and what goes on “under the hood”. I have built small and huge solutions for various domain spaces on top of Ruby. I have shared my knowledge giving talks, books, blog posts. I have also spent a lot of time expanding my own computer science knowledge, looking into other programming languages, frameworks, designs. My job at Sony was interesting due to the fact that Ruby is used in a quite singular way with very specific problems and a lot of C/C++ interaction. So while it is true that I am a Ruby specialist, I don’t like that label. I don’t like it because it’s very limiting. My goal is to solve problems and quite often Ruby is a great tool for that, but for some problems, it is not. So I need a job where my expertise is helpful but would not limit my desire to learn new approaches, languages and skills.

And this is why I will soon start working as a Code Alchemist in the LivingSocial R&D’s lab.

That might be a shocker at first glance. Going from working on video games to working for a daily-deal startup? It doesn’t seem like a very smart career move. To be honest, that was my first reaction. But LivingSocial’s VP of engineering is Chad Fowler (the author of The Passionate Programmer, mentioned earlier) and its VP of R&D is Rich Kilmer, InfoEther’s cofounder and recognized mad-scientist-coder. I’ve known Chad and Rich for many years, meeting at tech conferences that they organized or were invited to give talks to, and even getting to hack on some projects with Rich. So when they approached me to join them at LivingSocial, I wanted to know more about their own motivations, why I would be a good fit and how the company/job would rate against my list of criteria. We had long and honest discussions and the result is that I am excited to soon be working with one of the best teams I know of. More concretely, I will be working with Rich, Bruce and Michael in helping LivingSocial revolutionize the local market. LivingSocial is a fast growing and successful company with a strong focus on creating a great experience for both customers and merchants. The company’s vision is well defined and the desire to change the world the way we know it is palpable. Most of the founders have a technical background, they value good engineering practices and have created a nice, positive company culture. Chad has some really awesome, passionate and talented engineers on his team (too many to mention), for whom I have a lot of respect and with whom I look forward to working with.  I also value the fact that LivingSocial trusts and values me enough to let me work remotely. This is very important for me because it shows that my new employer really wants to work with me, trusts me but also is willing to embrace the challenges of having a remote team if that’s what’s needed to create the “right team”. After looking more deeply at what I need and what LivingSocial is, I believe that I can assist LivingSocial in making a very positive change to the way small businesses around the world are run. And I am looking forward to this.

 

25 Comments

Video game web framework design

In this post I will do my best to explain why and how I reinvented the wheel and wrote a custom web framework for some of Sony’s AAA console titles. My goal is to reflect on my work by walking you through the design process and some of the implementation decisions. This is not about being right or being wrong, it’s about designing a technical solution to solve concrete business challenges.

Problem Domain

The video game industry is quite special, to say the least. It shares a lot of similarities with the movie industry. The big difference is that the movie industry hasn’t evolved as quickly as the video game industry has. But the concept is the same, someone comes up with a great idea, finds a team/studio to develop the game and finds a publisher. The development length and budget depends on the type of game, but for a AAA console game, it usually takes a least a few million and a minimum of a year of work once the project has received the green light. The creation of such a game involves various teams, designers, artists, animators, audio teams, developers, producers, QA, marketing, management/overhead etc.. Once the game gets released, players purchase the whole game for a one time fee and the studio moves on to their next game. Of course things are not that simple, with the latest platforms, we now have the option to patch games, add DLC etc.. But historically, a console game is considered done when it ships, exactly like a movie, and very little work is scheduled post release.

Concretely such an approach exposes a few challenges when trying to implement online features for a AAA console title:

  • Communication with the game client network team
  • Scalability, performance
  • Insane deadlines, unstable design (constant change of requirements)
  • Can’t afford to keep on working on the system once released (time delimited projects)

 

Communication

As in most situations, communication is one of the biggest challenges. Communication is even harder in the video game industry since you have so many teams and experts involved. Each team speaks its own jargon, has its own expertise and its own deadlines. But all focus on the same goal: releasing the best game ever. The team I’m part of has implementing online features as its goal. That’s the way we bring business value to our titles. Concretely, that means that we provide the game client developers with a C++ SDK which connects to custom web APIs written in Ruby. The API implementations rely on various data stores (MySQL, Redis, Memcached, memory) to store and retrieve all sorts of game data.

Nobody but our team should care about the implementation details, after all, the whole point of providing an API is to provide a simple interface so others can do their part of the job in the easiest way possible. This is exactly where communication becomes a problem. The design of these APIs should be the result of the work of two teams with two different domains of expertise and different concerns. One team focuses on client performance, memory optimization and making the online resources available to the game engine without affecting the game play. The other, focuses on server performance, latency, scalability, data storage and system contention under load. Both groups have to come together to find a compromise making each other’s job doable. Unfortunately, things are not that simple and game designers (who are usually not technical people) have a hard time not changing their designs and requirements every other week (usually for good reasons) making API design challenging and creating tension between the teams.

From this perspective, the API is the most important deliverable for our team and it should communicate the design goal while being very explicit about how it works, why it works the way it does, and how to implement it client side. This is a very good place where we can improve communication by making sure that we focus on making clear, well designed, well documented, flexible APIs.

 

Scalability, performance

On the server side, the APIs need to perform and scale to handles tends of thousands of concurrent requests. Web developers often rely on aggressive HTTP caching but in our case, the web client (our SDK) has a limited amount of memory available and 90% of the requests are user specific (can’t use full page HTTP cache) and a lot of these are POST/DELETE requests (can’t be cached). That means that, to scale, we have to focus on what most developers don’t often have to worry too much about: all the small details which, put together with a high load, end up drastically affecting your performance.

While Ruby is a great language, a lot of the libraries and frameworks are not optimized for performance, at least not the type of performance needed for our use case. However, the good news is that this is easily fixable and many alternatives exist (lots of async, non-blocking drivers for i.e). When obsessed with performance, you quickly learn to properly load test, profile, and monitor your code to find the bottlenecks and the places where you should focus your attention. The big, unique challenge though, is that a console game will more than likely see its peak traffic in the first few weeks, not really giving the chance to the online team to iteratively handle the prod issues. The only solution is to do everything possible before going live to ensure that the system will perform as expected. Of course if we were to write the same services in a more performant language, we would need to spend less time optimizing. But we are gaining so much flexibility by using a higher level programming language that, in my mind, the trade off is totally worth it (plus you still need to spend a lot of time optimizing your code path, even if your code is written in a very fast language).

 

Deadlines, requirement changes

That’s just part of the way the industry works. Unless you work for Blizzard and you can afford to spend a crazy amount of time and money on the development of a title; you will have to deal with sliding deadlines, requirement changes, scope changes etc… The only way I know how to protect myself from such things is to plan for the worst. Being a non-idealistic (read pessimistic) person helps a lot. When you design your software, make sure your design is sound but flexible enough to handle any major change that you know could happen at any time. Pick your battles and make sure your assumptions are properly thought through, communicated and documented so others understand and accept them. In a nutshell, this is a problem we can’t avoid, so you need to embrace it.

 

Limited reusability

This topic has a lot to do with the previous paragraph. Because scopes can change often and because the deadlines are often crazy, a lot of the time, engineers don’t take the time to think about reusability. They slap some code together, pray to the lords of Kobol and hope that they won’t have to look at their code ever again (I’m guilty of having done that too). The result is a lot of throw away code. This is actually quite frequent and normal in our industry. But it doesn’t mean that it the right thing to do! The assumption/myth is that each game is different and therefore two games can’t be using the same tech solution. My take on that is that it’s partly true. But some components are the same for 80% of the games I work on. So why not design them well and reuse the common parts? (A lot of games share the same engines, such as Unreal for example, and there is no reason why we can’t build a core online engine extended for each title)

 

My approach

When I joined Sony, I had limited experience with the console video game industry and my experience was not even related to online gaming. So even though I had (strong) opinions (and was often quite (perhaps even too) vocal about them), I did my best to improve existing components and work with the existing system. During that time, the team shipped 4 AAA titles on the existing system. As we were going through the game cycles, I did my best to understand the problem domain, the reasons behind some of the design decisions and finally I looked at what could be done differently to improve our business value. After releasing a title with some serious technical difficulties, I spent some time analyzing and listing the problems we had and their root causes. I asked our senior director for a mission statement and we got the team together to define the desiderata/objectives of our base technology. Here is what we came up with:

  1. Stability
  2. Performance / Scalability
  3. Encapsulation / Modularity
  4. Documentation
  5. Conventions
  6. Reusability / Maintainability

These objectives are meant to help us objectively evaluate two options. The legacy solution was based on Rails, or more accurately: Rails was used in the legacy solution. Rails had been hacked in so many different ways that it was really hard to update anything without breaking random parts of the framework. The way to do basic things kept being changed, there was no consistent design, no entry points, no conventions and each new game would duplicate the source code of the previously released game and make the game specific changes. Patches were hard to back port and older titles were often not patched up. The performance was atrocious under load, mainly due to hacked-up Rails not performing well. (Rails was allocating so many objects per request that the GC was taking a huge amount of the request cycles, the default XML builder also created a ton load of objects etc…) This was your typical broken windows scenario. Engineers were getting frustrated, motivation was fainting, bugs were piling up and nobody felt ownership over the tech.

Now, to be fair, it is important to explain that the legacy system was hacked up together due to lack of time, lack of resources and a lot of pressure to release something ASAP. So, while the end result sounds bad, the context is very important to note. This is quite common in software engineering and when you get there, the goal is not to point fingers but to identify the good and the bad parts of the original solution. You then use this info to decide what to do: fix the existing system or rewrite, porting the good parts.

Our report also came up with a plan. A plan to redesign our technology stack to match the desiderata previously mentioned. To put it simply, the plan was to write a new custom web framework focusing on stability, performance, modularity and documentation. Now, there are frameworks out there which already do that or value these principles. But none of them focus on web APIs and none of them are specific to game development. Finally, the other issue was that we had invested a lot of time on game specific code and we couldn’t throw away all that work, so the new framework had to support a good chunk of legacy code but had to make it run much faster.

Design choices

Low conversion cost

Using node.jscoffee script/Scala/whatever new fancy tech was not really an option. We have a bunch of games out there which are running on the old system and some of these games will have a sequel or a game close enough that we could reuse part of the work. We don’t want to have to rewrite the existing code. I therefore made sure that we could reuse 90% of the business logic by adding an abstraction layer doing the heavy lifting at boot time and therefore not affecting the runtime performance. Simple conversion scripts were also written to import the core of the existing code over.

Lessons learned: It is very tempting to just redo everything and start from scratch. However, the business logic implementation wasn’t the main cause of our problems. Even though I wish we could have redesigned that piece of the puzzle, it didn’t make sense from a business perspective. A lot of thought had to be put into how to obtain the expected performance level while keeping the optional model/controller/view combos. By having full control of the “web engine”, we managed to isolate things properly without breaking the old paradigms. We also got rid of a lot of assumptions allowing us to design new titles a bit differently while being backward compatible and have our code run dramatically faster.

Web API centric

This is probably the most important design element. If I had to summarize what our system does in just a few words, I would say: a game web API. Of course, it’s much more than that. We have admin interfaces, producer dashboards, community websites, lobbies, p2p, BI reports, async processing jobs etc… But at the end of the day, the only one piece you can’t remove is the game web API. So I really wanted the design to focus on that aspect. When a developer starts implementing a new online game feature, I want him/her to think about the API. But I also want this API to be extremely well documented so the developer working client-side understands the purpose of the API, how to use it, and what the expected response is right away. I also wanted to be able to automatically test our APIs at a very basic level so we could validate that there are discrepancies between what the client expects and what the server provides. To do that, I created a standalone API DSL with everything needed to describe your API but without any implementation details whatsoever. The API DSL lets the developer define a route (url), the HTTP verb expected, if the request should be authenticated or not, SSL or not, the param rules, default values and finally a response description (which was quite a controversial choice). All of these settings can be documented by the developer. This standalone DSL can then be consumed by different tools. For instance we have a tool extracting all the info into nicely formatted HTML doc for the game client developers. This tool doesn’t need to load the framework to just render the documentation. We also use this description at boot time to compile the validation rules and routes, allowing for a much faster request dispatch. And we also use these API description to generate some low level data for the client. Finally, we used the service description DSL to help create mocked service responses allowing the client team to test service designs without having to wait for the implementation streamlining the process.

Lessons learned: We had a lot of internal discussions about the need to define the response within the service description. Some argued that it’s a duplication since we already had a view and we could parse that to get most of what we needed (which is what the old system was doing). We ended up going with the response description DSL for a few critical reasons: testing and implementation simplicity. Testing: we need to have an API expectation reference and to keep this reference sane so we can see if something is changed. If we were to magically parse the response, we couldn’t test the view part of the code against a frame of reference. Implementation simplicity: magically parsing a view template is more tricky that it sounds, you would need to render the template with the right data to make it work properly. Furthermore, you can’t document a response easily in the view, and if you do, you arguably break the separation of concern between the description and the implementation. Finally, generated documentation isn’t enough and that’s why we decided to write English documentation, some being close to the code and some being just good old documentation explaining things outside of the code context.

Modularity

In order to make our code reusable we had to isolate each component and limit the dependencies. We wrote a very simple extension layer allowing each extension to registers itself once detected. The extension interface exposes the path of the extension, its type, models, services, controllers, migrations, seed data, dependencies etc.. Each extension is contained in a folder. (The extension location doesn’t matter much but as part of the framework boot sequence, we check a few default places.) The second step of the process is to check a manifest/config file that is specific to each title. The manifest file lists the extensions that should be activated for the title. The framework then activates the marked extensions and has access to libs, models, views, migrations, seed data and of course to load services (DSL mentioned earlier) etc…

Even though we designed the core extensions the best we could, there are cases where some titles will need to extend these extensions. To do that, we added a bunch of hooks that could be implemented on the title side if needed (Ruby makes that super easy and clean to do!). A good example of that is the login sequence or the player data.

Lessons learned: The challenge with modularity is to keep things simple and highly performing yet flexible. A key element to manage that is to stay as consistent as possible. Don’t implement hooks three different ways, try to keep method signatures consistent, keep it simple and organized.

 

Conclusion

It’s a bit early to say if this rewrite is a success or not and there are still lots of optimizations and technology improvements we are looking forward to doing. Only time will give us enough retrospect to evaluate our work. But because we defined the business value (mission statement) and the technical objectives, it is safe to say that the new framework meets the expectations quite well. On an early benchmark we noted a 10X speed improvement and that’s before drilling into the performance optimizations such as making all the calls non-blocking, using better connection pools, cache write through layer… However, there is still one thing that we will have to monitor: how much business value will this framework generate. And I guess that’s where we failed to define an agreed upon evaluation grid. I presume that if our developers spend more time designing and implementing APIs and less time debugging that could be considered business value. If we spend less time maintaining or fighting with the game engine, that would also be a win. Finally, if the player experience is improved we will be able to definitely say that we made the right choice.

To conclude, I’d like to highlight my main short coming: I failed to define metrics that would help us evaluate the real business value added to our products. What I consider a technical success might not be a business success. How do you, in your own domain, find ways to define clear and objective metrics?

, ,

No Comments