Archive for category rails

Learning from Rails’ failures

Ruby on Rails undisputedly changed the way web frameworks are designed. Rails became a reference when it comes to leveraging conventions, easy baked in feature set and a rich ecosystem. However, I think that Rails did and still does a lot of things pretty poorly.  By writing this post, I’m not trying to denigrate Rails, there are many other people out there already doing that. My hope is that by listing what I think didn’t and still doesn’t go well, we can learn from our mistakes and improve existing solutions or create better new ones.

Migration/upgrades

Migrating a Rails App from a version to the other is very much like playing the lottery, you are almost sure you will lose. To be more correct, you know things will break, you just don’t know what, when and how. The Rails team seems to think that everybody is always running on the cutting edge version and don’t consider people who prefer to stay a few version behind for stability reasons. What’s worse is that plugins/gems might or might not compatible with the version you are updating to, but you will only know that by trying yourself and letting others try and report potential issues.

This is for me, by far, the biggest issue with Rails and something that should have been fixed a long time ago. If you’re using the WordPress blog engine, you know how easy and safe it is to upgrade the engine or the plugins. Granted WordPress isn’t a web dev framework, but it gives you an idea of what kind of experience we should be striving for.

 

Stability vs playground zone

New features are cool and they help make the platform more appealing to new comers. They also help shape the future of a framework. But from my perspective, that shouldn’t come to the cost of stability. Rails 3′s new asset pipeline is a good example of a half-baked solution shoved in a release at the last minute and creating a nightmare for a lot of us trying to upgrade. I know, I know, you can turn off the asset pipeline and it got better since it was first released. But shouldn’t that be the other way around? Shouldn’t fun new ideas risking the stability of an app or making migration harder, be off by default and turned on only by people wanting to experiment? When your framework is young, it’s normal that you move fast and sometimes break, but once it matures, these things shouldn’t happen.

 

Public/private/plugin APIs

This is more of a recommendation than anything else. When you write a framework in a very dynamic language like Ruby, people will “monkey patch” your code to inject features. Sometimes it is due to software design challenges, sometimes it’s because people don’t know better. However,  by not explicitly specifying what APIs are private (they can change at anytime, don’t touch), what APIs are public (stable, will be slowly deprecated when they need to be changed) and which ones are for plugin devs only (APIs meant for instrumentation, extension etc..), you are making migration to newer versions much harder. You see, if you have a small, clean public API, then it’s easy to see what could break, warn developers and avoid migration nightmares. However, you need to start doing that early on in your project, otherwise you will end up like Rails where all code can potentially change anytime.

 

Rails/Merb merge was a mistake

This is my personal opinion and well, feel free to disagree, nobody will ever be able to know to for sure. Without explaining what happened behind closed doors and the various personal motivations, looking at the end result, I agree with the group of people thinking that the merge didn’t turn up to be a good thing. For me, Rails 3 isn’t significantly better than Rails 2 and it took forever to be released. You still can’t really run a mini Rails stack like promised. I did hear that Strobe (company who was hiring Carl Lerche, Yehuda Katz and contracted Jose Valim) used to have an ActionPack based, mini stack but it was never released and apparently only Rails core members really knew what was going on there. Performance in vanilla Rails 3 are only now getting close to what you had with Rails 2 (and therefore far from the perf you were getting with Merb). Thread-safety is still OFF by default meaning that by default your app uses a giant lock only allowing a process to handle 1 request at a time. For me, the flexibility and performance focus of Merb were mainly lost in the merge with Rails. (Granted, some important things such as ActiveModel, cleaner internals and others have made their way into Rails 3)

But what’s worse than everything listed so far is that the lack of competition and the internal rewrites made Rails lose its headstart.  Rails is very much HTML/view focused, its primarily strength is to make server side views trivial and it does an amazing job at that. But let’s be honest, that’s not the future for web dev. The future is more and more logic pushed to run on the client side (in JS) and the server side being used as an API serving data for the view layer. I’m sorry but adding support for CoffeeScript doesn’t really do much to making Rails evolve ahead of what it currently is. Don’t get me wrong, I’m a big fan of CoffeeScript, that said I still find that Rails is far from being optimized to developer web APIs in Rails. You can certainly do it, but you are basically using a tool that wasn’t designed to write APIs and you pay the overhead for that. If there is one thing I wish Rails will get better at is to make writing pure web APIs better (thankfully there is Sinatra). But at the end of the day, I think that two projects with different philosophies and different approaches are really hard to merge, especially in the open source world. I wouldn’t go as far as saying like others that Rails lost its sexiness to node.js because of the wasted time, but I do think that things would have been better for all if that didn’t happen. However, I also have to admit that I’m not sure how much of a big deal that is. I prefer to leave the past behind, learn from my own mistake and move on.

 

Technical debts

Here I’d like to stop to give a huge props to Aaron “@tenderlove” Patterson, the man who’s actively working to reduce the technical debts in the Rails code base. This is a really hard job and definitely not a very glamorous one. He’s been working on various parts of Rails including its router and its ORM (ActiveRecord). Technical debts are unfortunately normal in most project, but sometimes they are overwhelming to the point that nobody dares touching the code base to clean it up. This is a hard problem, especially when projects move fast like Rails did. But looking back, I think that you want to start tackling technical debts on the side as you move on so you avoid getting to the point that you need a hero to come up and clean the piled errors made in the past. But don’t pause your entire project to clean things up otherwise you will lose market, momentum and excitement. I feel that this is also very much true for any legacy project you might pick up as a developer.

 

Keep the cost of entry level low

Getting started with Rails used to be easier. This can obviously argued since it’s very subjective, but from my perspective I think we forgot where we come from and we involuntary expect new comers to come with unrealistic knowledge. Sure, Rails does much more than it used to do, but it’s also much harder to get started. I’m not going to argue how harder  it is now or why we got there. Let’s just keep in mind that it is a critical thing that should always be re-evaluated. Sure, it’s harder when you have an open source project, but it’s also up to the leadership to show that they care and to encourage and mentor volunteers to  focus on this important part of a project.

 

Documentation

Rails documentation isn’t bad, but it’s far from being great. Documentation certainly isn’t one of the Ruby’s community strength, especially compared with the Python community, but what saddens me is to see the state of the official documentation which, should, in theory be the reference. Note that the Rails guides are usually well written and provide value, but they too often seem too light and not useful when you try to do something not totally basic (for instance use an ActiveModel compliant object). That’s probably why most people don’t refer to them or don’t spend too much time there. I’m not trying to blame anyone there. I think that the people who contributed theses guides did an amazing job, but if you want to build a strong and easy to access community, great documentation is key. Look at the Django documentation as a good example. That said, I also need to acknowledge the amazing job done by many community members such as Ryan Bates and Michael Hartl consistently providing high value external documentation via the railscasts and the intro to Rails tutorial available for free.

 

In conclusion, I think that there is a lot to learn from Rails, lots of great things as well as lots of things you would want to avoid. We can certainly argue on Hacker News or via comments about whether or not I’m right about Rails failures, my point will still be that the mentioned issues should be avoided in any projects, Rails here is just an example. Many of these issues are currently being addressed by the Rails team but wouldn’t it be great if new projects learn from older ones and avoid making the same mistakes? So what other mistakes do you think I forgot to mention and that one should be very careful of avoiding?

 

Updates:

  1. Rails 4 had an API centric app generator but it was quickly reverted and will live as gem until it’s mature enough.
  2. Rails 4 improved the ActiveModel API to be simpler to get started with. See this blog post for more info.

, , , , , ,

36 Comments

Quick dive into Ruby ORM object initialization

Yesterday I did some quick digging into how ORM objects are initialized and the performance cost associated to that. In other words, I wanted to see what’s going on when you initialize an ActiveRecord object.

Before I show you the benchmark numbers and you jump to conclusions, it’s important to realize that in the grand scheme of things, the performance cost we are talking is small enough that it is certainly not the main reason why your application is slow. Spoiler alert: ActiveRecord is slow but the cost of initialization isn’t by far the worse part of ActiveRecord. Also, even though this article doesn’t make activeRecord look good, and I’m not trying to diss it. It’s a decent ORM that does a great job in most cases.

Let’s get started by the benchmarks number to give us an idea of the damage (using Ruby 1.9.3 p125):

 

                                                             | Class | Hash  | AR 3.2.1 | AR no protection | Datamapper | Sequel |
--------------------------------------------------------------------------------------------------------------------------------------
.new() x100000                                               | 0.037 | 0.049 | 1.557    | 1.536            | 0.027      | 0.209  |
.new({:id=>1, :title=>"Foo", :text=>"Bar"}) x100000          | 0.327 | 0.038 | 6.784    | 5.972            | 4.226      | 1.986  |

 

You can see that I am comparing the allocation of a Class instance, a Hash and some ORM models. The benchmark suite tests the allocation of an empty object and one with passed attributes. The benchmark in question is available here.

As you can see there seems to be a huge performance difference between allocating a basic class and an ORM class. Instantiating an ActiveRecord class is 20x slower than instantiating a normal class, while ActiveRecord offers some extra features, why is it so much slower, especially at initialization time?

The best way to figure it out is to profile the initialization. For that, I used perftools.rb and I generated a graph of the call stack.

Here is what Ruby does (and spends its time) when you initialize a new Model instance (click to download the PDF version):

 

Profiler diagram of AR model instantiation by Matt Aimonetti

 

This is quite a scary graph but it shows nicely the features you are getting and their cost associated. For instance, the option of having the before and after initialization callback cost you 14% of your CPU time per instantiation, even though you probably almost never use these callbacks. I’m reading that by interpreting the node called ActiveSupport::Callback#run_callbacks, 3rd level from the top. So 14.1% of the CPU time is spent trying to run callbacks. As a quick note, note that 90.1% of the CPU time is spent initializing objects, the rest is spent in the loop and in the garbage collection (because the profiler runs many loops). You can then follow the code and see how the code works, creating a dynamic class callback method on the fly (the one with the long name) and then recreating the name of this callback to call it each time the object is allocated. It sounds like that’s a good place for some micro optimizations which could yield up to 14% performance increase in some cases.

Another major part of the CPU time is spent in ActiveModel’s sanitization. This is the piece of code that allows you to block some model attributes to be mass assigned. This is useful when you don’t want to sanitize your incoming params but want to create or update a model instance by using all the passed user params. To avoid malicious users to modify some specific params that might be in your model but not in your form, you can protect these attributes. A good example would be an admin flag on a User object. That said, if you manually initialize an instance, you don’t need this extra protection, that’s why in the benchmark above, I tested and without the protection. As you can see, it makes quite a big difference. The profiler graph of the same initialization without the mass assignment protection logically ends up looking quite different:

 


Matt Aimonetti shows the stack trace generated by the instantiation of an Active Record model

 

Update: My colleague Glenn Vanderburg pointed out that some people might assuming that the shown code path is called for each record loaded from the database. This isn’t correct, the graph represents instances allocated by calling #new. See the addition at the bottom of the post for more details about what’s going on when you fetch data from the DB.

I then decided to look at the graphs for the two other popular Ruby ORMs:

Datamapper

 

and Sequel

 

 

While I didn’t give you much insight in ORM code, I hope that this post will motivate you to sometimes take a look under the cover and profile your code to see what’s going on and why it might be slow. Never assume, always measure. Tools such as perftools are a great way to get a visual feedback and get a better understanding of how the Ruby interpreter is handling your code.

UPDATE:

I heard you liked graphs so I added some more, here is what’s going on when you do Model.first:

 

Model.all

 

And finally this is the code graph for a call to Model.instantiate which is called after a record was retrieved from the database to convert into an Object. (You can see the #instantiate call referenced in the graph above).

 

, , , ,

9 Comments

Deploying a Rails 3.1 app – gotchas

Recently I had to build a new app as part of my research & development job at LivingSocial. My goal was to get the app up and running in just a few weeks, solid application architecture and graphic design included.
When you need to build an app quickly and you want it to have some solid foundations, Rails is quite useful.
I used Rails 3.1RCx so if we would to keep my app and push it to production, the engineering team wouldn’t have to update it and the transition should be seamless. I also quite like CoffeeScript and the app being quite heavy on JavaScript, the choice was easy. Furthermore, my coworker Bruce Williams is a fan of SCSS and he’s writing a PragProg book called “The Rails View” with other LivingSocialist: John Athayde. So you got the point, I’m using Rails3.1, but this post is about the challenges I faced when it was time to deploy and the solutions I found.

I’ll skip the intro to Rails 3.1 and how to use the new asset pipeline, refer to the Rails guide or one of the mainly posts referenced in this post (if I had properly read the guide, it would have saved me some valuable time, trust me, read it carefuly).

At that point last night, I had my app working great locally, Bruce created some awesome scss code using mixins and nested rules, the HTML was clean and working great, my CoffeeScript was brewing nicely, all was great until I tried to deploy to our QA environment.

JavaScript runtime dependency

The first thing you will notice is that you need the proper JavaScript runtime so the asset pipeline works properly. Not a big deal, you’ll find a lot of documentation about that. The problem is that you need to update your production environment or use depend on gem that will compile the required runtime (sounds dirty to me). So if you are deploying to many machines and you are using an image solution (EC2 AMI or other), you will need to update your image or spin new instances via updated chef/puppet recipes. In this case, the awesome team at LivingSocial had an image ready for me, so that wasn’t a big deal, but still, you need to take that in consideration as you are planning to update.

So the asset pipeline optimizes your asset management by processing/compiling asset files for you and optimizing their delivery. Instead of serving static files directly via public/images or public/javascripts you know serve them via the asset pipeline which will take care of compiling your CoffeeScript, grouping and minifying your JS and preprocessing all sorts of format. It also optimizes the caching process by giving a unique filename to each file based on the file metadata and gziping files. This is great, but you really, really, really don’t want to have your apps take care of that in production. Why wasting precious resources to serve assets when they can be prepared ahead of time. (by making Rails serve static assets, you are seriously reducing the throughput of  your app, please think of the children (or the dolphins/trees if you don’t like children))

Capistrano

Rails obviously has a preprocessor available as a rake task and you should update your deployment recipe to use that new feature. Here is my Capistrano code:

after 'deploy:update_code', 'deploy:compile_assets'
namespace :deploy do
  task :compile_assets do
    run "cd #{release_path}; RAILS_ENV=production rake assets:precompile"
  end
end

Well, my real code doesn’t hardcode the RAILS_ENV constant value, it’s in fact set in each env file, but I simplified it since most people only use 1 env outside of dev & test.

What that will do is compile all the files and dump them in public/assets/. But the file I had called bubble.png now becomes bubble-27543c671a3ab45141ee0d3216085009.png which means that my app is totally broken because images use in Bruce SCSS don’t load, my js files don’t load and the app is totally broken. Now this is least fun part, that I wish I had known before. This is where you go back and change your code so it uses magic to get the right file names.

Images

Fixing images was actually quite simple, in all my views, I just had to make sure I was using the image_tag helper everywhere.

CSS

SCSS files were a bit more tricky, I had to use the new scss preprocessor helpers you will find in the Rails guide (image_path and image_url). I first looked into using erb, but turned out it wasn’t needed and the end result is much cleaner.

Javascript/CoffeeScript

For the CoffeeScript files, I was referring to image assets in the code and of course all the links were broken. So I had to use ERB in my coffee which looked funky but it worked.

But to get that to work, you need to rename your coffee script and append erb at the end. For instance my feature.js.coffee script had to be renamed feature.js.coffee.erb. That made me cry a little inside, but oh well, at least its not a XML config file. Maybe soon we will start seeing code in filenames or filenames called my_feature.js.compressed.minified.coffee.erb.from_rails.mattetti.org
Also, be careful about the order of the file extensions, otherwise it won’t work. I thought I was done, ready to deploy my apps and this time the assets will show up properly. Turns out I was wrong :(

Rails asset precompilation env specific configuration.

My css looked good, the precompiling task had run fine but I was missing some js files. I scratched my head as I could only see some of my js files. I then realized that all my JS files were there but some of my CoffeeScript files were missing. The answer was given to me by Bruce who asked me if I had updated my “config.assets.precompile” setting. Sometimes I feel that Rails is trying to compete with Struts and here I was really surprised that by default Rails, in production mode only precompiles all static JS and application.js files, but none of the other dynamic js files. Now it does precompile all the scss files, but for a reason I just don’t understand, it’s not the case for the JS files. So, you have to go edit production.rb in the config/environments folder and add the other js files you would like Rails to precompile for you.

After making all these changes, I was able to redeploy my app and everything was working again. (you might want to tweak your apache/nginx config as explained in the Rails guide)

 

Conclusion

Don’t be fooled like me and expect that because you have an app running locally, deployment will work right away. Make sure to read about the new features and what’s needed. Overall, I think that the asset pipeline is a nice addon to Rails and if you don’t feel like using it, just can put/leave all your files in the public folder and everything will work just like before. I do have to say that I was surprised to see that even in a brand new Rails 3.1 project, Rails isn’t running in threaded mode by default. But that’s a different (old) story and I guess people still get more excited about asset management than framework raw performance ;)

 

 

 

7 Comments

Discussion with a Java switcher

For the past 6 months, I have had regular discussions with an experienced Java developers who switched to Ruby a couple years ago. Names have been changed to protect the guilty but to help you understand my friend ‘Duke’ better, you need to know that he has been a developer for 10 years and lead many complicated, high traffic projects. He recently released two Ruby on Rails projects and he has been fighting with performance issues and scalability challenges.

Duke is a happy Ruby developer but he sometimes has a hard time understanding why things are done in a certain way in the Ruby community. Here are some extracts from our conversations. My answers are only based on my own experience and limited knowledge. They are probably not shared by the entire  community, feel free to use the comment section if you want to add more or share your own answers.

Threads / Concurrency

Duke: Why does the Ruby community hate threads so much. It seems to be a taboo discussion and the only answer I hear is that threads are hard to deal with and that Ruby does not have a good threading implementation. What’s the deal there? If you want concurrent processing, threads are important!

Me: This is a very good question and I think there are two main reasons why threads and thread safety are not hot topics in the Ruby world. First, look at Ruby’s main implementation itself. If you are using an old version of Ruby (pre Ruby 1.9) you don’t use native threads but green threads mapping to only 1 native thread. Ilya has a great (yet a bit old) blog post explaining the difference, why it matters and also the role and effect of the Global Interpreter Lock (GIL). Also, even though Rubyists like to say that they live in the edge, most of them still use Ruby 1.8 and therefore don’t really see the improvements in Ruby 1.9 nor yet understand the potential of fibers.

The other part of the explanation is that the Rails community never really cared until recently. Yehuda Katz recently wrote a good article on thread safety in Ruby and if you read his post and Zed Shaw’s comment you will understand a bit better the historical background. As a matter of fact, the current version of Rails is not multi-threaded by default and developers interested in handling concurrent requests in one process should turn on this option. Thread safety appeared for the first time in Rails 2.2 but from what I saw, most people still don’t enable this option. There are many reasons for that. First, enabling thread safety disables some Rails features like automatic dependency loading after boot and code reloading. A lot of Rails developers take these two features for granted and don’t understand that they are technically “hacks” to make their lives easier. I do believe a lot of Rails developers don’t understand how threads, thread safety, concurrency, blocking IO and dependencies work. They care about getting their app done and meet their deadlines. They usually use and know Rails without paying too much attention to how Rails extends Ruby. Imagine what would happen if their code wasn’t thread safe and Rails wasn’t not using a global lock by default. Now you see why things are not exactly as you expect and also why some Rubyists are getting excited about new projects like node.js which takes a different approach.

The other thing to keep in mind is that at least 90 to 95% of the Rails apps out there don’t get more than a dozen requests/second (a million requests/day). You can scale that kind of load pretty easily using simple approaches like caching,  optimize your DB queries, load balancing to a couple servers. As a matter of fact, compared to the amount of people using Rails on a daily basis, only a very little amount of people are struggling with performance and scalability like you do. This is not an excuse but that explains why these people don’t care about the things you care about.

Rails is slow

Duke: I don’t understand why Rails developers are not more concerned about the speed/performance penalty induced by Rails.

Me: Again, Rails is fast enough for the large majority of developers out there. As you know, as a developer you have to always make compromises. The Rails team always said that development time is more expensive than servers and therefore the focus is on making development easier, faster and more enjoyable. However to get there, they have to somewhat sacrifice some performance. What can be totally unacceptable for you is totally fine for others and your contribution is always welcome. This is probably the root cause of the things you don’t like in Rails. Rails was built for startups, by startup developers and you don’t fall in this category. People contributing new features and fixes are the people using Rails for what it is designed to do. There is no real ‘Enterprise’ support behind Rails and that might be why you feel the way you feel. Since you find yourself questioning some key Rails conventions and you are struggling with missing features, it looks  to me that you chose the wrong tool for the job since you don’t even use 70% of the Rails features and are dreaming of things such 3 tier architecture. Sinatra might be a better fit for you if you want lower level control, less conventions and less built-in features.

Object allocation / Garbage Collection

Duke: I recently read that Twitter was spending 20% of its request cycles in the GC, am I the only finding that concerning?

Me: Most people don’t realize how the GC works and what it means to allocate objects since Ruby does that automatically. But at the same time, most of these people don’t really see the affect of the Garbage Collection since they don’t have that much traffic or they scale in ways that just skips their Ruby stack entirely. (Or they just blame Ruby for being slow)

If you are app deals with mainly reads/GET requests, using HTTP caching (Rails has that built-in) and something like Varnish/Rack-cache will dramatically reduce the load on your server apps. Others don’t investigate their issues and just add more servers. As mentioned in a previous post, some libraries like Builder are allocating LOTS more objects than others (Nokogiri), use the existing debugging tools to see where your object allocations occur and try to fix/workaround these. In other words, Ruby’s GC isn’t great but by ignoring its limitations, we made things even worse. My guess is that the GC is going to improve (other implementations already have better GCs) and that people will realize that Ruby is not magic and critical elements need to be improved.

Tools

Duke: I really have a hard time finding good tools to help scale my apps better and understand where I should optimize my code.

Me: It is true that we area lacking tools but things are changing. On top of the built-in tools like ObjectSpace, GC::Profiler, people interested in performance/debugging are working to provide the Ruby community with their expertise, look at memprof and ruby-debug for instance. Of course you can also use tools such as Ruby-prof, Kcachegrind, Valgrind and GDB. (1.9.2 was scheduled to have DTrace support but I did not check yet). Maybe you should be more explicit about what tools you miss and how we could solve the gap.

ActiveRecord

Duke: ActiveRecord doesn’t do what I need. How come there is no native support for master/slave DBs, sharding, DB view support is buggy,  suggested indexes on queries is not built-in and errors are not handled properly (server is gone, out of sync etc..)?

Me: You don’t have to use ActiveRecord, you could use any ORM such as Sequel, DataMapper or your own. But to answer your question, I think that AR doesn’t do everything you want because nobody contributed these features to the project and the people maintaining ActiveRecord don’t have the need for these features.

What can we do?

We, as a community, need to realize that we have to learn from other communities and other programming languages, this kind of humorous graph is unfortunately not too far from reality.

Bringing your expertise and knowledge to the Ruby community is important. Looking further than just our own little will push us to improve and fulfill the gaps. Let the community know what tools you are missing, the good practices you think we should be following etc…

Take for instance Node.js, it’s a port of Ruby’s EventMachine / Python’s twisted. There is no reasons why the Ruby or Python versions could not do what the Javascript version does. However people are getting excited and are jumping ship. What do we do about that? One way would be to identify what makes node more attractive than EventMachine and what needs to be done so we can offer what people are looking for. I asked this question a few weeks ago and the response was that a lot of the Ruby libraries are blocking and having to check is too bothersome. Maybe that’s something that the community should be addressing. Node doesn’t have that many libraries and people will have to write them, in the mean time we can make our libs non-blocking. Also, let’s not forget that this is not a competition and people should choose the best tool for their projects.

Finally, things don’t change overnight, as more people encounter the issues you are facing, as we learn from others, part of the community will focus on the problems you are seeing and things will get better. Hopefully, you will also be able to contribute and influence the community to build an even better Ruby world.

,

14 Comments

Ruby object allocation & why you should care

Recently I was tasked with finding how to optimize a web application with heavy traffic. The application (a Rails 2.3.x app) gets about 3 million requests per hour and most of these requests cannot really be easily cached so they go through the entire stack.

This is probably not the case of most web apps out there. None the less, my findings my help you understand Ruby better and maybe think differently about memory management.

This is certainly not an advanced GC blog post, I will try to keep it as simple as possible. My goal is to show you how Ruby memory allocation works and why it can affect your app performance and finally, how can you avoid to allocate to many objects.

Ruby memory management.

Rubyists are quite lucky since they don’t have to manage the memory themselves. Because developers are lazy and Matz developed his language for people and not machine, memory is managed “magically”. Programming should be fun and managing memory isn’t really considered fun (ask video game developers or iOS programmers ;) ).

So in Ruby, the magical memory management is done by a Garbage Collector. The GC’s job is to run and free objects that were previously allocated but not used anymore. Without a GC we would saturate the memory available on the host running the program or would have to deallocate the memory manually. Ruby’s GC uses a conservative, stop the world, mark-and-sweep collection mechanism.  More simply, the garbage collection runs when the allocated memory for the process is maxed out. The GC runs and blocks all code from being executed and will free unused objects so new objects can be allocated.

Joe Damato did a great talk on that matter during last RailsConf

Garbage Collection and the Ruby Heap

The problem is that Ruby’s GC was not designed to support hundred thousand objects allocation per second. Unfortunately, that’s exactly what frameworks like Ruby on Rails do, and you might contribute to the problem too without even knowing it.

Does it really matter?

I believe it does. In my case improving the object allocation means much better response time, less servers, less support and less headaches. You might think that servers are cheaper than developers. But more servers mean more developer time spent fixing bugs and more IT support. That’s why I think, memory management is something Ruby developers should be aware of and should take in consideration, especially the ones writing frameworks, libraries or shared code.

I am using Ruby 1.9 so I could not profile my Rails 2.x app using memprof, instead I wrote a simple and basic middleware that keeps track of the memory allocation/deallocation and GC cycles during a web request (Ruby1.9 only). One of my simple Rails2 actions (1 DB call, simple view) is allocating 170,000 objects per requests. Yes, you read right: 170k objects every single request. At 3 million requests/hour, you can imagine that we are spending a LOT of time waiting for the GC. This is obviously not 100% Rails fault as I am sure our code is contributing to the problem. I heard from the memprof guys that Rails was allocating 40k objects. I decided to check Rails3.

After warming up, a basic Rails3 ‘hello world’ app clocks at about 8,500 objects allocated per request, forcing the GC to run more or less every 6 requests. On my machine (mac pro) the GC takes about 20ms to free the objects. A Rack ‘hello world’ app clocks at 7 objects per request and a Sinatra app at 181 objects. Of course you can’t really compare these different libraries/frameworks but that gives you an idea of the price you pay to get more features.

One thing to remember is that the more objects you allocate, the more time you “lose” at code execution. For more developers, it probably doesn’t matter much, but if you should still understand that concept especially if you decide to contribute to the OSS community and offer patches, libraries, plugins etc…

What can I do?

Be aware that you are allocating  objects, for instance something as simple as 100.times{ ‘foo’ } allocates 100 string objects (strings are mutable and therefore each version requires its own memory allocation).

Make sure to evaluate the libraries you use, for instance switching a Sinatra XML rendering action from Builder to Nokogiri XML Builder saved us about 12k object allocations (Thanks Aaron Patterson). Make sure that if you are using a library allocating a LOT of objects, that other alternatives are not available and your choice is worth paying the GC cost. (you might not have a lot of requests/s or might not care for a few dozen ms per requests). You can use memprof or one of the many existing tools to check on the GC cycles using load tests or in dev mode. Also, be careful to analyze the data properly and to not only look at the first request. Someone sent me this memory dump from a Rails3 ‘hello world’ with Ruby 1.8.7 and it shows that Rails is using 331973 objects.  While this is totally true, it doesn’t mean that 330k objects are created per request. Instead that means that 330k objects are currently in memory. Rubygems loading already allocate a lot of objects, Rails even more but these objects won’t be GC’d and don’t matter as much as the ones allocated every single request. The total amount of memory used by a Ruby process isn’t that important, however the fluctuation forcing the GC to run often is. This is why my middleware only cares about the allocation change during a request. (The GC should still traverse the entire memory so, smaller is better)

The more object allocation you do at runtime/per request, the more the GC will need to run, the slower your code will be. So this is not a question of memory space, but more of performance. If your framework/ORM/library/plugin allocates too many objects per request maybe you should start by reporting the problem and if you can, offer some patches.

Here are some hints about memory allocation:

Creating a hash object really allocates more than an object, for instance {‘joe’ => ‘male’, ‘jane’ => ‘female’} doesn’t allocate 1 object but 7. (one hash, 4 strings + 2 key strings) If you can use symbol keys as they won’t be garbage collected. However because they won’t be GC’d you want to make sure to not use totally dynamic keys like converting the username to a symbol, otherwise you will ‘leak’ memory.

Looking at a GC cycle in the Rails3 hello world example shows what objects get deallocated:

GC run, previous cycle was 6 requests ago.

GC 203 invokes. (amount of cycles since the program was started)
Index   1

Invoke Time(sec)   25.268

Use Size(byte)   4702440

Total Size(byte)   7307264

Total Object   182414

GC Time(ms) 22.35600000000204090611

## 56322 freed objects. ##
[78%] 44334 freed strings.
[7%] 4325 freed arrays.
[0%] 504 freed bignums.
[1%] 613 freed hashes.
[0%] 289 freed objects.
[5%] 3030 freed parser nodes (eval usage).

I did not list all the object types but it’s pretty obvious that the main issue in the case of Rails is string allocation. To a certain extend the allocated arrays and the runtime use of eval are not helping either. (what is being eval’d at runtime anyway?)

If you use the same string in various place of you code, you can “cache” them using a local var, instance variable, class variable or constant. Sometimes you can just replaced them by a symbol and save a few allocations/deallocations per request. Whatever you do tho, make sure there is a real need for it. My rule of thumb is that if some code gets exercised by 80% of the requests, it should be really optimized and avoid extra allocations so the GC won’t slow us down.

What about a better GC?

That’s the easy answer. When I mentioned this problem with Rails, a lot of people told me that I should use JRuby or Rubinius because their GC were much better. Unfortunately, that’s not that simple and choosing an alternative implementation requires much further research and tests.

But what annoys me with this solution is that using it is not solving the issue, it’s just working around it. Yes, Ruby’s GC isn’t that great but that’s the not the key issue, the key issue is that some libraries/frameworks allocate way too many objects and that nobody seems to care (or to even know it). I know that the Ruby Core Team is working on some optimizations and I am sure Ruby will eventually get an improved GC. In the meantime, it’s easy to blame Matz, Koichi and the rest of the core team but again, it’s ignoring that the root cause, totally uncontrolled memory allocation.

Maybe it’s time for us, Rubyists, to think a bit more about our memory usage.

15 Comments

Au Revoir Rails community

Time really flies!

Back in December 2005, Ruby on Rails 1.0 was released to the masses. I remember that was when I first got interested in Rails. Six months later, I was doing Rails development full time.

Rails pushed me to contribute to the project, to write plugins, to improve my Ruby knowledge, to release gems and to become a better engineer overall. I then joined the Merb project, focusing on problems I was facing in the various client projects I had back then.

The competition between Rails and Merb turned into a constant confrontation, splitting the Ruby community into two camps. A resolution was later achieved by merging the two teams and focusing our energy on Rails 3. This is how I became a part of the Activism team with Gregg and Ryan. In this new role I was given the opportunity to meet lots of different people from various backgrounds and different communities. I really had a lot of fun.

However, things have changed for me. I won’t be at Rails Conf 2010 because in a few weeks I will become a father for the first time. And with that, an obvious priority shift. My day job working on Playstation games is also quite time consuming and the little free time I manage to get to work on my own projects is spent on my MacRuby book. The disconnect between the Rails community and myself is probably more evident now than ever. The challenges encountered by most Railists are so different from the ones I face daily that I think others would do a much better job than I at advocating for Rails. So this is why I believe it’s time for me to step away from the Rails community, kick back and relax (and get ready to change a lot of diapers).

This is an “au revoir“, not an “Adieu“. I will continue to keep an eye on Rails 3 and the fast growing ecosystem.

I will still be writing Ruby for a living and will hopefully keep contributing to the projects I use. And I plan to keep on attending to Ruby conferences around the world just as soon as my kid is old enough to travel with me ;)

Finally, with the imminent release of Rails 3, I hope to see even more people stand up and advocate for Ruby on Rails the way Gregg Pollack, Ryan Bates and many others have done so far.

13 Comments

Ruby, Rack and CouchDB = lots of awesomeness

Over the weekend, I spent some time working on a Ruby + Rack +CouchDB project. Three technologies that I know quite well but that I never put to work together at the same time, at least not directly.  Let’s call this Part I.

Before we get started, let me introduce each component:

  • Ruby : if you are reading this blog, you more than likely know at least a little bit about, what I consider, one of the most enjoyable programming language out there. It’s also a very flexible language that lets us do some interesting things. I could have chosen Python to do the same project but that’s a whole different topic. For this project we will do something Ruby excels at: reopening existing classes and injecting more code.
  • Rack: a webserver interface written in Ruby and inspired by Python’s WSGI. Basically, it’s a defined API to interact between webservers and web frameworks. It’s used by most common Ruby web frameworks, from Sinatra to Rails (btw, Rails3 is going to be even more Rack-focused than it already is). So, very simply put, the webserver receives a request, passes it to Rack, that converts it, passes it to your web framework and the web framework sends a response in the expected format (more on Rack later).
  • CouchDB: Apache’s document-oriented database. RESTful API, schema-less, written in Erlang with built-in support for map/reduce. For this project, I’m using CouchRest, a Ruby wrapper for Couch.

Goal: Log Couch requests and analyze data

Let’s say we have a Rails, Sinatra or Merb application and we are using CouchRest (maybe we are using CouchRest and ActiveRecord, but let’s ignore that for now).

Everything works fine but we would like to profile our app a little and maybe optimize the DB usage. The default framework loggers don’t support Couch. The easy way would be to tail the Couch logs or look at the logs in CouchDBX. Now, while that works, we can’t really see what DB calls are made per action, so it makes any optimization work a bit tedious. (Note that Rails3 will have some better conventions for logging, making things even easier)

So, let’s see how to fix that. Let’s start by looking at Rack.

Rack Middleware

Instead of hacking a web framework specific solution, let’s use Rack. Rack is dead simple, you just need to write a class that has a call method.
In our case, we don’t care about modifying the response, we just want to instrument our app. We just want our middleware to be transparent and let our webserver deal with it normally.

Here we go … that wasn’t hard, was it? We keep the application reference in the @app variable when a new instance of the middleware is created. Then when the middleware is called, we just call the rest of the chain and pretend nothing happened.

As you can see, we just added some logging info around the request. Let’s do one better and save the logs in CouchDB:

Again, nothing complicated. In our rackup file we defined which Couch database to use and we passed it to our middleware (we change our initialize method signature to take the DB).
Finally, instead of printing out the logs, we are saving them to the database.

W00t! At this point all our requests have been saved in the DB with all the data there, ready to be manipulated by some map/reduce views we will write. For the record, you might want to use the bulk_save approach in CouchDB which will wait for X amount of records to save them in the DB all at once. Couch also let’s you send new documents, but only save it to the DB every X documents or X seconds.

As you can see, our document contains the timestamps and the full environment as a hash.

All of that is nice, but even though we get a lot of information, we could not actually see any of the DB calls made in each request. Let’s fix that and inject our logger in CouchRest (you could apply the same approach to any adapter).

Let’s reopen the HTTP Abstraction layer class used by CouchRest and inject some instrumentation:

Again, nothing fancy, we are just opening the module, reopening the methods and wrapping our code around the super call (for those who don’t know, super calls the original method).

This is all for Part I. In Part II, we’ll see how to process the logs and make all that data useful.

By the way, if you make it to RailsSummit, I will be giving a talk on Rails3 and the new exciting stuff you will be able to do including Rack based stuff, CouchDB, MongoDB, new DataMapper etc..

, , , , ,

3 Comments