Lots of rubies, now what?


If you were at RubyConf 2009 or looked at the schedule, you saw that the big thing happening in the Ruby scene is the maturation of a many of the Ruby implementations:
BlueRuby, HotRuby, IronRuby, JRuby, MacRuby, Maglev, MRI, REE, Rubinius, SmallRuby…
Ruby developers really have plenty of choice when it comes to choosing an alternative Ruby implementation.

Is that a bad thing?

Turns out, Matz , Ruby author mentioned one ore time during RubyConf that it was actually a good thing for the Ruby ecosystem.

But maybe we should take a step back and look at the big picture. So here are some of my thoughts on the topic. (disclaimer: being part of the MacRuby team, I’m certainly biased)

Stop comparing alternative Ruby implementations against Ruby 1.8.x.

Even though a lot of people are still using Ruby 1.8.x, it’s totally unfair to compare against such an old version of Matz’s Ruby.
Ruby 1.9 is what people should compare against and believe me, Ruby 1.9.2 is going to change the game (much faster, cleaned up version).
Hopefully, all the Ruby alternative implementations will quickly become 1.9 compatible.

(On a site note, Rails 2.3.5 was released a few days ago which fixes some of the last few remaining Ruby 1.9 bugs)

Don’t trust the benchmarks

As we all know, there are lies, damn lies and statistics. Evan Phoenix explained that very clearly with lots of examples during his RubyConf talk and I totally agree with him.
Micro benchmarks only help implementers know where they are standing and find performance issue.
Let’s take a very simple example: JRuby.
What you might not know is that most JRuby benchmarks are usually ‘warmed up’.  Basically, to get the results you see in most reported benchmarks, you need to run the benchmark in a loop X amount of times and then run the benchmark. The reason for that is that JRuby uses HotSpot which needs to process the code to optimize the JITting and finally get your code to run faster.
The problem is that it is not a fair comparison. I was benchmarking Rails3 for instance and JRuby starts way slower than Ruby 1.8.6 and takes a little while to get as fast/faster than 1.9.1.
This is something you need to take in consideration since each deployment will slow down your app and you probably shouldn’t use JRuby if you are not using long running processes.
Another example would be MacRuby, even though the MacRuby project never released any benchmarks, it’s known to be a fast implementation.
Currently, the allocation of objects in MacRuby is actually still a bit costly and if you benchmark that, you can show that MacRuby is slow. Other operations are super optimized and both Rubinius and MacRuby will be way faster than anyone else. (check Phoenix’s talk at Ruby conf for more examples)
So, before choosing an implementation for performance reasons, benchmark against the code you are going to use.

There is only one real original Ruby

I think nobody will disagree with the fact that there is only one Ruby reference and that’s Matz.
Therefore is only one “real” Ruby and that is Matz’s. (Matz actually disagreed with me when I said that, since all the other implementations are also “real” but you get the point)
There is nothing wrong with Ruby’s alternatives but they stay alternatives to the original
MRI might not be the best option for every single case out there but it is and will stay the reference.

During RubyConf, I was joking with Matz and Koichi that maybe they should claim a Ruby tax to the Ruby implementations.
The reality is that Ruby is maintained by 5 to 10 contributors, most of them doing that on the side of their full time job.
Most of the other implementations have full time staff working hard on their implementations. That’s exciting but sad at the same time.

During the different talks given by the Ruby core team, it was clear that they really want to bridge the West/East gap and asked for help. It’s very honorable on their part and I hope we will see more contributions coming from outside Japan.

Finally, I think that even if people work hard to write the best Ruby implementation ever, let’s not forget to respect Matz’s project, team and achievements. People spent hours/weeks/months creating the Ruby language we love and even though there are some aspects that could be and are being improved, let’s not forget that when criticizing/comparing their work.

There is not only one way to do things

I don’t know where that’s coming from, but some people in our community seem to believe that there is only one way to do things and if you don’t do it their way, you do it wrong.
I personally have a hard time believing that and I even think it goes against what Ruby was designed for. (However, I must admit that sometimes I did/do think that some approaches were totally wrong :p)
It reminds me of people trying to convince you that God exists, other people trying to convince you that their religion is the only true one, or finally the other people trying to convince believers that they are wrong.
Maybe it is just that as humans we have a hard time dealing with people coming to a different conclusion than us and therefore we have to convince them that we are right and they are wrong.

The way I see it, for any new Ruby project, you should start be using with Ruby1.9. If it doesn’t fit your needs, then, consider an alternative. Here is how I look at the other alternatives (feel free to disagree):

  • BlueRuby – use if you are writing a SAP app
  • HotRuby – use if you want to use Ruby to write a Flash app
  • IronRuby – use it if you want to integrate with .NET and/or write a silverlight app
  • JRuby – use if you want to integrate with the Java world
  • MacRuby – use it if you want to integrate with Cocoa/want a native OSX application
  • Maglev – use if you want to use an object persistence layer and a distributed shared cache
  • REE – use if you have a ruby 1.8 web app that uses too much memory
  • Rubinius – use if you want to know what’s going on in your code
  • SmallRuby – don’t use it yet, but keep an eye on it

This is the way I see the various existing implementations and I think this is where they shine and why they were created in the first place. They are all complimentary and help bring the Ruby language further. The goal is always the same.

The future is exciting

Ruby is getting everywhere or almost. Matz’s Ruby is going to get multiVM support, various other improvements and get even faster and faster.
Alternative implementations are getting more and more mature and they grow the community by bringing people from different communities (.NET, java, obj-c/cocoa)

At the end of the day, we are all beneficiaries of the work done by all the implementers, thank you guys and keep up the good work!


Similar Posts
  1. #1 by James Britt - November 30th, 2009 at 16:57

    It’s great that there are assorted Ruby implementations, but to the best of my knowledge they are not all complete.

    Is there a one-stop site that compares the implementations and indicates to what extent they implement all of 1.8.x and 1.9.x?

    This has to be a key consideration when deciding what to use.

    • #2 by Matt Aimonetti - November 30th, 2009 at 17:12

      Very good point James.

      As far as I know such a site doesn’t exist yet. One could look at the RubySpecs to see how “Ruby 1.8.6 compatible” an implementation is. RubySpecs are not perfect but they should give you a good idea.

      Unfortunately, the RubySpecs are far from covering all of 1.9 but with Rubinius and JRuby focusing on becoming 1.9 compatible, this should improve quickly.

      Also, in some instances, you might not need a full compatibility. For instance I don’t believe that BlueRuby will ever try to be fully compatible as they focus on their own usage of the language and decided to skip some language features making the implementation harder/slower or just less secure.

      Ruby is a very rich language and Matz didn’t exclude the possibility to define a core language to make Ruby more portable and offer better performance optimizations. Time will tell.

  2. #3 by Tomas Jogin - December 1st, 2009 at 02:56

    Rails 2.3.5 still *only* supports ASCII in erb-files, breaking pretty much every non-english Rails app out there.

    • #4 by Tomas Jogin - December 1st, 2009 at 02:57

      Using Ruby 1.9, I mean. Sorry for the confusion.

    • #5 by Jeremy Kemper - December 1st, 2009 at 10:08

      Hey Tomas, 2.3.5 uses Encoding.default_external for erb templates. Try setting it to UTF-8 and upgrading your database drivers to a version that supports 1.9 string encodings.

      • #6 by Tomas Jogin - December 2nd, 2009 at 01:13

        Thanks for the reply, I thought I *did* try that. I’ll have a look at the (mysql) database drivers, didn’t see how they were relevant.

  3. #7 by Thomas Enebo - December 1st, 2009 at 08:56

    Matt, I think you are totally correct that people should run their code against implementations and get numbers which are meaningful for them. In this day and age we should all accept that runtimes are complex and not all people get the same performance benefits.

    In fact, I would love to have people report any results they get to us so we can investigate any performance issues and I am sure all the other impls feel the same way.

    As far as I know we have never posted a benchmark result without showing warm-up time in the results. They are all a stream of numbers where you see the numbers improve over time. I think it is fine to point out that JRuby has hotspot and dynamic optimizations take time to kick-in. Hell, we spent 3-4 minutes talking about this very subject at Rubyconf. So thanks for that.

    However you start the first paragraph with a ‘there are lies, damn lies and statistics’ and mention that Evan has some good material on the subject (which he does of course), but then you cite JRuby and warm-up time in the same paragraph. If people are not reading closely doesn’t it look like you are implying we are being deceptive? Do you think we are? Or am I reading too much into this?

    • #8 by Matt Aimonetti - December 1st, 2009 at 09:39

      Thomas, I think that most benchmarks out there are deceptive but I don’t think that it’s your fault. People usually have a hard time reading benchmarks and get confused.

      Looking at your latest post on JRuby http://www.engineyard.com/blog/2009/j-is-for-jvm-why-the-j-in-jruby/ it’s obvious that you are showing warm up time and even explain it on details. However if you take a random post from Yehuda http://yehudakatz.com/2009/01/16/status-update-a-fresh-look-at-callbacks/ you can see it doesn’t.

      I picked on JRuby because it’s the main implementation people are comparing CRuby against and it’s also the one suffering the most of jitting cost making benchmarks harder to read.

      However, the blog post was more targeting end users jumping to conclusions a bit quickly and saying “Ruby1.9 sucks”, “Y implementations is XX times faster” etc… something I know most if not all implementators have been fighting.

  4. #9 by grantmichaels - December 2nd, 2009 at 17:29

    “The problem is that it is not a fair comparison. I was benchmarking Rails3 for instance and JRuby starts way slower than Ruby 1.8.6 and takes a little while to get as fast/faster than 1.9.1. This is something you need to take in consideration since each deployment will slow down your app and you probably shouldn’t use JRuby if you are not using long running processes.
    Another example would be MacRuby, even though the MacRuby project never released any benchmarks, it’s known to be a fast implementation.”

    Until there is hosting based on mac servers …

  5. #10 by Stephen Bannasch - December 2nd, 2009 at 18:24

    Matt,

    In what situations do you think the warmup that JRuby needs to achieve it’s highest speeds will cause a problem?

    At least in the work that I do whenever I am running something where speed matters the warmup time is irrelevant.

    In general I use both Ruby and JRuby at the same time and I find that even without warmup JRuby is faster for most of what I do (Rails, data import and processing tasks) than MRI Ruby.

    I do find the JVM startup time annoying and almost always use plain Ruby for quick scripts I run from the command line over and over.

    I’m very interested in functionalities that new Ruby VMs support: MacRuby’s integration with Mac OS, MagLev’s distributed object-store, and JRuby’s integration with Java. Of these three only JRuby can pass most of the RubySpec tests cleanly. Until a new VM can pass these tests cleanly I don’t consider it suitable for general purpose Ruby work.

    I ran three simple benchmarks recently on MRI 1.8.6 and the latest versions of Ruby 1.9, JRuby, MagLev, MacRuby, and Rubinius. You can see the results in more detail (and links to the code) here: http://gist.github.com/244822

    The first benchmark comes directly from work we do and it involves measuring the speed to parse a 100k XML file with REXML and count the number of a specific leaf node (466 total nodes of this type). Actually when we do this I’l either use hpricot, libxml, or nokogiri but running this test with REXML is a good test for measuring string handling and regex parsing. In this JRuby is fastest (after warming up) followed closely by MRI 1.8.6 and Ruby 1.9. Rubinius is 4 times slower than MRI 1.8.6 and MacRuby is nine times slower!

    The next two benchmarks don’t come directly from our work and test different aspects of Ruby.

    The second benchmark is a Takeuchi function I saw on Charlie Nutter’s blog. This is mostly testing numerical processing and MacRuby was the fastest followed closely by MagLev. Rubinius, JRuby, and Ruby 1.9 were bunched together between seven and 10 times slower than MacRuby.

    The last benchmark I ran was a mergesort benchmark that I found in a post by Hongli Lai. In this JRuby was the fastest followed closely by Rubinius and MagLev. MacRuby and Ruby 1.9 followed a bit behind the first bunch.

  6. #11 by yarrrrr - December 4th, 2009 at 07:36

    “”"”"
    IronRuby – use it if you want to integrate with .NET and/or write a silverlight app
    JRuby – use if you want to integrate with the Java world
    “”"”"

    I don’t trust the library ecosystem for Ruby or Python… that’s why Iron/J-Ruby/Python are so important…

Comments are closed.