Archive for category merb

Ruby, Rack and CouchDB = lots of awesomeness

Over the weekend, I spent some time working on a Ruby + Rack +CouchDB project. Three technologies that I know quite well but that I never put to work together at the same time, at least not directly.  Let’s call this Part I.

Before we get started, let me introduce each component:

  • Ruby : if you are reading this blog, you more than likely know at least a little bit about, what I consider, one of the most enjoyable programming language out there. It’s also a very flexible language that lets us do some interesting things. I could have chosen Python to do the same project but that’s a whole different topic. For this project we will do something Ruby excels at: reopening existing classes and injecting more code.
  • Rack: a webserver interface written in Ruby and inspired by Python’s WSGI. Basically, it’s a defined API to interact between webservers and web frameworks. It’s used by most common Ruby web frameworks, from Sinatra to Rails (btw, Rails3 is going to be even more Rack-focused than it already is). So, very simply put, the webserver receives a request, passes it to Rack, that converts it, passes it to your web framework and the web framework sends a response in the expected format (more on Rack later).
  • CouchDB: Apache’s document-oriented database. RESTful API, schema-less, written in Erlang with built-in support for map/reduce. For this project, I’m using CouchRest, a Ruby wrapper for Couch.

Goal: Log Couch requests and analyze data

Let’s say we have a Rails, Sinatra or Merb application and we are using CouchRest (maybe we are using CouchRest and ActiveRecord, but let’s ignore that for now).

Everything works fine but we would like to profile our app a little and maybe optimize the DB usage. The default framework loggers don’t support Couch. The easy way would be to tail the Couch logs or look at the logs in CouchDBX. Now, while that works, we can’t really see what DB calls are made per action, so it makes any optimization work a bit tedious. (Note that Rails3 will have some better conventions for logging, making things even easier)

So, let’s see how to fix that. Let’s start by looking at Rack.

Rack Middleware

Instead of hacking a web framework specific solution, let’s use Rack. Rack is dead simple, you just need to write a class that has a call method.
In our case, we don’t care about modifying the response, we just want to instrument our app. We just want our middleware to be transparent and let our webserver deal with it normally.

Here we go … that wasn’t hard, was it? We keep the application reference in the @app variable when a new instance of the middleware is created. Then when the middleware is called, we just call the rest of the chain and pretend nothing happened.

As you can see, we just added some logging info around the request. Let’s do one better and save the logs in CouchDB:

Again, nothing complicated. In our rackup file we defined which Couch database to use and we passed it to our middleware (we change our initialize method signature to take the DB).
Finally, instead of printing out the logs, we are saving them to the database.

W00t! At this point all our requests have been saved in the DB with all the data there, ready to be manipulated by some map/reduce views we will write. For the record, you might want to use the bulk_save approach in CouchDB which will wait for X amount of records to save them in the DB all at once. Couch also let’s you send new documents, but only save it to the DB every X documents or X seconds.

As you can see, our document contains the timestamps and the full environment as a hash.

All of that is nice, but even though we get a lot of information, we could not actually see any of the DB calls made in each request. Let’s fix that and inject our logger in CouchRest (you could apply the same approach to any adapter).

Let’s reopen the HTTP Abstraction layer class used by CouchRest and inject some instrumentation:

Again, nothing fancy, we are just opening the module, reopening the methods and wrapping our code around the super call (for those who don’t know, super calls the original method).

This is all for Part I. In Part II, we’ll see how to process the logs and make all that data useful.

By the way, if you make it to RailsSummit, I will be giving a talk on Rails3 and the new exciting stuff you will be able to do including Rack based stuff, CouchDB, MongoDB, new DataMapper etc..

, , , , ,


RailsConf 2009

RailsConf 2009 has now finished.  This time last year, no one would have ever guessed that the Merb and Rails teams would join forces and focus on what will hopefully be known as one of the best Web Frameworks.

It was encouraging to see so many people excited about what’s being ported over from Merb and the new options available to people who are currently limited by the existing stack. For those interested in pushing Rails further and doing stuff out of the norm, here are my slides. Arthur Zapparoli from Brazilian Rails squad recorded most of the talk and told me he will upload the video ASAP. You can also read Yehuda Katz’ blog which covers what he talked about.

It was really great to meet a lot of new people as well as people I only knew via IRC/IM/twitter.

It was a great honor to finally meet Dan Kubb (DataMapper), Ninh Hernandez-Búi & Hongli McLovin Lai (Phusion), Peter Cooper (RubyInside), Raimonds Simanovskis (Oracle adapter for AR), Arun Gupta (Sun/Glassfish),  Jeremy Hinegardner (crate), Michael Maxilien (IBM), Dana Jones (railsbridge), Zach Zolton & Geoff Buesing (CouchRest) and of course the Brazilian crew (lots of awesome .br guys came this year, I’m looking forward to RailsSummit) and last but not least, the French speaking crew (I’m glad to see Ruby is picking up back home). (I know I’m forgetting people… sorry about that)

It was also really nice to talk with some experts like Dave Astels, Aslak Hellesøy, Rich Kilmer, David Chelimsky, Ryan Brown, Derek Neighbors etc.. to get their feedback on various projects I’m working on.

Leaving Vegas, I feel like the Rails community is expanding quickly (it was the first RailsConf for 1/4 to 1/5 of the attendees) and that the community is organizing itself to welcome a new audience (better documentation, great initiatives like, willingness to help), as well as trying to be more available to the ‘Enterprise’ world.

These feelings were enforced during our Rails Activism BOF and after talking with 3rd party developers and sponsors really trying to solve problems that newcomers to Rails are now facing. This is an exciting time.

, , , ,


Merb 1.0.11 (minor release)

Following the DataMapper 0.9.11 release, we just pushed a new minor Merb release.

This release is mainly targeting new developers and Windows users wanting to install the full Merb stack. Others can simply update their dependencies if they use the dependencies.rb file or install the new gems if nothing is bundled and no hard dependencies are set.

Merb is a metagem which installs a bunch of other gems (merb-core, DataMapper and a lot of small gems). The problem was that Merb was trying to install DM and dm-types, unfortunately, dm-types had a dependency on a gem which couldn’t be installed on Windows. All of that is now fixed and Windows users can install Merb 1.0.11 without having to manually pick the gems they need.

This release also includes a fix for people using CouchRest, a CouchDB Document Mapping DSL.

Merb 1.1 is still planned to be released in April. A majority of the work has been done, but since Yehuda and myself are going to be traveling, the release will be slightly delayed.

The great news regarding Merb 1.1 is that, on top of being fully Ruby 1.9 compatible, and using action-orm, and being closer to Rack, Yehuda and Carl have been working on the router to make it awesomer and ready for mountable apps :)

Stay tuned for more news.

update: People using CouchRest or another CouchDB ORM/DSL make sure you define your resources route with an identifier:

resources :articles,        :identify => :id

Otherwise, resource(@article) won’t work. (This is usually done by the merb orm plugin and I might add it to CouchRest in the future)

1 Comment

Merb 1.0.10 (minor release)

We just pushed a really tiny update because of a bug in 1.0.9 affecting people using: Merb::Config[:max_memory]

Merb::Config[:max_memory] has been fixed and now polls for memory usage every 30s instead of 0.25s. (memory is set in KB)

This new version also uses DataMapper.repository instead of Kernel#repository (DM and Vlad related bug fix)

We are still on schedule for Merb 1.1 which is planned for early April. (If you install Merb from our edge server, the latest version should already be Ruby 1.9 compatible)


No Comments

Merb 1.1 roadmap

Yesterday, Carl Lerche, Yehuda Katz and myself had a meeting to discuss Merb 1.1′s roadmap.

Key items on the agenda were:

  • Ruby 1.9
  • Mountable apps
  • migration path to Rails3

After spending some time arguing back and forth, we decided that few things had to happen before we could migrate the current slices to pure mountable apps. Freezing the releases while waiting to get that done doesn’t seem like a good idea.

Therefore, here is the plan for Merb 1.1:

  • Ruby 1.9 full compatibility (with the very appreciated help from Maiha and Genki (preview of their work there)). Because Merb depends on different gems, we also need to work with 3rd party developers to make sure Merb’s dependencies are Ruby 1.9 compatible
  • Merb helpers (fixes, enhancement and missing helpers)
  • Make Merb controllers, rack endpoints. This is a fully transparent change for the framework users. By making this switch, we offer more flexibility to the router (you can mount a sinatra app for instance) and we adopt the same approach as Rails 2.3 making the transition to 3 much easier and facilitating the implementation of mountable apps. Again, this is an internal change and you won’t have to change anything in your application.
  • Router optimization, Carl has been working on few tricks/optimizations for the router that will be available in 1.1
  • Namespacing. If we want to make every single application, a potential mountable app, we need to namespace our applications. This is something we already do with slices, but currently generated applications are not namespaced. We are planning on doing that for 1.1 (backward compatible) to make mountable apps easier.
  • ActiveORM. ActiveORM is an ORM abstraction layer developed by Lori Holden (AT&T interactive) which helps with helpers and other parts of your code accessing your ORM directly. For instance, the errors_for method need to be implemented differently depending on the underlying ORM. ActiveORM offers mapping for the 3 major Ruby ORMs: ActiveRecord, DataMapper and Sequel but let you hook to it if you want to extend ActiveORM to support your own ORM.

There is plenty to do but we decided to still try to have an expected release date: around the end of March. As always in the OSS world, this is something we hope for, not a promise ;)

What about Merb 1.2?

1.2 will focus on mountable apps and we hope to get started on a separate branch before we release 1.1. However, mountable apps are hard to spec and we need a better feedback from the community. Tell us what you like with slices and what you don’t like. Let us know how you would like the new mountable apps to work. Be as precise as possible. (you can leave a comment here or on the mailing list)



merb progress

As they say: fail early, fail often. Well, I’ve been failing to blog recently, but as always I have some good excuses ;)

  • Yehuda has been blogging a lot about the work done on the merge.
  • I have been busy working on probably the awesomest CouchDB Ruby DSL/ORM.
  • I have been working with the Rails Activists on the new wiki and many other projects.
  • My sister is visiting from France :) (most of my free time is not spent in front of a computer anymore :p)
  • I’ve been playing with MacRuby (see the end of the post)
  • Ohh and spent some time maintaining Merb and preparing 1.0.9

Merb 1.0.9

Merb has some issues with the merb-cache settings set in init.rb being called twice when you use some Rake tasks. (rake db:automigrate for instance).The problem is due to the fact that the rake tasks load the dependencies twice:

  • Merb rake file loads all of your app dependencies and their rake tasks. (for instance rake db:automigrate is a rake task coming from merb_datamapper)
  • The task you are invoking might start merb itself to load the models etc.. and starting Merb reloads the dependencies.

Dependencies have the option to have a require block. A require block is a Proc that gets called when the dependency is being required. In the case of 1.8.1, we added a default block to set merb-cache. The problem was that the block was being set and called twice and merb-cache was complaining that the default cache was already setup.

To fix this issue we worked on a band-aid type solution (read: kinda evil but ok). Even if you start Merb multiple times, the init.rb file will no only load once and the dependency require blocks will only be called once. However, in the case of merb-cache, the setup block is being called everytime you go through the bootloader. That’s why we added a verification on the block itself. In the long term, we are going to fix things nicely and optimize the way merb-cache loads.

We also addressed some other issues. Some people have been pointed out issues with some merb-helpers and patches were provided by the community (thanks a lot).

Talking about patches, we also accepted a patch fixing a small problem with merb-auth and some open-id servers.

Ruby 1.9.1: We started going through all the different places were things are not compatible yet, most of the work that needs to be done right away is focused on the router. Carl has been working on that and you can check his branch to see how we are doing. During the next week we will try to get things ironed out, we still don’t know how we are going to deal with the fact that action-args uses parsetree which is not 1.9.1 compatible.

The upcoming version also gets a brand new feature: memory monitoring by the merb master process. The master process checks that the workers don’t use too much memory ( you can set what you consider beign too much memory using Merb::Config[:max_memory]) and if one of the workers reaches the limit, it does a kill -1. It then waits a configured amount of seconds (defaults to 5) then kill using -9.

Finally, we also fixed a bunch of tiny issues.

1.0.9 should be released in the next few days.


Finally, because for those interested in MacRuby here are my slides from last night’s SDRuby meetup:


latest Merb and Rails 3.0 news

Basically, the work started, we are getting familiar with the rails code base and are optimizing things slowly but surely with a focus on testing JRuby. The Merb router is being optimized and ported over to Rails 3.0. Rails and Merb developers will be able to stick to their DSL (so we stay backward compatible). Merb bootloader is also being ported over without breaking the backward compatibility. Finally, ActiveSupport is being broken down in more manageable/independant chunks so people will be able to pick only what they want to use. A “mini” version is also on the work.

  • Merb 1.0.7 got released yesterday with a bunch of bug fixes.
  • merb_sequel 1.0 should be released sometime this week and i’m planning on adding rails i18n syntax support to merb_babel.