Archive for category Misc

Point of view on career development

Last week I wrote a blog post that turned out to be my most popular blog post of the year with 40,000 unique visitors mainly due to the article being featured on the front page of Hacker News for almost 24 hours.

The article focuses on our difficulties as engineers to build a career and my own experience dealing with it.

 

Here are some Twitter responses:

 

 

I inbox almost exploded and days later I’m still trying to reply to all the people who contacted me for advice. Once manage to collect my thoughts I’ll probably write a follow up article based on what I learned from the various emails I received.

No Comments

Learning from Rails’ failures

Ruby on Rails undisputedly changed the way web frameworks are designed. Rails became a reference when it comes to leveraging conventions, easy baked in feature set and a rich ecosystem. However, I think that Rails did and still does a lot of things pretty poorly.  By writing this post, I’m not trying to denigrate Rails, there are many other people out there already doing that. My hope is that by listing what I think didn’t and still doesn’t go well, we can learn from our mistakes and improve existing solutions or create better new ones.

Migration/upgrades

Migrating a Rails App from a version to the other is very much like playing the lottery, you are almost sure you will lose. To be more correct, you know things will break, you just don’t know what, when and how. The Rails team seems to think that everybody is always running on the cutting edge version and don’t consider people who prefer to stay a few version behind for stability reasons. What’s worse is that plugins/gems might or might not compatible with the version you are updating to, but you will only know that by trying yourself and letting others try and report potential issues.

This is for me, by far, the biggest issue with Rails and something that should have been fixed a long time ago. If you’re using the WordPress blog engine, you know how easy and safe it is to upgrade the engine or the plugins. Granted WordPress isn’t a web dev framework, but it gives you an idea of what kind of experience we should be striving for.

 

Stability vs playground zone

New features are cool and they help make the platform more appealing to new comers. They also help shape the future of a framework. But from my perspective, that shouldn’t come to the cost of stability. Rails 3′s new asset pipeline is a good example of a half-baked solution shoved in a release at the last minute and creating a nightmare for a lot of us trying to upgrade. I know, I know, you can turn off the asset pipeline and it got better since it was first released. But shouldn’t that be the other way around? Shouldn’t fun new ideas risking the stability of an app or making migration harder, be off by default and turned on only by people wanting to experiment? When your framework is young, it’s normal that you move fast and sometimes break, but once it matures, these things shouldn’t happen.

 

Public/private/plugin APIs

This is more of a recommendation than anything else. When you write a framework in a very dynamic language like Ruby, people will “monkey patch” your code to inject features. Sometimes it is due to software design challenges, sometimes it’s because people don’t know better. However,  by not explicitly specifying what APIs are private (they can change at anytime, don’t touch), what APIs are public (stable, will be slowly deprecated when they need to be changed) and which ones are for plugin devs only (APIs meant for instrumentation, extension etc..), you are making migration to newer versions much harder. You see, if you have a small, clean public API, then it’s easy to see what could break, warn developers and avoid migration nightmares. However, you need to start doing that early on in your project, otherwise you will end up like Rails where all code can potentially change anytime.

 

Rails/Merb merge was a mistake

This is my personal opinion and well, feel free to disagree, nobody will ever be able to know to for sure. Without explaining what happened behind closed doors and the various personal motivations, looking at the end result, I agree with the group of people thinking that the merge didn’t turn up to be a good thing. For me, Rails 3 isn’t significantly better than Rails 2 and it took forever to be released. You still can’t really run a mini Rails stack like promised. I did hear that Strobe (company who was hiring Carl Lerche, Yehuda Katz and contracted Jose Valim) used to have an ActionPack based, mini stack but it was never released and apparently only Rails core members really knew what was going on there. Performance in vanilla Rails 3 are only now getting close to what you had with Rails 2 (and therefore far from the perf you were getting with Merb). Thread-safety is still OFF by default meaning that by default your app uses a giant lock only allowing a process to handle 1 request at a time. For me, the flexibility and performance focus of Merb were mainly lost in the merge with Rails. (Granted, some important things such as ActiveModel, cleaner internals and others have made their way into Rails 3)

But what’s worse than everything listed so far is that the lack of competition and the internal rewrites made Rails lose its headstart.  Rails is very much HTML/view focused, its primarily strength is to make server side views trivial and it does an amazing job at that. But let’s be honest, that’s not the future for web dev. The future is more and more logic pushed to run on the client side (in JS) and the server side being used as an API serving data for the view layer. I’m sorry but adding support for CoffeeScript doesn’t really do much to making Rails evolve ahead of what it currently is. Don’t get me wrong, I’m a big fan of CoffeeScript, that said I still find that Rails is far from being optimized to developer web APIs in Rails. You can certainly do it, but you are basically using a tool that wasn’t designed to write APIs and you pay the overhead for that. If there is one thing I wish Rails will get better at is to make writing pure web APIs better (thankfully there is Sinatra). But at the end of the day, I think that two projects with different philosophies and different approaches are really hard to merge, especially in the open source world. I wouldn’t go as far as saying like others that Rails lost its sexiness to node.js because of the wasted time, but I do think that things would have been better for all if that didn’t happen. However, I also have to admit that I’m not sure how much of a big deal that is. I prefer to leave the past behind, learn from my own mistake and move on.

 

Technical debts

Here I’d like to stop to give a huge props to Aaron “@tenderlove” Patterson, the man who’s actively working to reduce the technical debts in the Rails code base. This is a really hard job and definitely not a very glamorous one. He’s been working on various parts of Rails including its router and its ORM (ActiveRecord). Technical debts are unfortunately normal in most project, but sometimes they are overwhelming to the point that nobody dares touching the code base to clean it up. This is a hard problem, especially when projects move fast like Rails did. But looking back, I think that you want to start tackling technical debts on the side as you move on so you avoid getting to the point that you need a hero to come up and clean the piled errors made in the past. But don’t pause your entire project to clean things up otherwise you will lose market, momentum and excitement. I feel that this is also very much true for any legacy project you might pick up as a developer.

 

Keep the cost of entry level low

Getting started with Rails used to be easier. This can obviously argued since it’s very subjective, but from my perspective I think we forgot where we come from and we involuntary expect new comers to come with unrealistic knowledge. Sure, Rails does much more than it used to do, but it’s also much harder to get started. I’m not going to argue how harder  it is now or why we got there. Let’s just keep in mind that it is a critical thing that should always be re-evaluated. Sure, it’s harder when you have an open source project, but it’s also up to the leadership to show that they care and to encourage and mentor volunteers to  focus on this important part of a project.

 

Documentation

Rails documentation isn’t bad, but it’s far from being great. Documentation certainly isn’t one of the Ruby’s community strength, especially compared with the Python community, but what saddens me is to see the state of the official documentation which, should, in theory be the reference. Note that the Rails guides are usually well written and provide value, but they too often seem too light and not useful when you try to do something not totally basic (for instance use an ActiveModel compliant object). That’s probably why most people don’t refer to them or don’t spend too much time there. I’m not trying to blame anyone there. I think that the people who contributed theses guides did an amazing job, but if you want to build a strong and easy to access community, great documentation is key. Look at the Django documentation as a good example. That said, I also need to acknowledge the amazing job done by many community members such as Ryan Bates and Michael Hartl consistently providing high value external documentation via the railscasts and the intro to Rails tutorial available for free.

 

In conclusion, I think that there is a lot to learn from Rails, lots of great things as well as lots of things you would want to avoid. We can certainly argue on Hacker News or via comments about whether or not I’m right about Rails failures, my point will still be that the mentioned issues should be avoided in any projects, Rails here is just an example. Many of these issues are currently being addressed by the Rails team but wouldn’t it be great if new projects learn from older ones and avoid making the same mistakes? So what other mistakes do you think I forgot to mention and that one should be very careful of avoiding?

 

Updates:

  1. Rails 4 had an API centric app generator but it was quickly reverted and will live as gem until it’s mature enough.
  2. Rails 4 improved the ActiveModel API to be simpler to get started with. See this blog post for more info.

, , , , , ,

36 Comments

Books to read in 2012 – recommended to me by Twitter

Today, I asked on Twitter what non-technical books I should read in 2012.

I was nicely surprised to see so many of my followers send recommendations. Here is a list of 25 books that like-minded people suggested I read. Hopefully you will find a book or two to read too. Feel free to send more recommendations via the comments.

 

1Q84 by Haruki Murakami suggested by @mrb_bk and @chadfowler
The Floating Opera and The End of the Road by John Barth suggested by @chadfowler
Into Thin Air by Jon Krakauer suggested by @bradly
Cutting for Stone by Abraham Verghese suggested by @bradly
Atlas Shrugged by Ayn Rand suggested by @bradly
Cien años de soledad by Gabriel Garcia Marquez (es) suggested by @romanandreg & @jrfernandez & @edgarschmidt
One Hundred Years of Solitude by Gabriel García Marquez suggested by @romanandreg & @jrfernandez & @edgarschmidt
Jitterbug Perfume by Tom Robbins suggested by @supaspoida
The Sisters Brothers by Patrick deWitt suggested by @dennismajor1
The Glass Bead Game by Hermann Hesse suggested by @dj2sincl
The Wind-Up Bird Chronicle by Haruki Murakami suggested by @chadfowler
Mindfire by Scott Berkun suggested by @lucasdicioccio
Les Fourmis by Bernard Werber (fr) suggested by @twitty_tim
Perfume: The Story of a Murderer by Patrick Suskind suggested by @twitty_tim
Les Miserables by Victor Hugo (en, free ebook) suggested by @tutec
Song Of Ice and Fire by George R.R. Martin (Game of Thrones saga) suggested by @eeppa & @jarin
Clockwork Century by Cherie Priest suggested by @eeppa
The Darkness that Comes Before by R. Scott Bakker suggested by @eeppa
Drood by Dan Simmons suggested by @eeppa
This Is Water by David Foster Wallace suggested by @atduskgreg
Anathem by Neal Stephenson suggested by @jarin
Ender’s Game by Orson Scott Card (entire saga) suggested by @jarin & @edgarschmidt
Snow Crash by Neal Stephenson suggested by @jarin
Fixing the Game by Roger L. Martin suggested by @jarkko
The Road by Cormac McCarthy suggested by @mrreynolds

7 Comments

Developing a Curriculum

Recently I asked a friend of mine to give me pointers on how to develop a curriculum (he used to teach an education PHD program), after discussing his response on Twitter, people asked me to put it somewhere, so here it is:

Process to develop a curriculum:

Purpose. Know why you’re doing what you’re doing.

  • You know how to do this.

Product. Start with the end in mind.

  • What does the student look like when they walk out the door at the end of the training.
  • Usually, we break these down into Knowledge, Skills, or Attitudes.
  • Sometimes it’s helpful to see a photograph or drawing of a someone who finished the program and just talk about what they can do that makes them successful.
  • This “product” should be connected and help you accomplish your mission

Practices. Then ask yourself, “How do people become like this?”

  • If you can break down your Product into 3-5 bit-sized chunks, then see how people learn each one of those skills, gain each one of those knowledge points, and how to they gain the attitudes you want them to have.
  • This one is much easier the more experience you have in seeing people develop the “Product.”
  • This is also easier to determine when you understand Learning Theory.
  • The results from this section will result in a list of:
    •        Activities or experiences
    •        Resources. What books, website, teachers, software, etc. will help them learn more effectively and efficiently
    •        Assessments. How you would know if the activity was helpful?

Plans. Make your plans based on the practices you’ve determined you’ve needed.

 

On a related topic, Chad Fowler posted an interesting blog post about what LivingSocial is doing to change the software development education.

1 Comment

Data safety and GIL removal

After my recent RubyConf talk and follow up post addressing the Ruby & Python’s Global Interpreter Lock (aka GVL/Global VM Lock). a lot of people asked me to explain what I meant by “data safety”. While my point isn’t to defend one approach or the other, I spent a lot of time explaining why C Ruby and C Python use a GIL and where it matters and where it matters less. As a reminder and as mentioned by Matz himself, the main reason why C Ruby still has a GIL is data safety. But if this point isn’t clear to you, you might be missing the main argument supporting the use of a GIL.

Showing obvious concrete examples of data corruption due to unsafe threaded code isn’t actually as easy at it sounds. First of all, even with a GIL, developers can write unsafe threaded code. So we need to focus only on the safety problems raised by removing the GIL. To demonstrate what I mean, I will try to create some race conditions and show you the unexpected results you might get. Again, before you go crazy on the comments, remember that threaded code is indeterministic and the code below might potentially work on your machine and that’s exactly why it is hard to demonstrate. Race conditions depend on many things, but in this case I will focus on race conditions affecting basic data structures since it might be the most surprising.

Example:

@array, threads = [], []
4.times do
  threads << Thread.new { (1..100_000).each {|n| @array << n} }
end
threads.each{|t| t.join }
puts @array.size

In the above example, I’m creating an instance variable of Array type and I start 4 threads. Each of these threads adds 100,000 items to the array. We then wait for all the threads to be done and check the size of the array.

If you run this code in C Ruby the end result will be as expected:

400000

Now if you switch to JRuby you might be surprised by the output. If you are lucky you will see the following:

ConcurrencyError: Detected invalid array contents due to unsynchronized modifications with concurrent users
        << at org/jruby/RubyArray.java:1147
  __file__ at demo.rb:3
      each at org/jruby/RubyRange.java:407
  __file__ at demo.rb:3
      call at org/jruby/RubyProc.java:274
      call at org/jruby/RubyProc.java:233

This is actually a good thing. JRuby detects that you are unsafely modifying an instance variable across threads and that data corruption will occur. However, the exception doesn’t always get raised and you will potentially see results such as:

335467
342397
341080

This is a sign that the data was corrupted but that JRuby didn’t catch the unsynchronized modification. On the other hand MacRuby and Rubinius 2 (dev) won’t raise any exceptions and will just corrupt the data, outputting something like:

294278
285755
280704
279865

In other words, if not manually synchronized, shared data can easily be corrupted. You might have two threads modifying the value of the same variable and one of the two threads will step on top of the other leaving you with a race condition. You only need 2 threads accessing the same instance variable at the same time to get a race condition. My example uses more threads and more mutations to make the problem more obvious. Note that TDD wouldn’t catch such an issue and even extensive testing will provide very little guarantee that your code is thread safe.

 

So what? Thread safety isn’t a new problem.

That’s absolutely correct, ask any decent Java developer out there, he/she will tell how locks are used to “easily” synchronize objects to make your code thread safe. They might also mention the deadlocks and other issues related to that, but that’s a different story. One might also argue that when you write web apps, there is very little shared data and the chances of corrupting data across concurrent requests is very small since most of the data is kept in a shared data store outside of the process.

All these arguments are absolutely valid, the challenge is that you have a large community and a large amount of code out there that expects a certain behavior. And removing the GIL does change this behavior. It might not be a big deal for you because you know how to deal with thread safety, but it might be a big deal for others and C Ruby is by far the most used Ruby implementation. It’s basically like saying that automatic cars shouldn’t be made and sold, and everybody has to switch to stick shifts. They have better gas mileage, I personally enjoy driving then and they are cheaper to build. Removing the GIL is a bit like that. There is a cost associated with this decision and while this cost isn’t insane, the people in charge prefer to not pay it.

 

Screw that, I’ll switch to Node.js

I heard a lot of people telling me they were looking into using Node.js because it has a better design and no GIL. While I like Node.js and if I were to implement a chat room or an app keeping connections for a long time, I would certainly compare it closely to EventMachine, I also think that this argument related to the GIL is absurd. First, you have other Ruby implementations which don’t have a GIL and are really stable (i.e: JRuby) but then Node basically works the same as Ruby with a GIL. Yes, Node is evented and single threaded but when you think about it, it behaves the same as Ruby 1.9 with its GIL. Many requests come in and they are handled one after the other and because IO requests are non-blocking, multiple requests can be processed concurrently but not in parallel. Well folks, that’s exactly how C Ruby works too, and unlike popular believe, most if not all the popular libraries making IO requests are non blocking (when using 1.9). So, next time you try to justify you wanting to toy with Node, please don’t use the GIL argument.

 

What should I do?

As always, evaluate your needs and see what makes sense for your project. Start by making sure you are using Ruby 1.9 and your code makes good use of threading. Then look at your app and how it behaves, is it CPU-bound or IO-bound. Most web apps out there are IO-bound (waiting for the DB, redis or API calls), and when doing an IO call, Ruby’s GIL is released allowing another thread to do its work. In that case, not having a GIL in your Ruby implementation won’t help you. However, if your app is CPU-bound, then switching to JRuby or Rubinius might be beneficial. However, don’t assume anything until you proved it and remember that making such a change will more than likely require some architectural redesign, especially if using JRuby.  But, hey, it might totally be worth it as many proved it in the past.

 

I hope I was able to clarify things a bit further. If you wish to dig further, I would highly recommend you read the many discussions the Python community had in the last few years.

 

 

 

 

, , , ,

3 Comments

About management

I decided to save myself a session to the shrink and instead just write down my reflection on management. Who knows, some of you might help me and/or challenge my thought process.

I recently read a great management book called the five dysfunctions of a team by Patrick Lencioni . Instead of telling you what to do, the author highlights behavior patterns that are related to each other and when aggregated result in dysfunctional teams. I really liked the book because instead of a being a cookbook/playbook, this is more a fail book, in other words, it illustrates what you don’t want to do and explains why. It highlights very well the relation between various behaviors and nicely illustrates why teams of brilliant people can fail. The Kindle version is at less than $5, go get it and read it on your iPhone/iPad/computer/browser…

So this book somewhat changed my perception of management and leadership. Interesting enough, at Sony, my previous employer, they make a distinction between management and leadership. While they hope managers can be leaders, they don’t require them to be and to be honest very few are. I’m not sure that’s a good or a bad things, but I, for sure, was under different expectations. Finally, I spent a large amount of my life on the internet working on/with projects where meritocracy, respect and honor were key. The “ranking” is purely based on what your peers think of you and not based on your age/sex/origin/diploma/bank account. I do realize that this model has many pros but also some pretty major cons. My only point is that it did affect my worldview. In my world, seniority, a killer  job title or a fancy suit won’t buy you my automatic respect. On the other hand, job well done, great vision, honesty, over achievement will!

Taking these few trains of thoughts in consideration, I started thinking about my own expectations for a good manager/leader. I figured that if I were able to do that, I could possibly be able to define a work environment where I could thrive and maybe one day become a good “manager/leader”.

I’ve always questioned my ability to be a good leader. While most of the time, I have an opinion and can easily decide what I think should be done, I have a hard time relating to people who can’t see the “big picture”. While I usually can get decent results, I’m aware that it can unfortunately sometime be at the cost of a few bruised egos. I also know I have high expectations for myself and for others and I have a hard time understanding how some people can be ok with the “status-quo”. I’m a perfectionist who is only happy when he outperforms his previous achievement. I was raised to challenge and always push myself further, focusing on concrete end-results and achieved goals. And to be honest, that’s what I enjoy. But I also know for a fact, that many people are not like that and I can’t blame them for looking at things from a different angle and not sharing the same motivations. Furthermore, I know that most people actually don’t have the same driven temperament and that’s why I’ve questioned my abilities to lead others.

However, different temperaments can work together as long as there is respect. And by respect, I mean that everyone feel that they were being heard and know that their input was considered and addressed even though the outcome might not be as hoped for. But for respect to happen, you first need trust. And when people trust each other, Lencioni explains that “people don’t hold back one with another. They are unafraid to air their dirty laundry. They admit their mistakes, their weaknesses, and their concerns without fear of reprisal”. I think that as simple as it seems, it is the key to a successful team. A good leader should be able to create such an atmosphere where people can trust each other. In fact, I think that if a manger/leader/executive can manage to build trust as defined earlier, his technical skills or lack of vision don’t matter as much. He/she will be able to rely on people he trusts to help him make the right decisions. Of course, there is much more than to be a good leader, but I think that with this base, great things can be built, and without it, a much greater effort is required to get some good results.

Based on my findings, I think that I need to work on my communication so others don’t feel that they have to hold back and make sure everyone feels that their opinions were considered and addressed. To do that a key element is to admit my mistakes and weaknesses and asking others to help me improve. That’s it, sorry for the boring, not technical post. I promise the next one will have at least a code sample.

2 Comments

How to – cross domain ajax to a Ruby app

In some cases, you might have a bunch of apps running on different domains/subdomains and/or ports and you would like to make ajax requests between these services. The problem is that browsers wouldn’t let you make such requests because of the Same Origin Policy which only allowed them to make request to resources within the same domain.

However, most browsers (IE 8+, Firefox 3.5+, Safari 4+, Chrome) implement a simple way to allow cross domain requests as defined in this w3C document.

Of course, if your users have an old version of their browser, you  might have to look into jsonp or something else such as cheating by using iframes & setting document.domain. Let’s pretend for a minute that 100% of your users are on Chrome. The only thing you need to do is set a response header listing the accepted domains or “*” for all. A simple Rack middleware to do that would look like that.

 

class XOriginEnabler
  ORIGIN_HEADER = "Access-Control-Allow-Origin"
 
  def initialize(app, accepted_domain="*")
    @app = app
    @accepted_domain = accepted_domain
  end
 
  def call(env)
    status, header, body = @app.call(env)
    header[ORIGIN_HEADER] = @accepted_domain
    [status, header, body]
  end
end

And to use the middleware you would need to set it for use:

use XOriginEnabler

To enable all requests from whatever origin, or pass the white listed domain(s) as shown below.

use XOriginEnabler, "demo.mysite.com demo.mysite.fr demo.techcrunch.com"

For a full featured middleware, see this project.

6 Comments