I gave a talk today at the #prorubyconf on Ruby Deployment, past present and future. You can view my slides below.

We will start doing some private beta testing of our new EY platform in December so let me know if you want to beta test and havce some interesting apps to run in the cloud.

I gave a talk today at the #prorubyconf on Ruby Deployment, past present and future. You can view my slides below.

We will start doing some private beta testing of our new EY platform in December so let me know if you want to beta test and havce some interesting apps to run in the cloud.

Umm, w00t! It’s been a little over two years since merb was a twinkle in my eye, and a pastie. Since then it has undergone many drastic transformations, working its way towards a very solid, fast foundation for people to build their homesteads on.

It’s very exciting for me to see it grow into what it wants to be, based on the awesome merb community that has popped up around the project. It just shows that if you put in the work on something you love, polishing every little corner it will attract other like minded individuals and they will help you take the project to levels higher then your wildest dreams.

So, gem install merb and get the 1.0 final now, it is propogating its way through the rubyforge tubes. If you can’t get it from the ‘forge yet then you can get it from our gem server:

gem install merb—source http://edge.merbivore.com

Thank you to everyone who helped make this happen with code, feedback and blood, sweat and tears. It’s been totally worth the effort IMHO.

The merb is dead, long live the merb.

Umm, w00t! It’s been a little over two years since merb was a twinkle in my eye, and a pastie. Since then it has undergone many drastic transformations, working its way towards a very solid, fast foundation for people to build their homesteads on.

It’s very exciting for me to see it grow into what it wants to be, based on the awesome merb community that has popped up around the project. It just shows that if you put in the work on something you love, polishing every little corner it will attract other like minded individuals and they will help you take the project to levels higher then your wildest dreams.

So, gem install merb and get the 1.0 final now, it is propogating its way through the rubyforge tubes. If you can’t get it from the ‘forge yet then you can get it from our gem server:

gem install merb—source http://edge.merbivore.com

Thank you to everyone who helped make this happen with code, feedback and blood, sweat and tears. It’s been totally worth the effort IMHO.

The merb is dead, long live the merb.

Just a quick post to share some news. I just finished giving my keynote at MerbCamp about past and present merb and the core tenets of merb development.

I also introduced Nanite: self assembling cluster of ruby processes

You can watch the full video of my talk here: Keynote Video

WMV format ^^

And you can see my slides on slideshare below.

Thanks to all the merbcamp organizers, the venue is awesome and I have never spoken at a ruby conference that streamed all of the talks live. Very cool setup here.

I will follow up with some more posts on nanite and the what why and how of it shortly. But you can get a pretty good overview of it by watching my talk.

We’ve got a few job openings I wanted to float out there for your consideration. Here are the listings:

Ruby Developer

Systems Engineer in Sacremento

Systems Engineer

We need another team member for the Vertebra project. You know ruby like the back of your hand and are interested in distributed systems as well as learning erlang.

Systems engineers work on our cluster systems and will help with our custom Gentoo linux build as well as be involved with the Load Balancers, Switching Fabric, Linux, Xen, Coraid and helping design the latest and greatest platform for running Ruby applications in the cloud.

If you think you have the chops for any of these jobs then we want to hear from you. Please send a resume and a short email describing who you are and why you kick ass to jarnold@engineyard.com and cc ezra@engineyard.com

I gave a talk on Vertebra today at Pivotal Labs. This time it was a bit more technical then my last talk at railsconf as the system has evolved since then. We’re working furiously on this thing and hope to have the first release very soon.

We will be opening up an early beta for a few folks so if you have some wicked cool idea you want to use Vertebra for get in touch and a few people can get added to the early access.

Video was taken today of the talk and should get posted online within a few weeks.

Slides are on slideshare here if you cannot view the embedded version below.

Valery Kholodkov has written a very cool nginx module for handling uploads.

The way this works is that you specify a location block to handle the uploads. So if you are using the standard nginx.conf for rails apps then you would add this in your server block right above your “location /” block:

Then make a simple upload form that does a multipart POST to /upload. Now you can have your Rails or Merb app on the backend with a route called /upload. In the action of your app that responds to the /upload route you will get a set of params that look like this(assume the name of your upload fields is called ‘file1’ and ‘file2’):

What this is doing if parsing the multi-part mime boundaries in C in the nginx plugin, putting the parsed files into files in /tmp and then stipping the multipart stuff out of the POST body and replacing it with the info you need to get the name and location of the file on disk.

This means that by the time the request hits your application, the expensive mime parsing is already done and you simply move the file to it’s final resting place. This is a huge win since now the hard work is done in C in nginx before your app ever gets involved.

Of course this is a fresh new module so do your own testing and deciding whether or not this is a fit for your needs. But I think this is a great plugin and have verified it works as advertised.

Man it seems like yesterday that Engine Yard was a small 3 person startup with big ideas and little cash. People seemed to like what we were offering and the business grew into a thriving startup.

In the beginning we only had plans to be the best fully managed rails/ruby hosting company. And I think we met this goal early on. But it became apparent that we had a strong brand and we started kicking around ideas of becoming an open source software company as well.

We believe in Ruby as a platform, and we’ve put our money where our mouth is when we sponsored Rubinius and Merb, hiring many developers to work on both projects.

Engine Yard has become much more then just a managed rails hosting company, we want to strengthen the Ruby ecosystem for everyone by providing the infrastructure and open source software for the next wave of Ruby deployments in the cloud.

We’ve also been delving into the cloud computing arena as I think the next 5 years are going to see huge transition from standard hosting models into the cloud. Our upcoming Vertebra project is a new application programming platform for building distributed cloud applications with XMPP. You can expect to see the first open source release of Vertebra this summer, I think this is a truly unique and very fun project to work on. I think a lot of folks out there will have tons of different use cases for it.

So fast forward to right now, Engine Yard now has more then 80 employees worldwide. Half of these people are Application Support and SysAdmins working to support all the awesome applications we host. Providing 24/7 support. But we also have a growing Engineering team working on all kinds of exciting stuff. I consider myself lucky to work with such a talented team.

So with all that being said, I’m excited to annouce that we have just closed our Series B round of VC funding totalling $15million dollars(insert austin powers joke here)! This round includes investment from NEA, Amazon and Benchmark.

We’re going to use this money towards making Ruby the platform of choice for cloud computing and web development in startups and the enterprise alike.

Watch this space, we have lots of exciting announcements in the coming months. I’d like to thank all of our customers and users of our open source software and the Ruby community in general. It has been a wild ride these last 2 years and I expect many more exciting things to come!

Chris Matthieu and Steven Bristol interviewed me for the Rubyology Podcast a few days ago and they have posted the interview here. I talk about my early days of computing, merb, vertebra, rubinius and the story behind how Engine Yard got started and has grown. I’ll warn you though it’s a long interview, 120 minutes! Have a listen if you want to hear the story.

I’m going to be speaking at Oreilly’s Velocity conference next week on Monday June 23rd. Velocity is a conference about Web Scale operations and performance, lot’s of stuff about infrastructure and cloud computing.

I’ll be giving a short talk on Vertebra at 4:45pm on Monday and I will also be on a panel called “Getting into the Clouds” with some guys from Amazon, 3terra, Rackspace and joyent at 3:45pm. Should be fun.

Come see it if you’re in the area or already coming to the conference.

Velocity Conference

I’d like to point folks to our new Engine Yard Express VMWare image. This is a replica of an EY slice that you can download and use locally whether you are an EY customer or not. When you boot it up it will autotune mysql to use the proper amount of ram, boot up a merb and a rails app along with nginx and monit et all. It will give you a user/pass and show you the IP address as part of the boot process. So just boot it up and peek at the IP address on port 80 and port 81, the index pages of the rails and the merb app have detailed instruction on how to get a deploy.rb and deploy your own code.

Enjoy!

Here are the slides from my talk at RailsConf 2008. I’m too tired to write much more right this instant so I’ll let the slides talk. Rest assured you will here more about vertebra in the near future…

There is a great tutorial/screencast about a new merb feature that I really like a lot called merb-slices. This is now part of merb-more, contributed by a long time friend of mine, Fabien Franzen(worked on ez-where with me)

merb-slices are “Little slices of MVC cake”. These are self contained merb apps with models, controlers, views and assets that you can distribute as rubygems. You can mount a merb-slice at a specific point in your router definition and you can override any part of the slice up in your main app. So in a way these are similar to what Rails-Engines promise, except merb-slices are built into the framework and will not break when merb itself is updated.

Check out the tutorial/screencast for a peek at how merb-slices work.

It seems like a lot of folks out there want there to be a battle royal between Merb and Rails. I just wanted to dispel this myth.

Merb was written because it scratched an itch that I and many of my hosting customers had, mainly memory footprint, speed, concurrent requests and simple maintainable code. This does not mean that Merb has to be thought of as a Rails competitor. Merb stands on its own merits just as Rails does.

I feel that both of these frameworks have great strengths and weaknesses and that they compliment each other in many ways. Face it Merb would never be around if Rails had not come first. And I see Merb as a great experimentation playground for exploring web framework functionality while learning a great many things from Rails and building a fresh view based on Rails strengths and weaknesses.

There is no reason that a lot of the goodness in Merb cannot be applied back to Rails, but it is no small feat to trim down rails size without breaking backwards compatibility.

I view it like this, more choices make the Ruby ecosystem a better place. So let’s just stop with the Rails VS Merb stuff. How about people choose what framework they want to use based on the frameworks merits and features rather then religious arguments about how my framework can beat up your framework.

The Ruby community in general has been exceedingly nice as long as I have been a part of it. I wrote Merb for a reason, if Merb’s philosophies agree with your tastes then great, use Merb. If Rails is your thing then that’s great too! We do not need a monoculture.

Rails has been Ruby’s killer app and has brought Ruby to the masses. But Ruby is growing up, there are many alternate Ruby implementations now and they all work together for the common good of Ruby.

I’d like the same thing to happen with Rails, Merb and all the other ruby web frameworks. Let’s all work together to make the Ruby ecosystem stronger. We do not need to fight amongst ourselves when there are plenty of other Programming language communities to fight with :P

I guess what I am saying is that I see the proliferation of ruby web frameworks as a sign that Ruby itself has “arrived” and is here to stay. So can’t we all just get along in Ruby web framework land?

My Deploying Rails Applications Book has shipped in print form! This book has been two years in the making and was rewritten a few times as the Rails deployment landscape shifted. I’m so glad it is finally in print \m/ Big thanks to all my co-authors and beta readers who gave valuable early feedback.

And in other amazingly beautiful news, Rails runs on Rubinius as of last night!. A huge round of applause for Evan, Wilson, Brian, Ryan, Eric and Eero, the rubinius core team and also to all 150 rubinius contributors. This is a huge milestone for the project. Being able to run rails, which is one of the largest and most complex ruby programs out there is huge. This means that rubinius is now ruby compatible for the most part. Now the team can start to steer their focus on speed improvements like JIT , polymorphic inline caching, LLVM and various other techniques for making rbx smoking fast.

Michael Klishin made a sweet mapping of a merb server’s boot process. Hopefully we can get stuff like this made for other parts of the framework as well as I think it can really help you find your way around the source code of merb-core.

Merb 0.9.3 is mostly (but not only) a bugfix release. We’ve not only added several features very useful for developers who want to use Merb for web services, but fixed many bugs, greatly improved spec coverage and quality, made documentation better, and did some performance-related work.

diffstat shows 95 files changed, 3359 insertions(+), 837 deletions(-)

Engine Yard is looking for one or two kick ass Erlang programmers to help work on our next generation cloud computing platform. Knowledge of ruby/xmpp/ejabberd/linux is a plus.

If you think you fit the bill then please email me directly at ezra@engineyard.com.

So i’ve spent this week hacking on Rails, specifically going spelunking in ActionPack and porting Merb’s rack machinery to rails. I figure that merb is a very nice experimentation ground and decided it was time to give some love back to the framework that inspired merb.

While still not complete, I have made significant headway on racking up rails in my github fork of Rails. I’ve added rack adapters for mongrel, eventedmongrel, thin, ebb and webrick. All of this is controlled via ./script/rackup in a rails app. So to start a cluster of 5 thin servers you would run this command:

I do have to say that parts of ActionPack haven’t been touched in a long time and had accumulated some cruft, with the rails rack adapter that thin and ebb use there was a dogpile of wrappers going on. A web request was wrapped in many layers like this(the → means ‘wrapped in’)

That was far too many wrappers, it also turned out that some of these wrapper were making duplicates of all the CGI headers, meaning quit a bit of wasted memory on every request. I’ve slimmed it down to this now:

With no more duplication of CGI headers, win!

I’ve also changed the giant mutex lock to be much smaller, it used to lock around the dispatcher callbacks, route recognition, controller instantiation, filters and action dispatch. Now it only locks around filters and action dispatch, Dispatcher callbacks, route recognition and controller instantiation all happen outside of the lock. This makes for a nice little speed boost with standard mongrel under concurrent load.

Of course this doesn’t work so hot in development mode when you have concurrent requests, it falls apart due to route set reloading and class reloading done in dev mode. But if you are just using your browser to test the app in dev mode you won;t ever notice since you won;t be making concurrent requests. In production mode the route recognition appears to be thread safe, more extensive testing and stressing will be needed to be 100% certain though.

All in all I feel like these are some big wins for Rails. And I’m not done yet, I plan on beating up ActionPack quite a bit more until it submits ;)

There is a classic tradeoff between threaded servers and event driven servers. Event driven servers tend to be much faster than threaded servers when all the requests are fairly fast. But the event model falls down if you have long requests like file uploads or reporting actions. This is because the long action blocks the event loop, effectively keeping other requests from running.

There are two new event driven ruby webservers at your disposal these days, Thin and Ebb. Both of these servers support the Rack web server interface that merb uses. Until now both of these servers were not the best choice for file uploads or long blocking actions but that’s all changing.

Both ebb and thin have added a deferred?(env) method to their rack adapter interface. Both webservers will call this method on your Rack @app object before they call your call(env) method. This allows your rack adapter to determine if the request should be run in its own thread if it’s slow.

I’ve just committed support for this deferred_actions construct in merb-core. If you want to run on thin or ebb but you have say a file upload action and some slow reporting actions you could add this to your init.rb in your app:

What this means is that all of your actions will run in fast event driven mode except for requests to /uploads/create and /reports/longaction. Any request for either of these urls will be served from a newly spawned thread, all other requests will be served by the main event loop.

This allows us to have the best of both world. Combining threaded and event driven styles to use the strengths of both makes a lot of sense here.

Cheers to the authors of ebb and thin for making the same interface. All of this requires the HEAD versions of ebb or thin so if you want to play along at home you will need to build thin or ebb from source on github: ebb and thin

Jed Hurt and Lance Carlson made a nice wiki engine in merb that we’ve thrown up at http://wiki.merbivore.com. Thanks Guys!

You can get the source to the wiki app on github: Collective Wiki Engine

There are the starts of some good information published already, please feel free to add to it!

GitHub has officially launched! No more invites required to sign up.

In preparation for their launch we set them up on a shiny new Engine Yard cluster. With all the adoption of git/github as well as Rails itself about to switch to github we wanted to make sure everyones favorite git repo hosting can scale as far as folks want to take it.

So go Sign Up already and fork your favorite projects to your hearts desire!

Confreaks has done a great job of recording Mountain West Rubyconf and getting the video online quickly. You can go see my talk here:

Strengthening the Ruby Ecosystem: Merb

Here are my slides from my keynote at Mountain West Rubyconf this morning:

http://www.slideshare.net/ezmobius/merb-core/

We’ve released the 0.9.1 version of merb to the merbivore.com gem server. This release has a lot of polish and is getting very close to being stable api wise after the big 0.5.3 → 0.9.x refactoring.

You can see the big changelog here.

I’m pretty happy with how the codebase is shaping up, this was a major refactoring of merb and we’ve come out with a very clean system. Performance is improved quite a bit from older 0.5 versions of merb.

We’ve split the code base into multiple parts, merb-core, merb-more and merb-plugins. merb-core is the heart of the system, it has the rack abstraction along with the dispatcher, router, controller and view layers. You can make very fast, small footprint services and apps with just merb-core.

And merb-plugins consists of:

Wow that’s a lot of gems! Merb is built in a modular way that allows you to cherry pick features so you never have to load code you aren’t going to use. This helps keep the memory footprint down when building service style apps. But still allows for all the advanced features you want.

You see, Merb is built on rubygems and Merb plugins are just rubygems so plugins have the same standing as built-ins code loading wise. Merb is just begging you to peek under the hood to see how it ticks ;)

To make things easier to get started we still have a merb gem that will install all of merb-core and merb-more for you. You should uninstall all of your old merb gems before you install the new version.

Merb development has moved to GitHub for source control and Lighthouse for ticketing.

The best place to get merb questions answered is still #merb on irc.freenode.net. We will have a wiki and a google group up shortly.

Let us know what you think, kick the tires and all that. After it settles for a few days we will push it to rubyforge, hopefully by this weekend.

So after being in the works for almost 2 years now and after going through many rewrites as Rails deployment tech has changed, my book is finally done! I just have to thank all of my coauthors who put in tons of work to get this thing as polished as it is now. Especially Bruce Tate and Clinton Begin, thanks guys!

New in this release is the chapter on Apache and Nginx scaling as well as a performance chapter. ALso included is setting up your own Xen installation and setting up Mysql Master → Slave and Master → Master replicated databases.

If you were holding off on getting this book until the advanced content was in there then now is the time to snap it up. This is the final beta release and the book will be going to print fairly soon. I really like the way the book has come together and I think it is a wealth of knowledge about deploying Rails applications.

\m/ Yay! Writing a book was a lot more work then I thought it would be and I could not have done it without my wonderful co-authors and all of the great feedback from the early beta readers.

Thanks to everyone who provided feedback. You can grab your copy of the book here:

Deploying Rails Applications

Rack is an awesome webserver abstraction that distills the idea of a ruby web app down to an object or Proc that has a call method. The call method takes the rack environment, which is all of the cgi style headers for the current request, and returns an array of [status, headers, body]. The status is a number like 200 or 404, the headers is a hash of header key value pairs for the response and the body is the output from your application, The body must respond to each and yield lines or chunks of strings to be flushed down the socket to the client.

Here is the most basic example of a rack app using the rackup builder DSL for mounting and running rack apps:

This app will return “hello world!” to the client. Pretty simple eh?

Now that merb-core is all based on rack, there are some very interesting things you can do with this knowledge. In the config/ directory of a freshly generated merb-core app you will see a rack.rb file. By default the file just contains this:

This will run the main merb rack app class that handles the standard dispatching of requests through the whole merb framework. Now merb-core is small and fast, but what if you have some certain requests that don’t really need the router or controllers/views of a full merb stack.

Say we need to handle a ton of concurrent file uploads, and we don’t want to invoke the full merb stack just for these uploads. The solution is to use the config/rack.rb file in your merb app along with the Rack::Cascade middleware to mount a simple Proc object to handle the uploads instead of merb proper. Here is what a rack.rb file for this would look like:

Rack::Cascade works by trying to call each app you specified in order and actually use the results from the first one that does not return a 404 error. So what our new uploader Proc/app does is creates a Merb::Request object and checks if the request.path matches /images/upload. If it does match then you can use code in here to handle the file upload and place it wherever you want. Once you’ve done whatever you need to the file upload you can redirect by setting the Location header like in the above example, or you can render out a page or return JSON or really anything you like.

Having the power of merb-core and also the power of raw rack access conveniently in the same app is very powerful. I think this is one very compelling thing merb-core has going for it now that we are all rack based.

Hopefully you enjoyed this installment of what is new and cool in merb. Hope to write some more articles shortly about special features in merb-core that go undernoticed.

So we are just getting started on mod_rubinius here at EY. We’ve hired Eero Saynatkari ( rue in the #rubinius irc channel) full time to work on the project.

The architecture for mod_rubinius is still up in the air at this point. We do know that it will be rack based so the interface from mod_rubinis into ruby apps will be via rack. Other then that we don’t yet know what the best way to architect the platform will be. It could be an embeded rubinius VM inside the apache processes. Or it could be a process manager that manages separate rubinius VM’s, or a combination of both of these approaches.

We would like to hear from you folks.. What would you like to see in a mod_rubinius? How is deployment of ruby apps painful for you and how would you like it to work in a perfect world?

We’d appreciate some feedback so please leave feature requests and ideas in the comments.

Want to work for the best and with the best? Are you interested in merb, rails, rubinius, and other ruby projects? Are you passionate about learning what you don’t know?

This position requires direct interaction with customers through the phone, ticketing system, irc, and email. You will be supporting some of the largest rails deployments on the internet. Can you handle it?

Desired: Experience with memcached, merb, monit, postgres, swiftiply.

Required: Solid ability to speak and write English. Good Attitude. Great work ethic. Experience with capistrano, mongrel, mysql, nginx, rails, subversion. Basic networking knowledge.

This position is right someone who isn’t afraid of hard work, loves facing new challenges each day, and wants to contribute to Engine Yard’s projects.

We offer comprehensive dental and medical insurance as well as the ability to work from home (or anywhere). Stock options are around the corner too.

Drop us a line at info@engineyard.com and tell us why you would be a great addition to our team. If you are the right person—we will send you an offer immediately. It’s a bonus if you are on the West Coast, maintain open source projects, or have a great community presence.