Andrew Cholakian’s Blog

ZeroMQ: What You Need to Know Braindump

In a recent email conversation, I was asked about getting started with ZeroMQ by another Rubyist (the awesome Ilya Grigorik)

This was my caffeine addled response, which others may find useful as well:

I'm going to warn you, some things you hear when you first read about ZMQ sound crazy. If there's one thing to be said for ZeroMQ, it's this sentence, from the Mongrel2 book "[ZeroMQ is] sockets the way programmers think sockets work". 

The more time I spend with ZeroMQ, the less I can think of a reason I'd ever have to open up a raw TCP or UDP socket, except in extraordinary circumstances, again. I think of ZMQ as common IPC and network communication patterns abstracted into messages and sockets that don't require a broker infrastructure. The whole message queue aspect of it is great, but ZMQ is really designed for a whole range of situations you'd never use AMQP for, and IMHO, that's the truly interesting thing about it. You don't really run ZMQ brokers (mostly), you communicate socket to socket using queue semantics.

The mongrel2 book has the most concise rundown of ZMQ, and IMHO best frames how to use it properly. It's not very in depth, maybe 15 minutes of reading, but is probably the single best introduction to ZeroMQ.

http://mongrel2.org/doc/tip/docs/manual/book.wiki#x1-590005.2

After that, I'd read the rbzmq rdocs for ZMQ::Socket. It's actually a fantastic rdoc, and a better intro to ZMQ than a lot of the docs on the official ZMQ site. It's kind of sad that someone wrote that fantastic doc and it doesn't get any publicity.

Another good post on ZMQ is this blog post:

The mongrel2 project would probably be the best example of a project that really is aggressively using ZMQ in an interesting way. You could check out the m2r ruby adapter to get an idea about ZMQ use in an actual app: http://github.com/perplexes/m2r . You may notice a lot of people using ffi-rzmq rather than the zmq gem. They're almost completely compatible at the API level, except ffi-rzmq works on non MRI/YARV rubies. I'm personally using it exclusively. 

The project I'm working on, dripdrop. is just a simple serialization format and reactor api on top of ZMQ, to make building an app out of a lot of ZMQ building blocks easy, and as much of the tedium out possible. It initially started because I wanted a simple async interface to events on multiple servers running Rails, but I think it made me realize ZMQ needs a proper super-simple API, preferably one that integrates non-ZMQ exit points, like HTTP,  Websockets, XMPP, etc.

DripDrop is still definitely still at the playing around with different ideas stage, and isn't really stable yet. I'm happy with the serialization format though (which is pretty much just BERT with a header for PUB/SUB filtering at the ZMQ level, similar in approach to mongrel2's format, but using BERT). I already have plans to redo about most of it, and need to start adding in support for socket types other than pub/sub. I really do like pub/sub though, it may not have the right semantics as far as making sure messages get delivered, but it enables semi-aspect-oriented architectures where random other processes can hook into a message stream (am I abusing the term aspect-oriented?)

What I plan on working on is redoing it style-wise into a simple DSL like this (this is a simple forwarder that uppercases and copies everything to a websocket and a ZMQ pub socket as an example):


I like the conciseness of this, I can see integrating ZMQ, HTTP, Websockets and other protocols in an async fashion, using this style of API, as worthwhile. What do you think? Biting off too much?

Anyway, I've clearly had too much coffee :)

App monitoring/messaging with ZeroMQ + Ruby = dripdrop

I just this week had an idea for realtime app stats/messaging using ZeroMQ. I wanted to be able to view events from my app in real time, and be able to archive them or do any random thing with them. Hence, dripdrop was born. It's a pretty cool way of doing collecting stats or performing async tasks. There's a full description of it on the GitHub project page at: http://github.com/andrewvc/dripdrop.

I'm mostly leveraging the awesome libraries that are zmq, zmqmachine, bert, and em-websocket.

I've diagrammed my use case for it below:

My New Job: Developer at Online Greetings Company Cocodot

So, it's been a month, and I still haven't put it up here, I'm now working as a Rails developer for online invitations and greetings company cocodot.

If you haven't see cocodot, it's a pretty interesting concept that I'm quite excited about. We occupy the the greetings space, but have better execution than the competition, we probably have the best designed greetings and cards of anyone out there, thanks to an amazing creative staff. The wedding invitations system is pretty slick as well, it's probably the best way to do online wedding invitations at the moment.

Sendgrid + Sendgrid Toolkit is pretty awesome

Sendgrid's a great way to send email from your app. They provide statistics, reliable delivery, and a fantastic XML/JSON API. If you're using Ruby, I recommend checkout out Sendgrid Toolkit, which I just found out about.

Sendgrid Toolkit is pretty much a thin wrapper using jnunemaker's HTTParty, which I haven't heard about till now. HTTParty's a pretty cool way to painlessly create an interface to a Web API.

Sendgrid Toolkit's use of HTTParty makes adding functionality very easy. I added bounce retrieve and delete functionality by merely adding the small number of lines seen in the code sample.

Kernel Async IO, not worth it?

I'm currently preparing my slides for my presentation about async IO at next week's Los Angeles Hacker News meetup, and in the process stumbled across this interesting thread on kernel  AIO on the libevent mailing list. This segment of a post from William Ahern stood out in particular. To put this in context, the question is whether to use the kernel's own AIO API, or just roll your own with your own threadpool using one blocking call per thread.

There are many ways to do AIO. I know of none that don't use threads, either kernel threads or user process threads. Now, the threads could be "light weight", but there's still a calling context that blocks on an operation. But no implementation is equivalent to, say, how the socket subsystem [state machines] work, where the only state that needs to be maintained is a queue of waiting objects (where object != thread). To put it simply, all the AIO implementations use too much memory and do too much book keeping (like thread scheduling) than strictly necessary, because kernels don't have a way to to "poll" on VM pages (as opposed to polling on disk interrupts, which you can accomplish when do doing direct I/O).

From what I know, Node.js manages its own thread pool, as does Ruby's event machine (using Ruby green threads). If anyone has any more info on this just tweet me, @andrewvc with what you know.

My Current Rails3 Gemfile

I've been toying around with Rails3 the past few weeks. My current Gemfile looks like this:

 gem 'rails', '3.0.0.beta4'
 gem 'capybara'
 gem 'database_cleaner'
 gem 'cucumber-rails'
 gem 'cucumber'
 gem 'rspec-rails', '2.0.0.beta.11'
 gem 'rspec', '2.0.0.beta.11'
 gem 'haml'
 gem 'spork'
 gem 'launchy'
 gem 'mongoid', "2.0.0.beta6"
 gem 'bson_ext'
 gem 'ruby-debug'
 gem 'machinist_mongo', :require => 'machinist/mongoid'
 gem 'faker'

MongoMapper's got some interesting stuff going on

I've been playing around a bit with MongoDB + Rails recently, and have been trying to figure out whether to use Mongo Mapper or Mongoid. I was pretty certain I wanted to use Mongoid last week, due to its excellent docs, but after reading John Nunemaker's post about Mongo Mapper 0.8 goodies I think I may be switching back over to MongoMapper (luckily I haven't done much yet).

One other thing MongoMapper has in its favor is mongrations. Yes, while MongoDB doesn't need migrations like SQL does, you DO still need to account for things when you say, need to convert all your distance measurements from miles to feet, or start adding indexes, or just want to delete a field in old records that's no longer needed.

Beautifully Illustrated Periodic Table of Perl Operators

This Periodic table of Perl 6 Operators is beautiful, though it is quite daunting..