Comments on braindump: A compact and cross language serialization library...

Müzik Dinle

2009-05-20T08:55:04.591+01:00

Michael: Thanks for the info, I drew the line fair...

2009-02-27T11:12:00.000Z

Michael: Thanks for the info, I drew the line fairly arbitrarily based on my own personal favorites :) I'll update the table when I get a minute.
Can't get the extprot link to work right now, will check later.

Techmilind: I do mention the raw comparators in passing, there's even a link. I certainly appreciate the speedup they give.

I'm also sure that implementing Record IO in new languages would be possible, but there doesn't seem to be a community or momentum to do so right now (correct me if I'm wrong). Perhaps it should be broken out into a separate project? That would give it more attention and a chance to build a community.

One more advantage of Record IO that should be men...

2009-02-26T18:39:00.000Z

One more advantage of Record IO that should be mentioned here is that the generated classes contain a static raw comparator method that is used by Hadoop Map-Reduce while sorting/shuffling map outputs and reduce inputs. Thus, for simply comparing keys, the records do not have to be instantiated and deserialized, the serialized versions can be compared directly. This speeds up the shuffle, which is the most time-consuming stage of many map-reduce jobs.

For debugging ease, record I/O supports different serialization methods (in addition to compact binary), such as CSV, and XML.

Supporting additional languages is simple too, since the code generators are modularly supported from the serialization mechanisms.

(But then I may be biased, since I was the primary author of Record IO.)

You put it in quotes, so I can't fault you too muc...

2009-02-26T03:52:00.000Z

You put it in quotes, so I can't fault you too much, but another important issue with these libraries is support for C and C#, which combined are over 20% on the TIOBE index. C# support is great in both Thrift and (as far as I can tell) Protocol Buffers. No for Record IO. C support, on the other hand, is weak. There have been submissions to the Thrift project, but none with complete implementations. The C library for protobuf uses its own custom compiler and is pretty new as well.

As far as additional libraries, http://eigenclass.org/R2/writings/extprot-extensible-protocols-intro looks interesting... certainly more interesting than Etch.

You could have mentioned typed bytes as well. The ...

2009-02-25T11:24:00.000Z

You could have mentioned typed bytes as well. The simplicity of the typed bytes format might eventually lead to very good language support (although, afaik, there currently only are Java and Python implementations), but it does (intentionally) sacrifice some compactness in order to reach this level of simplicity.