frsyuki
Joined 16 years ago
Karma
380
Submissions
52
Member since
May 11
Author of MessagePack http://msgpack.org
View on Hacker NewsRecent submissions
Comment · 11 years ago
ACK of network transfer is available ("require_ack_response" option). This option ends up choice of at-most-once semantics vs. at-least-once semantics. You need to choose and you can choose.
Fuentd provides "buffer_type file" to buffer records on disk. Shutting down won't loose data. If you need to choose memory buffer for performance reasons, fluentd enables "flush_at_shutdown" option by default.
You would also want to use <secondary> feature. This lets you to write a buffer chunk to another storage if the primary destination is not available "retry_limit" times.
Those concerns would be solved by the document: http://docs.fluentd.org/articles/out_forward#buffered-output...
Comment · 14 years ago
Did you guys see this video?: http://vimeo.com/53261709
Github uses MessagePack for their internal RPC protocol.
Comment · 14 years ago
Here is a benchmark result of msgpack, json, bson and protobuf: https://gist.github.com/4348013
An important point we should consider here is that actual performance depends on implementation.
Comment · 14 years ago
Disclaimer: I'm authoer of MessagePack for C++/Ruby and committer of one for Java.
As for strings, JSON has to allocate memory and copy to deserialize strings because strings are escaped.
MessagePack does't have to allocate/copy because the serialized format of strings is same as the format in memory. But it depends on the implementation whether actually it doesn't allocate/copy.
C++ and Ruby implementations try to suppress allocation and copying (zero-copy). But Java implementation doesn't support zero-copy feature so far (we have plan to do so. Here is "TODO" comment: https://github.com/msgpack/msgpack-java/blob/master/src/main...).
As for the other types, C++ implementation (and new Ruby implementation which is under development) has memory pool which optimizes those memory allocations petterns. But it's hard to implement such optimizations for Java because JVM (and Dalvik VM) doesn't allow to hook object allocation.
Comment · 14 years ago
We're using MessagePack in a Rails application to log user behaviors and analyze them.
Compared with other serialization libraries such as Protocol Buffers, Avro or BSON, one of the advantages of MessagePack is compatibility with JSON. (In spite of its name, BSON has special types which cause incompatibility with JSON)
It means we can exchange objects sent from browsers (in JSON format) between servers written in different languages without losing information.
I will not use MessagePack with browsers but it's still useful to use it with web applications.
Comment · 14 years ago
MessagePack includes a concept named "type conversion" to support types which are not supported by its wire format.
With the concept, we can serialize/deserialize user-defined classes as well as strings with encodings.
So far, MessagePack for Java, C++ and D implement the concept.
Comment · 14 years ago
Although the original blog post focuses on JavaScript and browsers, MessagePack itself doesn't mainly focus on them.
A major use case of MessagePack is to store serialized objects in memcached. A blog post written by Pinterest describes this use case (http://engineering.pinterest.com/posts/2012/memcache-games/). They use MessagePack with Python which is faster than one with JavaScript. They could store more objects in a server without performance declination (e.g. gzip).
It's true that MessagePack is not always faster than JSON (e.g. within browsers), and it's not always smaller than other serialization methods (e.g. with gzip compression). So we should consider that which serialization methods should I use for "my" case.
There are also general tendency which is helpful to select MessagePack or JSON:
MessagePack is faster to serialize binary data such as thumbnail images.
MessagePack is better to reduce overheads to exchange small objects between servers.
JSON is better to use it with browsers.Comment · 14 years ago
We're also using Fluentd as well as original JSON-based logging libraries.
Fluentd deals with JSON-based logs. JSON is good for human facing interface because it is human readable and GREPable.
On the other side, Fluentd handles logs in MessagePack format internally. Msgpack is a serialization format compatible with JSON and can be an efficient replacement of JSON.
I wrote plugin for Fluentd that send those structured logs to Librato Metrics (https://metrics.librato.com/) which provides charting and dashboard features.
With Fluentd, our logs became program-friendly as well as human-firnedly.