MessagePack

It's like JSON.
but fast and small.

MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON. But it's faster and smaller. Small integers are encoded into a single byte, and typical short strings require only one extra byte in addition to the strings themselves.

Redis scripting has support for MessagePack because it is a fast and compact serialization format with a simple to implement specification. I liked it so much that I implemented a MessagePack C extension for Lua just to include it into Redis.

Salvatore Sanfilippo, creator of Redis

Fluentd uses MessagePack for all internal data representation. It's crazy fast because of zero-copy optimization of msgpack-ruby. Now MessagePack is an essential component of Fluentd to achieve high performance and flexibility at the same time.

Sadayuki Furuhashi, creator of Fluentd

Treasure Data built a multi-tenant database optimized for analytical queries using MessagePack. The schemaless database is growing by billions of records every month. We also use MessagePack as a glue between components. Actually we just wanted a fast replacement of JSON, and MessagePack is simply useful.

Kazuki Ohta, CTO

MessagePack has been simply invaluable to us. We use MessagePack + Memcache to cache many of our feeds on Pinterest. These feeds are compressed and very quick to unpack thanks to MessagePack while Memcache gives us fast atomic pushes.

Marty Weiner, Software Engineer

Also use MessagePack? Waiting for your testimonial!

API

 

List your implementation here!

msgpack/msgpack-java https://github.com/msgpack/msgpack-java

MessagePack for Java

QuickStart for msgpack-java is available here.

How to install

You can install msgpack via maven:

<dependencies>
  ...
  <dependency>
    <groupId>org.msgpack</groupId>
    <artifactId>msgpack</artifactId>
    <version>${msgpack.version}</version>
  </dependency>
  ...
</dependencies>
Simple Serialization/Deserialization/Duck Typing using Value
// Create serialize objects.
List<String> src = new ArrayList<String>();
src.add("msgpack");
src.add("kumofs");
src.add("viver");

MessagePack msgpack = new MessagePack();
// Serialize
byte[] raw = msgpack.write(src);

// Deserialize directly using a template
List<String> dst1 = msgpack.read(raw, Templates.tList(Templates.TString));
System.out.println(dst1.get(0));
System.out.println(dst1.get(1));
System.out.println(dst1.get(2));

// Or, Deserialze to Value then convert type.
Value dynamic = msgpack.read(raw);
List<String> dst2 = new Converter(dynamic)
    .read(Templates.tList(Templates.TString));
System.out.println(dst2.get(0));
System.out.println(dst2.get(1));
System.out.println(dst2.get(2));

msgpack/msgpack-d https://github.com/msgpack/msgpack-d

Build Status

MessagePack for D

MessagePack is a binary-based JSON-like serialization library.

MessagePack for D is a pure D implementation of MessagePack.

Features
  • Small size and High performance
  • Zero copy serialization / deserialization
  • Stream deserializer / Direct-conversion deserializer
  • Supports D features (Ranges, Tuples, real type)

Note: The real type is only supported in D. Don't use the real type when communicating with other programming languages. Note that Unpacker will raise an exception if a loss of precision occurs.

Current Limitations
  • No circular references support
Install

msgpack-d is implemented as a single module. Either copy src/msgpack.d to your project or use dub to add it as a dependency:

% dub install msgpack-d
Usage

Example code can be found in the example directory.

The documentation can be found here

pack / unpack

msgpack-d is very simple to use. Use pack for serialization, and unpack for deserialization:

import std.file;
import msgpack;

struct S { int x; float y; string z; }

void main()
{
    S input = S(10, 25.5, "message");

    // serialize data
    ubyte[] inData = pack(input);

    // write data to a file
    write("file.dat", inData);

    // read data from a file
    ubyte[] outData = cast(ubyte[])read("file.dat");

    // unserialize the data
    S target = outData.unpack!S();

    // verify data is the same
    assert(target.x == input.x);
    assert(target.y == input.y);
    assert(target.z == input.z);
}
Feature: Skip serialization/deserialization of a specific field.

Use the @nonPacked attribute:

struct User
{
    string name;
    @nonPacked int level;  // pack / unpack will ignore the 'level' field
}
Feature: Use your own serialization/deserialization routines for custom class and struct types.

msgpack-d provides the functions registerPackHandler / registerUnpackHandler to allow you to use custom routines during the serialization or deserialization of user-defined class and struct types. This feature is especially useful when serializing a derived class object when that object is statically typed as a base class object.

For example:

class Document { }
class XmlDocument : Document
{
    this() { }
    this(string name) { this.name = name; }
    string name;
}

void xmlPackHandler(ref Packer p, ref XmlDocument xml)
{
    p.pack(xml.name);
}

void xmlUnpackHandler(ref Unpacker u, ref XmlDocument xml)
{
    u.unpack(xml.name);
}

void main()
{
    /// Register the 'xmlPackHandler' and 'xmlUnpackHandler' routines for
    /// XmlDocument object instances.
    registerPackHandler!(XmlDocument, xmlPackHandler);
    registerUnpackHandler!(XmlDocument, xmlUnpackHandler);

    /// Now we can serialize/deserialize XmlDocument object instances via a
    /// base class reference.
    Document doc = new XmlDocument("test.xml");
    auto data = pack(doc);
    XmlDocument xml = unpack!XmlDocument(data);
    assert(xml.name == "test.xml");  // xml.name is "test.xml"
}
The PackerImpl / Unpacker / StreamingUnpacker types

These types are used by the pack and unpack functions.

See the documentation of PackerImpl, Unpacker and StreamingUnpacker for more details.

Links
Copyright
Copyright (c) 2010- Masahiro Nakagawa
License

Distributed under the Boost Software License, Version 1.0.

msgpack/msgpack-python https://github.com/msgpack/msgpack-python

MessagePack for Python
Author: INADA Naoki
Version: 0.4.1
Date: 2014-02-17
https://secure.travis-ci.org/msgpack/msgpack-python.png
What's this

MessagePack is a fast, compact binary serialization format, suitable for similar data to JSON. This package provides CPython bindings for reading and writing MessagePack data.

Install

You can use pip or easy_install to install msgpack:

$ easy_install msgpack-python
  or
$ pip install msgpack-python
PyPy

msgpack-python provides pure python implementation. PyPy can use this.

Windows

When you can't use binary distribution, you need to install Visual Studio or Windows SDK on Windows. (NOTE: Visual C++ Express 2010 doesn't support amd64. Windows SDK is recommanded way to build amd64 msgpack without any fee.)

Without extension, using pure python implementation on CPython runs slowly.

Notes
Note for msgpack 2.0 support

msgpack 2.0 adds two types: bin and ext.

raw was bytes or string type like Python 2's str. To distinguish string and bytes, msgpack 2.0 adds bin. It is non-string binary like Python 3's bytes.

To use bin type for packing bytes, pass use_bin_type=True to packer argument.

>>> import msgpack
>>> packed = msgpack.packb([b'spam', u'egg'], use_bin_type=True)
>>> msgpack.unpackb(packed, encoding='utf-8')
['spam', u'egg']

You shoud use it carefully. When you use use_bin_type=True, packed binary can be unpacked by unpackers supporting msgpack-2.0.

To use ext type, pass msgpack.ExtType object to packer.

>>> import msgpack
>>> packed = msgpack.packb(msgpack.ExtType(42, b'xyzzy'))
>>> msgpack.unpackb(packed)
ExtType(code=42, data='xyzzy')

You can use it with default and ext_hook. See below.

Note for msgpack 0.2.x users

The msgpack 0.3 have some incompatible changes.

The default value of use_list keyword argument is True from 0.3. You should pass the argument explicitly for backward compatibility.

Unpacker.unpack() and some unpack methods now raises OutOfData instead of StopIteration. StopIteration is used for iterator protocol only.

How to use
One-shot pack & unpack

Use packb for packing and unpackb for unpacking. msgpack provides dumps and loads as alias for compatibility with json and pickle.

pack and dump packs to file-like object. unpack and load unpacks from file-like object.

>>> import msgpack
>>> msgpack.packb([1, 2, 3])
'\x93\x01\x02\x03'
>>> msgpack.unpackb(_)
[1, 2, 3]

unpack unpacks msgpack's array to Python's list, but can unpack to tuple:

>>> msgpack.unpackb(b'\x93\x01\x02\x03', use_list=False)
(1, 2, 3)

You should always pass the use_list keyword argument. See performance issues relating to use_list_ below.

Read the docstring for other options.

Streaming unpacking

Unpacker is a "streaming unpacker". It unpacks multiple objects from one stream (or from bytes provided through its feed method).

import msgpack
from io import BytesIO

buf = BytesIO()
for i in range(100):
   buf.write(msgpack.packb(range(i)))

buf.seek(0)

unpacker = msgpack.Unpacker(buf)
for unpacked in unpacker:
    print unpacked
Packing/unpacking of custom data type

It is also possible to pack/unpack custom data types. Here is an example for datetime.datetime.

import datetime

import msgpack

useful_dict = {
    "id": 1,
    "created": datetime.datetime.now(),
}

def decode_datetime(obj):
    if b'__datetime__' in obj:
        obj = datetime.datetime.strptime(obj["as_str"], "%Y%m%dT%H:%M:%S.%f")
    return obj

def encode_datetime(obj):
    if isinstance(obj, datetime.datetime):
        return {'__datetime__': True, 'as_str': obj.strftime("%Y%m%dT%H:%M:%S.%f")}
    return obj


packed_dict = msgpack.packb(useful_dict, default=encode_datetime)
this_dict_again = msgpack.unpackb(packed_dict, object_hook=decode_datetime)

Unpacker's object_hook callback receives a dict; the object_pairs_hook callback may instead be used to receive a list of key-value pairs.

Extended types

It is also possible to pack/unpack custom data types using the msgpack 2.0 feature.

>>> import msgpack
>>> import array
>>> def default(obj):
...     if isinstance(obj, array.array) and obj.typecode == 'd':
...         return msgpack.ExtType(42, obj.tostring())
...     raise TypeError("Unknown type: %r" % (obj,))
...
>>> def ext_hook(code, data):
...     if code == 42:
...         a = array.array('d')
...         a.fromstring(data)
...         return a
...     return ExtType(code, data)
...
>>> data = array.array('d', [1.2, 3.4])
>>> packed = msgpack.packb(data, default=default)
>>> unpacked = msgpack.unpackb(packed, ext_hook=ext_hook)
>>> data == unpacked
True
Advanced unpacking control

As an alternative to iteration, Unpacker objects provide unpack, skip, read_array_header and read_map_header methods. The former two read an entire message from the stream, respectively deserialising and returning the result, or ignoring it. The latter two methods return the number of elements in the upcoming container, so that each element in an array, or key-value pair in a map, can be unpacked or skipped individually.

Each of these methods may optionally write the packed data it reads to a callback function:

from io import BytesIO

def distribute(unpacker, get_worker):
    nelems = unpacker.read_map_header()
    for i in range(nelems):
        # Select a worker for the given key
        key = unpacker.unpack()
        worker = get_worker(key)

        # Send the value as a packed message to worker
        bytestream = BytesIO()
        unpacker.skip(bytestream.write)
        worker.send(bytestream.getvalue())
Note about performance
GC

CPython's GC starts when growing allocated object. This means unpacking may cause useless GC. You can use gc.disable() when unpacking large message.

use_list option

List is the default sequence type of Python. But tuple is lighter than list. You can use use_list=False while unpacking when performance is important.

Python's dict can't use list as key and MessagePack allows array for key of mapping. use_list=False allows unpacking such message. Another way to unpacking such object is using object_pairs_hook.

Test

MessagePack uses pytest for testing. Run test with following command:

$ py.test

msgpack/msgpack-erlang https://github.com/msgpack/msgpack-erlang

MessagePack Erlang

Travis

Drone.io

prequisites for runtime

Erlang/OTP, >= R15B -- for older version, rebar won't work. Also based on the new msgpack spec 232a0d.

edit rebar.config to use in your application
{deps, [
  {msgpack, ".*",
    {git, "git://github.com/msgpack/msgpack-erlang.git", "master"}}
]}.
Simple deserialization
Ham = msgpack:pack(Spam),
{ok, Spam} = msgpack:unpack(Ham).
Stream deserialization
{Term0, Rest0} = msgpack:unpack_stream(Binary),
{Term1, Rest1} = msgpack:unpack_stream(Rest0),
...
String type

Now this supports string type!

Opt = [{enable_str, true}]
{ok, "埼玉"} = msgpack:unpack(msgpack:pack("埼玉", Opt), Opt).
 => {ok,[22524,29577]}

There are several options for msgpack:pack/2 and msgpack:unpack/2 . See msgpack:options() in msgpack.hrl.

Map Style

Since Erlang/OTP 17.0

msgpack:pack(#{ <<"key">> => <<"value">> }, [{format, map}]).

Or use old jiffy/jsx style

msgpack:pack({[{<<"key">>, <<"value">>}]}, [{format, jiffy}]),
msgpack:pack([{<<"key">>, <<"value">>}], [{format, jsx}]).
Ext type

Now msgpack-erlang supports ext type. Now you can serialize everything with your original (de)serializer. That will enable us to handle erlang- native types like pid(), ref() contained in tuple(). See test/msgpack_ext_example_tests.erl for example code.

Packer = fun({ref, Ref}, Opt) when is_reference(Ref) -> {ok, {12, term_to_binary(Ref)}} end,
Unpacker = fun(12, Bin) -> {ok, {ref, binary_to_term(Bin)}} end,
Ref = make_ref(),
Opt = [{ext,{Packer,Unpacker}}],
{ok, {ref, Ref}} = msgpack:unpack(msgpack:pack({ref, Ref}, Opt), Opt).

This is still experimental feature, so I'm waiting for your feedback.

Compatibility mode

To use as same with old spec:

OldHam = msgpack:pack(Spam, [{enable_str,false}]),
{ok, Spam} = msgpack:unpack(OldHam, [{enable_str,false}]).

Since 0.2.3 now it's false by default.

Further tests

See msgpack-erlang-tests for further tests

License

Apache License 2.0

Release Notes
0.3.2
  • set back default style as jiffy
  • fix bugs around nil/null handling
0.3.0
  • supports map new in 17.0
  • jiffy-style maps will be deprecated in near future
  • set default style as map
0.2.8

0.2 series works with OTP 17.0, R16, R15, and with MessagePack's new and old format. But does not support map type introduced in OTP 17.0.

It also supports JSX-compatible mode.

msgpack/msgpack-ruby https://github.com/msgpack/msgpack-ruby

MessagePack for Ruby
require 'msgpack'
msg = [1,2,3].to_msgpack  #=> "\x93\x01\x02\x03"
MessagePack.unpack(msg)   #=> [1,2,3]
Install
gem install msgpack
Use cases
  • Create REST API returing MessagePack using Rails + RABL
  • Store objects efficiently in memcached or Redis
  • Upload data in efficient format from mobile devices. See also MessagePack for Objective-C and Java
Links
Streaming API
# serialize a 2-element array [e1, e2]
pk = MessagePack::Packer.new(io)
pk.write_array_header(2).write(e1).write(e2).flush
# deserialize objects from an IO
u = MessagePack::Unpacker.new(io)
u.each { |obj| ... }
# event-driven deserialization
def on_read(data)
  @u ||= MessagePack::Unpacker.new
  @u.feed_each(data) { |obj| ... }
end

msgpack/msgpack-scala https://github.com/msgpack/msgpack-scala

MessagePack for Scala

An implementation of MessagePack for Scala.

Quick Start

Quick Start is available at MessagePack official site.

Requirement
  • sbt 0.12.2 or later
Installation
By using sbt

This project also supports sbt 0.13.5 To build JAR file of MessagePack for Scala, please type the following command:

$ package
How to release the project

To relese the project (compile, test, tagging, deploy), please use the commands as follows:

$ mvn release:prepare
$ mvn release:perform
Make intellij project

Please type the following command.Then intellij project will be generated.

$ sbt gen-idea

msgpack/msgpack-haskell https://github.com/msgpack/msgpack-haskell

mneumann/MessagePack https://github.com/mneumann/MessagePack

MessagePack

An alternative msgpack.org implementation for Ruby and C++

msgpack/msgpack-cli https://github.com/msgpack/msgpack-cli

MessagePack for CLI
What is it?

This is MessagePack serialization/deserialization for CLI (Common Language Infrastructure) implementations such as .NET Framework, Silverlight, Mono (including Moonlight.) This library can be used from ALL CLS compliant languages such as C#, F#, Visual Basic, Iron Pyhton, Iron Ruby, PowerShell, C++/CLI or so.

Usage

You can serialize/deserialize objects as following: 1. Create serializer via MessagePackSerializer.Create generic method. This method creates dependent types serializers as well. 1. Invoke serializer as following: ** Pack method with destination Stream and target object for serialization. ** Unpack method with source Stream.

// Creates serializer.
var serializer = SerializationContext.Default.GetSerializer<T>();
// Pack obj to stream.
serializer.Pack(stream, obj);
// Unpack from stream.
var unpackedObject = serializer.Unpack(stream);
' Creates serializer.
Dim serializer = SerializationContext.Default.GetSerializer(Of T)()
' Pack obj to stream.
serializer.Pack(stream, obj)
' Unpack from stream.
Dim unpackedObject = serializer.Unpack(stream)
Features
  • Fast and interoperable binary format serialization with simple API.
  • Generating pre-compiled assembly for rapid start up.
  • Flexible MessagePackObject which represents MessagePack type system naturally.
Documentation

See wiki

Installation
  • Binary files distributed via the NuGet package MsgPack.Cli.
  • You can extract binary (DLL) file as following:
    1. Download *.zip file from GitHub Release page.
    2. Extract it.
    3. Under the bin directory, binaries are there!
      • For mono, you can use net45 or net35 drops as you run with.
      • For Unity, unity3d drop is suitable.
How to build
For .NET Framework
  1. Install recent Windows SDK (at least, .NET Framework 4 Client Profile and MSBuild is needed.)
    Or install Visual Studio or Visual Studio Express.
    1. If you want to build unit test assemblies, install NuGet and then restore NUnit packages.
  2. Run:

    msbuild MsgPack.sln

    Or (for .NET 3.5 drops, Silverlight 4 drops, and Silverlight 5 drops):

    msbuild MsgPack.compats.sln

Or open one of above solution files in your IDE and run build command in it.

For Mono

Open MsgPack.mono.sln with MonoDevelop and then click Build menu item. (Of cource, you can build via xbuild.)

See also

msgpack/msgpack-c https://github.com/msgpack/msgpack-c

Msgpack for C/C++

It's like JSON but small and fast.

Overview

MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON. But it's faster and smaller. Small integers are encoded into a single byte, and typical short strings require only one extra byte in addition to the strings themselves.

License

Msgpack is Copyright (C) 2008-2014 FURUHASHI Sadayuki and licensed under the Apache License, Version 2.0 (the "License"). For details see the COPYING file in this directory.

Contributing

The source for msgpack-c is held at msgpack-c github.com site.

To report an issue, use the msgpack-c issue tracker at github.com.

Version

0.5.9 Build Status

Using Msgpack
Header only library for C++

When you use msgpack on C++03 and C++11, you just add msgpack-c/include to your include path. You don't need to link any msgpack libraries.

e.g.)

g++ -I msgpack-c/include your_source_file.cpp

If you want to use C version of msgpack, you need to build it. You can also install C and C++ version of msgpack.

Building and Installing Install from git repository
Using autotools

You will need gcc (4.1.0 or higher), autotools.

For C: C++03 and C:

$ git clone https://github.com/redboltz/msgpack-c/tree/cxx_separate
$ cd msgpack-c
$ ./bootstrap
$ ./configure
$ make
$ sudo make install

For C++11:

$ git clone https://github.com/msgpack/msgpack-c.git
$ cd msgpack-c
$ ./bootstrap
$ ./configure CXXFLAGS="-std=c++11"
$ make
$ sudo make install

You need the compiler that fully supports C++11.

Using cmake
CUI

You will need gcc (4.1.0 or higher), cmake.

$ git clone https://github.com/msgpack/msgpack-c.git
$ cd msgpack-c
$ cmake .
$ make
$ sudo make install

If you want to setup C++11 version of msgpack, execute the following command:

$ git clone https://github.com/msgpack/msgpack-c.git
$ cd msgpack-c
$ cmake -DMSGPACK_CXX11=ON .
$ sudo make install

You need the compiler that fully supports C++11.

GUI on Windows

Clone msgpack-c git repository.

$ git clone https://github.com/msgpack/msgpack-c.git

or using GUI git client.

e.g.) tortoise git https://code.google.com/p/tortoisegit/

  1. Launch cmake GUI client. http://www.cmake.org/cmake/resources/software.html

  2. Set 'Where is the source code:' text box and 'Where to build the binaries:' text box.

  3. Click 'Configure' button.

  4. Choose your Visual Studio version.

  5. Click 'Generate' button.

  6. Open the created msgpack.sln on Visual Studio.

  7. Build all.

Code Example
#include <msgpack.hpp>
#include <vector>
#include <string>
#include <iostream>

int main() {
    // This is target object.
    std::vector<std::string> target;
    target.push_back("Hello,");
    target.push_back("World!");

    // Serialize it.
    msgpack::sbuffer sbuf;  // simple buffer
    msgpack::pack(&sbuf, target);

    // Deserialize the serialized data.
    msgpack::unpacked msg;    // includes memory pool and deserialized object
    msgpack::unpack(msg, sbuf.data(), sbuf.size());
    msgpack::object obj = msg.get();

    // Print the deserialized object to stdout.
    std::cout << obj << std::endl;    // ["Hello," "World!"]

    // Convert the deserialized object to staticaly typed object.
    std::vector<std::string> result;
    obj.convert(&result);

    // If the type is mismatched, it throws msgpack::type_error.
    obj.as<int>();  // type is mismatched, msgpack::type_error is thrown
}
Documents

You can get addtional information on the wiki:

https://github.com/msgpack/msgpack-c/wiki/cpp_overview

msgpack/msgpack-ocaml https://github.com/msgpack/msgpack-ocaml

MsgPack for OCaml
BULID
$ make
$ sudo make install
EXAMPLE
Serialize/Deserialize for Msgpack Object
(* serialize *)
let bytes = 
  Msgpack.Serialize.serialize_string (`FixArray [`PFixnum 1; `PFixnum 2; `PFixnum 3])

(* deserialize *)
let obj =
  Msgpack.Serialize.deserialize_string bytes
Serialize/Deserialize for OCaml types (with meta_conv)
open Msgpack_conv

type t = {
  int : int;
  str : string;
} with conv(msgpack)

(* serialize *)
let bytes = 
  Msgpack.Serialize.serialize_string (msgpack_of_t { int = 42; str = "ans" })

(* deserialize *)
let obj =
  t_of_msgpack (Msgpack.Serialize.deserialize_string bytes)

See also, examlpe/

TEST
$ ocaml setup.ml -configure --enable-tests
$ make test
PROOF

If you want to use msgpack at OCaml, you need not do this section. This section for user intrested in formal verification.

You need Coq 8.4 and omake.

$ cd proof
$ make
$ cp *.ml ../lib

msgpack/msgpack-smalltalk https://github.com/msgpack/msgpack-smalltalk

msgpack-smalltalk

MessagePack serialization library for various Smalltalk dialects.

  • Squeak
  • Pharo
  • VisualWorks
  • VA Smalltalk
  • Dolphin Smalltalk
  • GNU Smalltalk (Beta)
  • Cuis

Sources are put as Cypress for the neutral accesses from various Smalltalk dialects.

How to use
Serialization
MpMessagePack pack: <your object>

or:

<your object> messagePacked
Deserialization
MpMessagePack unpack: msgpackBytes

or:

Object fromMessagePack: msgBytes
Samples
map := Dictionary new.
map at: 'someArray' asByteArray put: #(1 2.2 #[3 4 5]).
packed := map messagePacked.
(Object fromMessagePack: packed) inspect.
writeStream := WriteStream on: ByteArray new.
encoder := MpEncoder on: writeStream.
encoder nextPut: 1.
encoder nextPut: #(2 3).
dic := Dictionary new.
dic at: 4 put: 5.
encoder nextPut: dic.
encoder nextPut: 'four' asByteArray.
bytes := encoder contents.
readStream := ReadStream on: bytes.
decoder := MpDecoder on: readStream.
[decoder atEnd] whileFalse: [
        Transcript cr; show: decoder next printString
]

We are moving from Google Code site. Old contents including installation guide are still there.

loteixeira/as3-msgpack https://github.com/loteixeira/as3-msgpack

MessagePack for Actionscript3 (Flash, Flex and AIR).

as3-msgpack was designed to work with the interfaces IDataInput and IDataOutput, thus the API might be easily connected with the native classes that handle binary data (such as ByteArray, Socket, FileStream and URLStream).
Moreover, as3-msgpack is capable of decoding data from binary streams.
Get started: http://loteixeira.github.io/lib/2013/08/19/as3-msgpack/

Basic usage (encoding/decoding):

// create messagepack object
var msgpack:MsgPack = new MsgPack();

// encode an array
var bytes:ByteArray = msgpack.write([1, 2, 3, 4, 5]);

// rewind the buffer
bytes.position = 0;

// print the decoded object
trace(msgpack.read(bytes));

For downloads, source code and further information, check the project repository: https://github.com/loteixeira/as3-msgpack.

msgpack/msgpack-php https://github.com/msgpack/msgpack-php

Msgpack for PHP

Build Status

This extension provide API for communicating with MessagePack serialization.

MessagePack is a binary-based efficient object serialization library. It enables to exchange structured objects between many languages like JSON. But unlike JSON, it is very fast and small.

Requirement
  • PHP 5.0 +
Install
Install from PECL

Msgpack is an PECL extension, thus you can simply install it by:

pecl install msgpack
Compile Msgpack from source
$/path/to/phpize
$./configure 
$make && make install
Example
<?php
$data = array(0=>1,1=>2,2=>3);
$msg = msgpack_pack($data);
$data = msgpack_unpack($msg);
?>
Resources

fperrad/lua-MessagePack http://fperrad.github.io/lua-MessagePack/

MessagePack for Lua (spec v5)

Build Status Coverage Status Licence Dependencies

local mp = require 'MessagePack'
mpac = mp.pack(data)
data = mp.unpack(mpac)
Install

You can use LuaRocks to install lua-MessagePack:

$ luarocks install lua-messagepack

or from the source, with:

$ make install

It is a pure Lua implementation, without any dependency.

Links
Copyright and License

Copyright (c) 2012-2014 Francois Perrad rank

This library is licensed under the terms of the MIT/X11 license, like Lua itself.

mneumann/rust-msgpack https://github.com/mneumann/rust-msgpack

rust-msgpack Build Status

Msgpack implementation for Rust language.

Installation

Simply include the rust-msgpack in your Cargo dependencies.

[dependencies.msgpack]

git = "git@github.com:mneumann/rust-msgpack.git"
Quickstart
extern crate msgpack;

fn main() {
  let arr = vec!["str1".to_string(), "str2".to_string()];
  let str = msgpack::Encoder::to_msgpack(&arr).ok().unwrap();
  println!("Encoded: {}", str);

  let dec: Vec<String> = msgpack::from_msgpack(str).ok().unwrap();
  println!("Decoded: {}", dec);
}

To enable your own data structures to be automatically serialized from and to msgpack, derive from Encodable and Decodable as shown in the following example:

extern crate serialize;

#[deriving(Encodable,Decodable)]
struct MyStruct {
  a: Vec<u32>,
  s: String
}
Testing
cargo test
License

This code licensed under the same terms as Rust itself: dual MIT/Apache2 license options.

mururu/msgpack-elixir https://github.com/mururu/msgpack-elixir

MessagePack for Elixir

Build Status

Installation

Add :message_pack as a dependency in your mix.exs file.

defp deps do
  [{:message_pack, "~> 0.1.4"}]
end
Usage
# pack
MessagePack.pack([1,2,3]) #=> { :ok, <<147,1,2,3>> }
MessagePack.pack!([1,2,3]) #=> <<147,1,2,3>>

# unpack
MessagePack.unpack(<<147,1,2,3>>) #=> { :ok, [1,2,3] }
MessagePack.unpack!(<<147,1,2,3>>) #=> [1,2,3]

# unpack_once
MessagePack.unpack_once(<<147,1,2,3,4>>) #=> {:ok, {[1, 2, 3], <<4>>}}
MessagePack.unpack_once!(<<147,1,2,3,4>>) #=> {[1, 2, 3], <<4>>}
Options
  • enable_string

Support string type. This options is false by default.

iex(1)> { :ok, bin } = MessagePack.pack(<<255>>)
{:ok, <<161, 255>>}
iex(3)> MessagePack.unpack(<<161, 255>>)
{:ok, <<255>>}
iex(4)> MessagePack.unpack(<<161, 255>>, enable_string: true)
{:error, {:invalid_string, <<255>>}}
  • ext

Support extention type.

See test/message_pack_ext_test.exs.

License

MIT

ktakashi/r6rs-msgpack https://github.com/ktakashi/r6rs-msgpack

MessagePack for R6RS Scheme

This is an implementation of MessagePack for R6RS Scheme.

API references

Function (pack! bv message)
Function (pack! bv message offset)

Pack message to message pack format bytevector and put it into the bv destructively. Given bv must have enough length to hold the message.

Optional argument offset indicates where to start with, default is 0.

Function (pack message)

The same as pack! but this one creates a new bytevector.

Function (pack-size message)

Calculate the converted message size.

Function (unpack bv)
Function (unpack bv offset)

Unpack the given message format bytevector to Scheme object.

Optional argument offset indicates where to start with, default is 0.

Function (get-unpack in)

Unpack the given binary input port to Scheme object.

Conversion rules

As you already know, Scheme doesn't have static types so the conversion of Scheme objects to message pack data might cause unexpected results. To avoid it, I will describe how conversion works.

Scheme to message pack Integer conversion

The library automatically decides proper size. More specifically, if it can fit to message pack's fixnum then library uses it, so are uint8-64. If the number is too big, then an error is raised. Users must know it tries to use uint as much as possible. If the given number was negative then sint will be used.

Floating point conversion

Unfortunately R6RS doesn't have difference between float and double. So when flonum is given then it always converts to double number.

Collection conversion

Message pack has collections which are map and array. And these are associated with alist (association list) and vector respectively. When you want to convert alist to message pack data, then you need to make sure the cdr part will be the data and if you put (("key" "value))_ then it will be converted to nested map.

The collection size calculation is done automatically. It tries to use the smallest size.

Message pack to Scheme

The other way around is easy, it can simply restore the byte data to Scheme object. Following describes the conversion rules;

Positive fixnum -> integer
Negative fixnum -> integer
uint8, uint16, uint32, uint64 -> integer
sint8, sint16, sint32, sint64 -> integer
Map -> alist
Array -> vector
fixstr, str8, str16, str32 -> string bit8, bit16, bit32 -> bytevector

Tested Scheme implementations

Sagittarius Scheme

Mosh Scheme

Ypsilon

Your contributions are always welcome.

TODO
  • More testing
  • Extended type handling

ugorji/go https://github.com/ugorji/go

MessagePack and Binc Codec for Go Language.

A High Performance, Feature-Rich, Idiomatic encode/decode and rpc library.

To install:

go get github.com/ugorji/go/codec

Source: [http://github.com/ugorji/go]
Online documentation: [http://godoc.org/github.com/ugorji/go/codec]

Typical usage:

    // create and use decoder/encoder
    var (
        v interface{} // value to decode/encode into
        r io.Reader
        w io.Writer
        b []byte
        mh codec.MsgpackHandle
    )

    dec = codec.NewDecoder(r, &mh)
    dec = codec.NewDecoderBytes(b, &mh)
    err = dec.Decode(&v) 

    enc = codec.NewEncoder(w, &mh)
    enc = codec.NewEncoderBytes(&b, &mh)
    err = enc.Encode(v)

    //RPC Server
    go func() {
        for {
            conn, err := listener.Accept()
            rpcCodec := codec.GoRpc.ServerCodec(conn, h)
            //OR rpcCodec := codec.MsgpackSpecRpc.ServerCodec(conn, h)
            rpc.ServeCodec(rpcCodec)
        }
    }()

    //RPC Communication (client side)
    conn, err = net.Dial("tcp", "localhost:5555")
    rpcCodec := codec.GoRpc.ClientCodec(conn, h)
    //OR rpcCodec := codec.MsgpackSpecRpc.ClientCodec(conn, h)
    client := rpc.NewClientWithCodec(rpcCodec)

suzukaze/mruby-msgpack MessagePack for mruby

mruby-msgpack Build Status
Welcome to MessagePack for mruby

MessagePack for mruby is MessagePack implimented in mruby language.

msg = [1, 2, 3].to_msgpack  #=> "\x93\x01\x02\x03"
MessagePack.unpack(msg)     #=> [1, 2, 3]

This is early vesion. Please check the methods that work in test foloder.

Platform

I test MessagePack for mruby in mac OSX 10.8.4. In the future it will work in Windows and Linux OS.

Getting Started
  1. Download MessagePack for mruby at the command prompt:

    git clone https://github.com/suzukaze/mruby-msgpack.git
    
  2. Add config.gem line to build_config.rb

MRuby::Build.new do |conf|

  # ...(snip)...
  conf.gem :git => 'https://github.com/suzukaze/mruby-msgpack.git'
end
  1. Test at the command prompt:

    rake test
    
  2. Build at the command prompt:

    rake
    
msgpack-ruby commit

mruby-msgpack is based on msgpack-ruby(a7c2dc34ef69c9132167e38009baa8420c460c9b)

Contributing

I encourage you to contribute to MessagePack for mruby!

License

Author : Jun Hiroe

Copyrigh : Copyright (c) 2013 Jun Hiroe

License : MIT License

jingweno/msgpack_rails https://github.com/jingweno/msgpack_rails

MessagePack for Rails Build Status

The Rails way to serialize/deserialize objects with Message Pack. It implements the ActiveSupport encoder & decoder and the ActiveModel serializer for Message Pack.

Installation

Add this line to your application's Gemfile:

gem 'msgpack_rails'

And then execute:

$ bundle

Or install it yourself as:

$ gem install msgpack_rails
Usage

msgpack_rails converts data type using as_json before feeding it into msgpack. Here are a few examples:

$ ActiveSupport::MessagePack.encode(:a => :b)
=> "\x81\xA1a\xA1b"

$ ActiveSupport::MessagePack.encode(Time.now)
=> "\xB92013-09-11T10:40:39-07:00"

$ Time.now.as_msgpack
=> "2013-09-11T10:48:13-07:00"

$ Time.now.to_msgpack
=> "\xB92013-09-11T10:40:39-07:00"

$ ActiveSupport::MessagePack.decode Time.now.to_msgpack
=> "2013-09-11T11:23:07-07:00"

# After setting ActiveSupport.parse_msgpack_times to true
$ ActiveSupport::MessagePack.decode Time.now.to_msgpack
=> Wed, 11 Sep 2013 11:25:18 -0700

You can also use it as part of ActiveModel, similar to to_json:

class Contact
  include ActiveModel::Serializers::MessagePack

  ...
end

@contact = Contact.new
@contact.name = 'Owen Ou'
@contact.age = 28
@contact.created_at = Time.utc(2006, 8, 1)
@contact.awesome = true
@contact.preferences = { 'shows' => 'anime' }

@contact.to_msgpack                # => msgpack output
@contact.to_msgpack(:root => true) # => include root in msgpack output
Contributing
  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

vsergeev/u-msgpack-python https://github.com/vsergeev/u-msgpack-python

u-msgpack-python v2.0

u-msgpack-python is a lightweight MessagePack serializer and deserializer module written in pure Python, compatible with both Python 2 and 3, as well CPython and PyPy implementations of Python. u-msgpack-python is fully compliant with the latest MessagePack specification.

u-msgpack-python is currently distributed on PyPI: https://pypi.python.org/pypi/u-msgpack-python and as a single file: umsgpack.py

Installation

With pip:

$ pip install u-msgpack-python

With easy_install:

$ easy_install u-msgpack-python

or simply drop umsgpack.py into your project!

$ wget https://raw.github.com/vsergeev/u-msgpack-python/master/umsgpack.py
Examples

Basic Example:

>>> import umsgpack
>>> umsgpack.packb({u"compact": True, u"schema": 0})
'\x82\xa7compact\xc3\xa6schema\x00'
>>> umsgpack.unpackb(_)
{u'compact': True, u'schema': 0}
>>> 

A more complicated example:

>>> umsgpack.packb(
        [1, True, False, 0xffffffff, {u"foo": b"\x80\x01\x02",
         u"bar": [1,2,3, {u"a": [1,2,3,{}]}]}, -1, 2.12345] )
'\x97\x01\xc3\xc2\xce\xff\xff\xff\xff\x82\xa3foo\xc4\x03\x80\x01
\x02\xa3bar\x94\x01\x02\x03\x81\xa1a\x94\x01\x02\x03\x80\xff\xcb
@\x00\xfc\xd3Z\x85\x87\x94'
>>> umsgpack.unpackb(_)
[1, True, False, 4294967295, {u'foo': '\x80\x01\x02',
 u'bar': [1, 2, 3, {u'a': [1, 2, 3, {}]}]}, -1, 2.12345]
>>> 

Streaming serialization with file-like objects:

>>> f = open('test.bin', 'w')
>>> umsgpack.pack({u"compact": True, u"schema": 0}, f)
>>> umsgpack.pack([1,2,3], f)
>>> f.close()
>>> 
>>> f = open('test.bin')
>>> umsgpack.unpack(f)
{u'compact': True, u'schema': 0}
>>> umsgpack.unpack(f)
[1, 2, 3]
>>> f.close()
>>> 

Encoding and decoding an application-defined ext type:

>>> # Create an Ext object with type 0x05 and data b"\x01\x02\x03"
... foo = umsgpack.Ext(0x05, b"\x01\x02\x03")
>>> umsgpack.packb({u"special stuff": foo, u"awesome": True})
b'\x82\xadspecial stuff\xc7\x03\x05\x01\x02\x03\xa7awesome\xc3'
>>> bar = umsgpack.unpackb(_)
>>> print(bar["special stuff"])
Ext Object (Type: 0x05, Data: 01 02 03)
>>> bar["special stuff"].type
5
>>> bar["special stuff"].data
b'\x01\x02\x03'
>>> 

Python standard library style names dump, dumps, load, loads are also available:

>>> import umsgpack
>>> umsgpack.dumps({u"compact": True, u"schema": 0})
'\x82\xa7compact\xc3\xa6schema\x00'
>>> umsgpack.loads(_)
{u'compact': True, u'schema': 0}
>>> 
>>> f = open('test.bin', 'w')
>>> umsgpack.dump({u"compact": True, u"schema": 0}, f)
>>> f.close()
>>> 
>>> f = open('test.bin')
>>> umsgpack.load(f)
{u'compact': True, u'schema': 0}
>>> 
More Information

See the project page for more information on old specification compatibility mode, exceptions, behavior, and testing.

License

u-msgpack-python is MIT licensed. See the included LICENSE file for more details.

kmsquire/MsgPack.jl https://github.com/kmsquire/MsgPack.jl

MsgPack

Build Status

Provides basic support for the msgpack format.

julia> import MsgPack

julia> MsgPack.pack("hi")
3-element Array{Uint8,1}:
 0xa2
 0x68
 0x69

julia> a = MsgPack.pack([1,2,"hi"])
6-element Array{Uint8,1}:
 0x93
 0x01
 0x02
 0xa2
 0x68
 0x69

julia> MsgPack.unpack(MsgPack.pack(4.5))
4.5

julia> f = open("in.mp")
julia> MsgPack.unpack(f)
"hello"

julia> f2 = open("out.mp", "w")
julia> MsgPack.pack(f2, [1,2,"hi"])



NOTE: The standard method for encoding integers in msgpack is to use the most compact representation possible, and to encode negative integers as signed ints and non-negative numbers as unsigned ints.

For compatibility with other implementations, I'm following this convention. On the unpacking side, every integer type becomes an Int64 in Julia, unless it doesn't fit (ie. values greater than 2^63 are unpacked as Uint64).

I might change this at some point, and/or provide a way to control the unpacked types.

edma2/clojure-msgpack https://github.com/edma2/clojure-msgpack

clojure-msgpack

clojure-msgpack is a library for

  • serializing native Clojure data structures as MessagePack byte arrays.
  • deserializing MessagePack byte arrays into native Clojure data structures.
Installation

See https://clojars.org/clojure-msgpack

Usage
(require '[msgpack.core :refer :all])

(pack {:compact true :schema 0})
; #<byte[] [B@5984b649>

; Declare records and types as Extended types.
(defrecord Employee [name])

(defext Employee 1 [e]
  (.getBytes (:name e)))

(let [bob (Employee. "Bob")
      bytes (pack bob)]
  (map #(format "0x%x" %) bytes))
; ("0xc7" "0x3" "0x1" "0x62" "0x6f" "0x62")

; Or encode an extension directly
(pack (->Extension 1 (.getBytes "Bob")))

(pack {:compact true :schema 0})
; #<byte[] [B@59ce1eed>

(unpack (pack {:compact true :schema 0}))
; {"schema" 0, "compact" true}
License

clojure-msgpack is MIT licensed. See the included LICENSE file for more details.

tkob/mlmsgpack https://github.com/tkob/mlmsgpack

ML-MessagePack

MessagePack implementation for Standard ML (SML)

Features
  • Portable: Depends only on the required components of the SML Basis Library specification.
  • Composable: Composable combinators for encoding and decoding.
Usage
MLton and MLKit

Include mlmsgpack.mlb in your MLB file.

Poly/ML

From the interactive shell, use .sml files in the following order.

  • mlmsgpack-aux.sml
  • realprinter-default.sml
  • mlmsgpack.sml
SML/NJ

Use mlmsgpack.cm.

Moscow ML

From the interactive shell, use .sml files in the following order.

  • large.sml
  • mlmsgpack-aux.sml
  • realprinter-fail.sml
  • mlmsgpack.sml

Makefile.mosml is also provided.

HaMLet

From the interactive shell, use .sml files in the following order.

  • mlmsgpack-aux.sml
  • realprinter-fail.sml
  • mlmsgpack.sml
Alice ML

From the interactive shell, use .sml files in the following order.

  • mlmsgpack-aux.sml
  • realprinter-fail.sml
  • mlmsgpack.sml
SMLSharp

For separate compilation, .smi files are provided. Require mlmsgpack.smi from your .smi file.

From the interactive shell, use .sml files in the following order.

  • mlmsgpack-aux.sml
  • realprinter-default.sml
  • mlmsgpack.sml
Tutorial

See TUTORIAL.md.

Known Problems

Our recommendation is MLton, MLKit, Poly/ML and SML#(>=2.0.0) as all tests passed on them. SML/NJ and Moscow ML are fine if you don't use real values.

SML/NJ

Packing real values fail or produces imprecise results in some cases.

Moscow ML

Packing real values is not supported, since some components of the SML Basis Library are not provided.

HaMLet

Packing real values is not supported, since some components of the SML Basis Library are not provided. Some functions are very slow, although they work properly. (We tested HaMLet compiled with MLton.)

Alice ML

Packing real values is not supported, since some components of the SML Basis Library are not provided. Also, some unit tests fail.

SMLSharp

Most functions do not work properly because of bugs of SML# prior to version 2.0.0.

Status

Ext is not supported yet.

See Also

There already exists another MessagePack implemenatation for SML, called MsgPack-SML, which is targeted for MLton.

https://msgpacksml.codeplex.com/

ML-MessagePack is written from scratch and not a fork of MsgPack-SML.

For information on MessagePack, see:

http://msgpack.org/

camgunz/cmp https://github.com/camgunz/cmp

cmp

CMP is a C implementation of the MessagePack serialization format. It currently implements version 5 of the MessagePack Spec.

CMP's goal is to be lightweight and straightforward, forcing nothing on the programmer.

License

While I'm a big believer in the GPL, I license CMP under the MIT license.

Example Usage

The following examples use a file as the backend, and are modeled after the examples included with the msgpack-c project.

#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>

#include "cmp.h"

static bool read_bytes(void *data, size_t sz, FILE *fh) {
    return fread(data, sizeof(uint8_t), sz, fh) == (sz * sizeof(uint8_t));
}

static bool file_reader(cmp_ctx_t *ctx, void *data, size_t limit) {
    return read_bytes(data, limit, (FILE *)ctx->buf);
}

static size_t file_writer(cmp_ctx_t *ctx, const void *data, size_t count) {
    return fwrite(data, sizeof(uint8_t), count, (FILE *)ctx->buf);
}

void error_and_exit(const char *msg) {
    fprintf(stderr, "%s\n\n", msg);
    exit(EXIT_FAILURE);
}

int main(void) {
    FILE *fh = NULL;
    cmp_ctx_t cmp;
    uint32_t array_size = 0;
    uint32_t str_size = 0;
    char hello[6] = {0, 0, 0, 0, 0, 0};
    char message_pack[12] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};

    fh = fopen("cmp_data.dat", "w+b");

    if (fh == NULL)
        error_and_exit("Error opening data.dat");

    cmp_init(&cmp, fh, file_reader, file_writer);

    if (!cmp_write_array(&cmp, 2))
        error_and_exit(cmp_strerror(&cmp));

    if (!cmp_write_str(&cmp, "Hello", 5))
        error_and_exit(cmp_strerror(&cmp));

    if (!cmp_write_str(&cmp, "MessagePack", 11))
        error_and_exit(cmp_strerror(&cmp));

    rewind(fh);

    if (!cmp_read_array(&cmp, &array_size))
        error_and_exit(cmp_strerror(&cmp));

    /* You can read the str byte size and then read str bytes... */

    if (!cmp_read_str_size(&cmp, &str_size))
        error_and_exit(cmp_strerror(&cmp));

    if (str_size > (sizeof(hello) - 1))
        error_and_exit("Packed 'hello' length too long\n");

    if (!read_bytes(hello, str_size, fh))
        error_and_exit(cmp_strerror(&cmp));

    /*
     * ...or you can set the maximum number of bytes to read and do it all in
     * one call
     */

    str_size = sizeof(message_pack);
    if (!cmp_read_str(&cmp, message_pack, &str_size))
        error_and_exit(cmp_strerror(&cmp));

    printf("Array Length: %zu.\n", array_size);
    printf("[\"%s\", \"%s\"]\n", hello, message_pack);

    fclose(fh);

    return EXIT_SUCCESS;
}
Advanced Usage

See the examples folder.

Fast, Lightweight, Flexible, and Robust

CMP uses no internal buffers; conversions, encoding and decoding are done on the fly.

CMP's source and header file together are < 2,500 LOC.

CMP makes no heap allocations.

CMP uses standardized types rather than declaring its own, and it depends only on stdbool.h, stdint.h and string.h. It has no link-time dependencies, not even the C Standard Library.

CMP is written using C89 (ANSI C), aside, of course, from its use of fixed-width integer types and bool.

On the other hand, CMP's test suite depends upon the C Standard Library and requires C99.

CMP only requires the programmer supply a read function and a write function. In this way, the programmer can use CMP on memory, files, sockets, etc.

CMP is portable. It uses fixed-width integer types, and checks the endianness of the machine at runtime before swapping bytes (MessagePack is big-endian).

CMP provides a fairly comprehensive error reporting mechanism modeled after errno and strerror.

CMP is threadsafe; while contexts cannot be shared between threads, each thread may use its own context freely.

CMP is tested using the MessagePack test suite as well as a large set of custom test cases. Its small test program is compiled with clang using -Wall -Werror -Wextra ... along with several other flags, and generates no compilation errors.

CMP's source is written as readably as possible, using explicit, descriptive variable names and a consistent, clear style.

CMP's source is written to be as secure as possible. Its testing suite checks for invalid values, and data is always treated as suspect before it passes validation.

CMP's API is designed to be clear, convenient and unsurprising. Strings are null-terminated, binary data is not, error codes are clear, and so on.

Building

There is no build system for CMP. The programmer can drop cmp.c and cmp.h in their source tree and modify as necessary. No special compiler settings are required to build it, and it generates no compilation errors in either clang or gcc.

danellis/dart-msgpack https://github.com/danellis/dart-msgpack

dart-msgpack

This is a very early release of my MessagePack library for Dart. Currently, message classes must be written by hand. For example:

class NotificationFrame extends Message {
    String kind;
    Map<String, Object> data;

    NotificationFrame(this.kind, this.data);

    static NotificationFrame fromList(List f) => new NotificationFrame(f[0], f[1]);
    List toList() => [kind, data];
}

For each class you need to define the fromList and toList methods, which convert from and to a list of fields respectively.

For example usage, see the unit tests.

reeze/msgpack-hhvm https://github.com/reeze/msgpack-hhvm

Gab-km/msgpack-fsharp https://github.com/Gab-km/msgpack-fsharp

MessagePack for F#
Build status
What is this?

MessagePack is a fast and compact binary serialization library.

MessagePack for F# is a MessagePack implementation of F#, by F#, for F#.

Usage
open MsgPack

[| 1uy; 2uy; 3uy |]
|> Array.map (Value.UInt8)
|> Value.Array
|> Packer.packOne
//=> val it : byte [] = [|147uy; 1uy; 2uy; 3uy|]

Unpacker.unpack [|147uy; 1uy; 2uy; 3uy|]
//=> [|Value.Array [|Value.UInt8 1uy; Value.UInt8 2uy; Value.UInt8 3uy|]|]
Copyright

Copyright (c) 2014- Kazuhiro Matsushima

License

Distributed under the Apache License, Version 2.0 .

komamitsu/jackson-dataformat-msgpack https://github.com/komamitsu/jackson-dataformat-msgpack

jackson-dataformat-msgpack

Build Status

This Jackson extension library handles reading and writing of data encoded in MessagePack data format. It extends standard Jackson streaming API (JsonFactory, JsonParser, JsonGenerator), and as such works seamlessly with all the higher level data abstractions (data binding, tree model, and pluggable extensions).

Maven dependency

To use this module on Maven-based projects, use following dependency:

<dependency>
  <groupId>org.komamitsu</groupId>
  <artifactId>jackson-dataformat-msgpack</artifactId>
  <version>0.0.3</version>
</dependency>
Usage

Only thing you need to do is to instantiate MessagePackFactory and pass it to the constructor of ObjectMapper.

  ObjectMapper objectMapper = new ObjectMapper(new MessagePackFactory());
  ExamplePojo orig = new ExamplePojo("komamitsu");
  byte[] bytes = objectMapper.writeValueAsBytes(orig);
  ExamplePojo value = objectMapper.readValue(bytes, ExamplePojo.class);
  System.out.println(value.getName()); // => komamitsu

Also, you can exchange data among multiple languages.

Java

  // Serialize
  Map<String, Object> obj = new HashMap<String, Object>();
  obj.put("foo", "hello");
  obj.put("bar", "world");
  byte[] bs = objectMapper.writeValueAsBytes(obj);
  // bs => [-126, -93, 102, 111, 111, -91, 104, 101, 108, 108, 111,
  //        -93, 98, 97, 114, -91, 119, 111, 114, 108, 100]

Ruby

  require 'msgpack'

  # Deserialize
  xs = [-126, -93, 102, 111, 111, -91, 104, 101, 108, 108, 111,
        -93, 98, 97, 114, -91, 119, 111, 114, 108, 100]
  MessagePack.unpack(xs.pack("C*"))
  # => {"foo"=>"hello", "bar"=>"world"}

  # Serialize
  ["zero", 1, 2.0, nil].to_msgpack.unpack('C*')
  # => [148, 164, 122, 101, 114, 111, 1, 203, 64, 0, 0, 0, 0, 0, 0, 0, 192]

Java

  // Deserialize
  bs = new byte[] {(byte) 148, (byte) 164, 122, 101, 114, 111, 1,
                   (byte) 203, 64, 0, 0, 0, 0, 0, 0, 0, (byte) 192};
  TypeReference<List<Object>> typeReference = new TypeReference<List<Object>>(){};
  List<Object> xs = objectMapper.readValue(bs, typeReference);
  // xs => [zero, 1, 2.0, null]

vertexclique/thealchemist https://github.com/vertexclique/thealchemist

The Alchemist

Build Status

Elixir implementation bundle for MessagePack

The Alchemist

Elixir implementation relies on mururu/msgpack-elixir GitHub version

Pure erlang implementation used with msgpack/msgpack-erlang GitHub version

Usage
 In pure elixir:
# pack
TheAlchemist.pack([1,2,3]) #=> { :ok, <<147,1,2,3>> }
TheAlchemist.pack!([1,2,3]) #=> <<147,1,2,3>>

# unpack
TheAlchemist.unpack(<<147,1,2,3>>) #=> { :ok, [1,2,3] }
TheAlchemist.unpack!(<<147,1,2,3>>) #=> [1,2,3]

# unpack_once
TheAlchemist.unpack_once(<<147,1,2,3,4>>) #=> {:ok, {[1, 2, 3], <<4>>}}
TheAlchemist.unpack_once!(<<147,1,2,3,4>>) #=> {[1, 2, 3], <<4>>}

 With underlying erlang: Deserialization
msgpacked = TheAlchemistErl.pack('{"test":false}')
TheAlchemistErl.unpack(msgpacked)
Stream deserialization
mat = TheAlchemistErl.unpack_stream(data)
TheAlchemistErl.unpack_stream(elem(mat, 1)),
String type

Now this supports string type!

opt = [{:enable_str, true}]
TheAlchemistErl.unpack(TheAlchemistErl.pack("埼玉kanji", opt), opt)
 => {:ok, "埼玉kanji"}
Map Style

Since Erlang/OTP 17.0

msgpack:pack(#{ <<"key">> => <<"value">> }, [{format, map}]).

Or use old jiffy/jsx style

msgpack:pack({[{<<"key">>, <<"value">>}]}, [{format, jiffy}]),
msgpack:pack([{<<"key">>, <<"value">>}], [{format, jsx}]).
Running interactively

Firstly make all to ensure all source files are compiled after that you can bring the ERL shell with

make run

If you want to bring the IEX you can type

make runex

briandw/SwiftPack https://github.com/briandw/SwiftPack

SwiftPack

SwiftPack is MessagePack packer and unpacker written almost entirely in Swift.

Issues
  • There are few Objective-C dependencies, mostly NSData. I hope to remove these dependencies as Swift matures.
  • Maps currently only support String types for keys.
Copyright and License

Copyright (c) 2014 Brian Williams

This software is licensed under the terms of the MIT license.

gabriel/MPMessagePack https://github.com/gabriel/MPMessagePack

Install
pod "MPMessagePack"
Writing
#import <MPMessagePack/MPMessagePack.h>

NSDictionary *dict =
@{
@"n": @(32134123),
@"bool": @(YES),
@"array": @[@(1.1f), @(2.1)],
@"body": [NSData data],
};

NSData *data = [dict mp_messagePack];

// To get error info
NSError *error = nil;
NSData *data = [MPMessagePackWriter writeObject:dict error:&error];
Reading
id obj = [MPMessagePackReader readData:data error:&error];

rodrigosetti/messagepack http://hackage.haskell.org/package/messagepack

MessagePack for Haskell

This implementation defines an messagepack Object type, which is an instance of Serialize (from cereal ):

data Object = ObjectInt    Int
            | ObjectNil
            | ObjectBool   Bool
            | ObjectFloat  Float
            | ObjectDouble Double
            | ObjectString Text
            | ObjectBinary ByteString
            | ObjectArray  [Object]
            | ObjectMap    (Map Object Object )
    deriving (Eq, Ord, Show)

instance Serialize Object where
    -- ...

Thus, you can use cereal's encode and decode to pack and unpack objects.

ymofen/msgpack-delphi https://github.com/ymofen/msgpack-delphi

Msgpack for Delphi

It's like JSON but small and fast.

unit Owner: D10.Mofen, qdac.swish
contract:
       qq:185511468, 
    email:185511468@qq.com
welcome to report bug
Works with
  • Delphi 7 (tested)
  • Delphi 2007 (tested)
  • XE5, XE6, XE7, FMX (tested)
Code Example
var
  lvMsg, lvMsg2:TSimpleMsgPack;
  lvBytes:TBytes;
  s:string;
begin
  lvMsg := TSimpleMsgPack.Create;
  lvMsg.S['key.obj'] := '汉字,ascii';
  if dlgOpen.Execute then
  begin
    lvMsg.S['key.image.name'] := ExtractFileName(dlgOpen.FileName);

    // file binary data
    lvMsg.ForcePathObject('key.image.data').LoadBinaryFromFile(dlgOpen.FileName);
  end;

  //
  lvBytes := lvMsg.EncodeToBytes;

  lvMsg2 := TSimpleMsgPack.Create;
  lvMsg2.DecodeFromBytes(lvBytes);
  //
  Memo1.Lines.Add(lvMsg2.S['key.obj']);
  if lvMsg2.S['key.image.name'] <> '' then
  begin
    s := ExtractFilePath(ParamStr(0)) + lvMsg2.S['key.image.name'];
    Memo1.Lines.Add('file saved');
    Memo1.Lines.Add(s);
    lvMsg2.ForcePathObject('key.image.data').SaveBinaryToFile(s);    
  end;

chinawsb/qmsgpack-delphi https://github.com/chinawsb/qmsgpack-delphi

Msgpack for Delphi
QMsgPack is a MessagePack protocol implementation for Delphi and C++ Builder.

The lowest version is Delphi/C++ Builder 2006,and tested with Delphi/C++ Builder 2007/XE/XE6,Other version should support,but not tested. QMsgPack is a part of QDAC 3.0,so if you has any question,please contact QDAC 3.0 team with follow information:

E-Mail:chinawsb@sina.com
QQ Group:250530692
Forum:http://tieba.baidu.com/f?kw=qdac
Blog:http://hi.baidu.com/chineseswish
SVN:http://svn.code.sf.net/p/qdac3/code/ or svn://svn.code.sf.net/p/qdac3/code/
Author:swish
Code Example
var
  lvMsg, lvMsg2:TQMsgPack;
  lvBytes:TBytes;
  s:string;
begin
  lvMsg := TQMsgPack.Create;
  lvMsg.ForcePath('key.obj').AsString := '汉字,ascii';

  //
  lvBytes := lvMsg.Encode;

  lvMsg2 := TQMsgPack.Create;
  lvMsg2.Parse(lvBytes);
  //
  showMessage(lvMsg.ForcePath('key.obj').AsString);
  ....

philhofer/msgp https://github.com/philhofer/msgp

MessagePack Code Generator

forthebadge forthebadge

This is a code generation tool for serializing Go structs using the MesssagePack standard. It is targeted at the go generate tool.

Use

*** Note: go generate is a go 1.4+ feature. ***

In a source file, include the following directive:

//go:generate msgp

The msgp command will generate serialization methods for all exported struct definitions in the file. You will need to include that directive in every file that contains structs that need code generation. The generated files will be named {filename}_gen.go by default (but can be overridden with the -o flag.) Additionally, it will write tests and benchmarks in {{filename}}_gen_test.go.

Field names can be overridden in much the same way as the encoding/json package. For example:

type Person struct {
    Name       string `msg:"name"`
    Address    string `msg:"address"`
    Age        int    `msg:"age"`
    Hidden     string `msg:"-"` // this field is ignored
    unexported bool             // this field is also ignored
}

Running msgp on a file containing the above declaration would generate the following methods:

func (z *Person) MarshalMsg() ([]byte, error)

func (z *Person) UnmarshalMsg(b []byte) ([]byte, error)

func (z *Person) EncodeTo(en *enc.MsgWriter) (n int, err error)

func (z *Person) DecodeFrom(dc *enc.MsgReader) (n int, err error)

func (z *Person) EncodeMsg(w io.Writer) (n int, err error)

func (z *Person) DecodeMsg(r io.Reader) (n int, err error)

Each method is optimized for a certain use-case, depending on whether or not the user can afford to recycle *enc.MsgWriter and *enc.MsgReader object, and whether or not the user is dealing with io.Reader/io.Writer or []byte. (Pro tip: EncodeTo is the fastest marshaller, and UnmarshalMsg is the fastest unmarshaller. Also, *enc.MsgReaders are buffered, and you can create one with an arbitrary buffer size with enc.NewDecoderSize.)

The msgp/enc package has utility functions for transforming MessagePack to JSON directly, both from io.Readers and []byte.

Features
  • Proper handling of pointers, nullables, and recursive types
  • Backwards-compatible decoding logic
  • Efficient and readable generated code
  • JSON interoperability
  • Support for embedded fields, anonymous structs, and multi-field inline declarations
  • Identifier resolution (see below)
  • Support for Go's time.Time, complex64, and complex128 types through extensions
  • Generation of both []byte-oriented and io.Reader/io.Writer-oriented methods.

For example, this will work fine:

type MyInt int
type Data []byte
type Struct struct {
    Which  map[string]*MyInt `msg:"which"`
    Other  Data              `msg:"other"`
}

As long as the declarations of MyInt and Data are in the same file, the parser will figure out that MyInt is really an int, and Data is really just a []byte. Note that this only works for "base" types (no slices or maps, although []byte is supported as a special case.) Unresolved identifiers are (optimistically) assumed to be struct definitions in other files. (The parser will spit out warnings about unresolved identifiers.)

Status

Alpha. Here are the known limitations/restrictions:

  • All fields of a struct that are not Go built-ins are assumed to satisfy the enc.MsgEncoder and enc.MsgDecoder interfaces. This will only actually be the case if the declaration for that type is in a file touched by the code generator. The generator will output a warning if it can't resolve an identifier in the file, or if it ignores an exported field. The generated code will fail to compile if you encounter this issue, so it shouldn't catch you by surprise.
  • Like most serializers, chan and func fields are ignored, as well as non-exported fields.
  • Methods are only generated for struct definitions. Chances are that we will keep things this way.
  • Encoding/decoding of interface{} can be flaky. It works fine for go builtins, but don't count on it working well for anything beyond that.
  • Maps must have string keys. This is intentional (as it preserves JSON interop.) The generator will yell at you if you try to use something else. Although non-string map keys are not explicitly forbidden by the messagepack standard, many serializers impose this restriction.

I have no idea whether or not this generator will work with your code. Luckily, if the output compiles, then there's a pretty good chance things are fine. (Plus, we generate tests for you.) Please, please, please file an issue if you think the generator is writing broken code.

Performance

If you like benchmarks, this is the fastest and lowest-memory-footprint serializer for Go in this test. (I put considerable effort into generating high-performance code. I figure there's no point in putting up with the extra complexity of generated code if it does not buy you a substantial performance improvement.)

As you might imagine, the generated code is quite a lot more performant than reflection-based serialization; generally you can expect a ~4x speed improvement and an order of magnitude fewer memory allocations when compared to other reflection-based messagepack serialization libraries. (Typically, the EncodeTo method does zero allocations. UnmarshalMsg can also do zero allocations in the best case.) YMMV.

JSON

MessagePack (hex)