MessagePack

MessagePack for Actionscript3 (Flash, Flex and AIR).

as3-msgpack was designed to work with the interfaces IDataInput and IDataOutput, thus the API might be easily connected with the native classes that handle binary data (such as ByteArray, Socket, FileStream and URLStream).
Moreover, as3-msgpack is capable of decoding data from binary streams.
Get started: http://loteixeira.github.io/lib/2013/08/19/as3-msgpack/

Basic usage (encoding/decoding):

// create messagepack object
var msgpack:MsgPack = new MsgPack();

// encode an array
var bytes:ByteArray = msgpack.write([1, 2, 3, 4, 5]);

// rewind the buffer
bytes.position = 0;

// print the decoded object
trace(msgpack.read(bytes));

For downloads, source code and further information, check the project repository: https://github.com/loteixeira/as3-msgpack.

MsgPack

MessagePack implementation for Arduino (compatible with other C++ apps)

Feature

• one-line [serialize / deserialize] for almost all standard type of C++ same as msgpack-c
• support custom class [serialization / deserialization]
• support working with ArduinoJSON
• one-line [save / load] between custom serializable MsgPack class and JSON file
• one-line [save / load] custom serializable MsgPack class [to / from] EEPROM

Typical Usage

This library is only for serialize / deserialize. To send / receive serialized data with Stream class, please use MsgPacketizer.

#include <MsgPack.h>

// input to msgpack
int i = 123;
float f = 1.23;
MsgPack::str_t s = "str"; // std::string or String
MsgPack::arr_t<int> v {1, 2, 3}; // std::vector or arx::vector
MsgPack::map_t<String, float> m {{"one", 1.1}, {"two", 2.2}, {"three", 3.3}}; // std::map or arx::map

// output from msgpack
int ri;
float rf;
MsgPack::str_t rs;
MsgPack::arr_t<int> rv;
MsgPack::map_t<String, float> rm;

void setup() {
    delay(2000);
    Serial.begin(115200);
    Serial.println("msgpack test start");

    // serialize to msgpack
    MsgPack::Packer packer;
    packer.serialize(i, f, s, v, m);

    // deserialize from msgpack
    MsgPack::Unpacker unpacker;
    unpacker.feed(packer.data(), packer.size());
    unpacker.deserialize(ri, rf, rs, rv, rm);

    if (i != ri) Serial.println("failed: int");
    if (f != rf) Serial.println("failed: float");
    if (s != rs) Serial.println("failed: string");
    if (v != rv) Serial.println("failed: vector<int>");
    if (m != rm) Serial.println("failed: map<string, int>");

    Serial.println("msgpack test success");
}

void loop() {}

Encode / Decode to Collections without Container

In msgpack, there are two collection types: Array and Map. C++ containers will be converted to one of them but you can do that from individual parameters. To pack / unpack values as such collections in a simple way, please use these functions.

packer.to_array(i, f, s); // becoms array format [i, f, s];
unpacker.from_array(ii, ff, ss); // unpack from array format to ii, ff, ss

packer.to_map("i", i, "f", f); // becoms {"i":i, "f":f}
unpacker.from_map(ki, ii, kf, ff); // unpack from map to ii, ff, ss

The same conversion can be achieved using serialize and deserialize.

packer.serialize(MsgPack::arr_size_t(3), i, f, s); // [i, f, s]
unpacker.deserialize(MsgPack::arr_size_t(3), ii, ff, ss);

packer.serialize(MsgPack::map_size_t(2), "i", i, "f", f); // {"i":i, "f":f}
unpacker.deserialize(MsgPack::map_size_t(2), ki, ii, kf, ff);

Here, MsgPack::arr_size_t and MsgPack::map_size_t are used to identify the size of Array and Map format in serialize or deserialize. This way is expandable to pack and unpack complex data structure because it can be nested.

// {"i":i, "arr":[ii, iii]}
packer.serialize(MsgPack::map_size_t(2), "i", i, "arr", MsgPack::arr_size_t(2), ii, iii);
unpacker.deserialize(MsgPack::map_size_t(2), ki, i, karr, MsgPack::arr_size_t(2), ii, iii);

Custom Class Adaptation

To serialize / deserialize custom type you defined, please use MSGPACK_DEFINE() macro inside of your class. This macro enables you to convert your custom class to Array format.

struct CustomClass {
    int i;
    float f;
    MsgPack::str_t s;
    MSGPACK_DEFINE(i, f, s); // -> [i, f, s]
};

After that, you can serialize your class completely same as other types.

int i;
float f;
MsgPack::str_t s;
CustomClass c;

MsgPack::Packer packer;
packer.serialize(i, f, s, c);
// -> packer.serialize(i, f, s, arr_size_t(3), c.i, c.f, c.s)

int ii;
float ff;
MsgPack::str_t ss;
CustomClass cc;

MsgPack::Unpacker unpacker;
unpacker.feed(packer.data(), packer.size());
unpacker.deserialize(ii, ff, ss, cc);

You can also wrap your custom class to Map format by using MSGPACK_DEFINE_MAP macro. Please note that you need "key" string for Map format.

struct CustomClass {
    MsgPack::str_t key_i {"i"}; int i;
    MsgPack::str_t key_f {"f"}; float f;
    MSGPACK_DEFINE_MAP(key_i, i, key_f, f); // -> {"i":i, "f":f}
};

CustomClass c;
MsgPack::Packer packer;
packer.serialize(c);
// -> packer.serialize(map_size_t(2), c.key_i, c.i, c.key_f, c.f)

CustomClass cc;
MsgPack::Unpacker unpacker;
unpacker.feed(packer.data(), packer.size());
unpacker.deserialize(cc);

Also you can use MSGPACK_BASE() macro to pack values of base class.

struct Base {
    int i;
    float f;
    MSGPACK_DEFINE(i, f);
};

struct Derived : public Base {
    MsgPack::str_t s;
    MSGPACK_DEFINE(s, MSGPACK_BASE(Base));
    // -> packer.serialize(arr_size_t(2), s, arr_size_t(2), Base::i, Base::f)
};

If you wamt to use Map format in derived class, add "key" for your MSGPACK_BASE.

struct Derived : public Base {
    MsgPack::str_t key_s; MsgPack::str_t s;
    MsgPack::str_t key_b; // key for base class
    MSGPACK_DEFINE_MAP(key_s, s, key_b, MSGPACK_BASE(Base));
    // -> packer.serialize(map_size_t(2), key_s, s, key_b, arr_size_t(2), Base::i, Base::f)
};

You can nest custom classes to express complex data structure.

// serialize and deserialize nested structure
// {"i":i, "f":f, "a":["str", {"first":1, "second":"two"}]}

// {"first":1, "second":"two"}
struct MyMap {
    MsgPack::str_t key_first; int i;
    MsgPack::str_t key_second; MsgPack::str_t s;
    MSGPACK_DEFINE_MAP(key_first, i, key_second, s);
};

// ["str", {"first":1, "second":"two"}]
struct MyArr {
    MsgPack::str_t s;
    MyMap m;
    MSGPACK_DEFINE(s, m):
};

// {"i":i, "f":f, "a":["str", {"first":1, "second":"two"}]}
struct MyNestedClass {
    MsgPack::str_t key_i; int i;
    MsgPack::str_t key_f; int f;
    MsgPack::str_t key_a;
    MyArr arr;
    MSGPACK_DEFINE_MAP(key_i, i, key_f, f, key_a, arr);
};

And you can serialize / deserialize as same as other types.

MyNestedClass c;
MsgPack::Packer packer;
packer.serialize(c);

MyNestedClass cc;
MsgPack::Unpacker unpacker;
unpacker.feed(packer.data(), packer.size());
unpacker.deserialize(cc);

JSON and Other language's msgpack compatibility

In other languages like JavaScript, Python and etc. has also library for msgpack. But some libraries can NOT convert msgpack in "plain" style. They always wrap them into collections like Array or Map by default. For example, you can't convert "plain" format in other languages.

packer.serialize(i, f, s);                // "plain" format is NOT unpackable
packer.serialize(arr_size_t(3), i, f, s); // unpackable if you wrap that into Array

It is because the msgpack is used as based on JSON (I think). So you need to use Array format for JSON array, and Map for Json Object. To achieve that, there are several ways.

• use to_array or to_map to convert to simple structure
• use serialize() or deserialize() with arr_size_t / map_size_t for complex structure
• use custom class as JSON array / object which is wrapped into Array / Map
• use custom class nest recursively for more complex structure
• use ArduinoJson for more flexible handling of JSON

• you can [serialize / deserialize] StaticJsonDocument<N> and DynamicJsonDocument directly

#include <ArduinoJson.h>  // include before MsgPack.h
#include <MsgPack.h>

void setup() {
    StaticJsonDocument<200> doc_in;
    MsgPack::Packer packer;
    packer.serialize(doc_in); // serialize directly

    StaticJsonDocument<200> doc;
    MsgPack::Unpacker unpacker;
    unpacker.feed(packer.data(), packer.size());
    unpacker.deserialize(doc); // deserialize directly
}

Utilities

You can directly save/load to/from JSON file with this library. SD, SdFat, SD_MMC, SPIFFS, etc. are available for the target file system. Please see save_load_as_json_file example for more details.

#include <SD.h>
#include <MsgPack.h>

struct MyConfig {
    Meta meta;
    Data data;
    MSGPACK_DEFINE(meta, data);
};

MyConfig config;

void setup() {
    SD.begin();

    // load json data from /config.txt to config struct directly
    MsgPack::file::load_from_json_static<256>(SD, "/config.txt", config);

    // change your configuration...

    // save config data from config struct to /config.txt as json directly
    MsgPack::file::save_as_json_static<256>(SD, "/config.txt", config);
}

In Arduino, you can use the MsgPack utility to save/load to/from EEPROM. Following code shows how to use them. Please see save_load_eeprom example for more details.

struct MyConfig {
    Meta meta;
    Data data;
    MSGPACK_DEFINE(meta, data);
};

MyConfig config;

void setup() {
    EEPROM.begin();

    // load current config
    MsgPack::eeprom::load(config);

    // change your configuration...

    // save
    MsgPack::eeprom::save(config);

    EEPROM.end();
}

Supported Type Adaptors

These are the lists of types which can be serialize and deserialize. You can also pack() or unpack() variable one by one.

• MsgPack::object::nil_t

• bool

• char (signed/unsigned)
• ints (signed/unsigned)

• float
• double

• char*
• char[]
• std::string or String(Arduino) (MsgPack::str_t)

• unsigned char* (need to serialize(ptr, size) or pack(ptr, size))
• unsigned char[] (need to serialize(ptr, size) or pack(ptr, size))
• std::vector<char> (MsgPack::bin_t<char>)
• std::vector<unsigned char> (MsgPack::bin_t<unsigned char>)
• std::array<char>
• std::array<unsigned char>

• T[] (need to serialize(ptr, size) or pack(ptr, size))
• std::vector (MsgPack::arr_t<T>)
• std::array (MsgPack::fix_arr_t<T, N>)
• std::deque
• std::pair
• std::tuple
• std::list
• std::forward_list
• std::set
• std::multiset
• std::unordered_set
• std::unordered_multiset

• std::map (MsgPack::map_t<T>)
• std::multimap
• std::unordered_map
• std::unordered_multimap

• MsgPack::object::ext

• MsgPack::object::timespec

• std::queue
• std::priority_queue
• std::bitset
• std::stack

• unordered_xxx cannot be used in all Arduino
• C-style array and pointers are supported only packing.
• for NO-STL Arduino, following types can be used
  • all types of NIL, Bool, Integer, Float, Str, Bin
  • for Array, only T[], MsgPack::arr_t<T> (arx::vector<T>), and MsgPack::fix_arr_t<T, N> (arx::array<T, N>) can be used
  • for Map, only MsgPack::map_t<T, U> (arx::map<T, U>) can be used
  • for the detail of arx::xxx, see ArxContainer

There are some additional types are defined to express msgpack formats easily.

These types have type aliases like this:

• MsgPack::str_t = String (Arduino only)
• MsgPack::bin_t<T> = std::vector<T>
• MsgPack::arr_t<T> = std::vector<T>
• MsgPack::fix_arr_t<T, N> = std::array<T, N>
• MsgPack::map_t<T, U> = std::map<T, U>

For general C++ apps (not Arduino), str_t is defined as:

• MsgPack::str_t = std::string

MsgPack::object::nil_t is used to pack and unpack Nil type. This object is just a dummy and do nothing.

MsgPack::object::ext holds binary data of Ext type.

// create ext type with args: int8_t, const uint8_t*, uint32_t
MsgPack::object::ext e(type, bin_ptr, size);
MsgPack::Packer packer;
packer.serialize(e); // serialize ext type

MsgPack::object::ext r;
msgPack::Unpacker unpacker;
unpacker.feed(packer.data(), packer.size());
unpacker.deserialize(r); // deserialize ext type

MsgPack::object::timespec is used to pack and unpack Timestamp type.

MsgPack::object::timespec t = {
    .tv_sec  = 123456789, /* int64_t  */
    .tv_usec = 123456789  /* uint32_t */
};
MsgPack::Packer packer;
packer.serialize(t); // serialize timestamp type

MsgPack::object::timespec r;
msgPack::Unpacker unpacker;
unpacker.feed(packer.data(), packer.size());
unpacker.deserialize(r); // deserialize timestamp type

Other Options

Error information report is disabled by default. You can enable it by defining this macro.

#define MSGPACK_DEBUGLOG_ENABLE

Also you can change debug info stream by calling this macro (default: Serial).

DEBUG_LOG_ATTACH_STREAM(Serial1);

See DebugLog for details.

STL is used to handle packet data by default, but for following boards/architectures, ArxContainer is used to store the packet data because STL can not be used for such boards. The storage size of such boards for max packet binary size and number of msgpack objects are limited.

• AVR
• megaAVR
• SAMD

As mentioned above, for such boards like Arduino Uno, the storage sizes are limited. And of course you can manage them by defining following macros. But these default values are optimized for such boards, please be careful not to excess your boards storage/memory.

// msgpack serialized binary size
#define MSGPACK_MAX_PACKET_BYTE_SIZE  128
// max size of MsgPack::arr_t
#define MSGPACK_MAX_ARRAY_SIZE          8
// max size of MsgPack::map_t
#define MSGPACK_MAX_MAP_SIZE            8
// msgpack objects size in one packet
#define MSGPACK_MAX_OBJECT_SIZE        24

These macros have no effect for STL enabled boards.

In addtion for such boards, type aliases for following types are different from others.

• MsgPack::str_t = String
• MsgPack::bin_t<T> = arx::vector<T, N = MSGPACK_MAX_PACKET_BYTE_SIZE>
• MsgPack::arr_t<T> = arx::vector<T, N = MSGPACK_MAX_ARRAY_SIZE>
• MsgPack::map_t<T, U> = arx::map<T, U, N = MSGPACK_MAX_MAP_SIZE>

Please see "Memory Management" section and ArxContainer for detail.

For such boards, there are several STL libraries, like ArduinoSTL, StandardCPlusPlus, and so on. But such libraries are mainly based on uClibc++ and it has many lack of function. I considered to support them but I won't support them unless uClibc++ becomes much better compatibility to standard C++ library. I reccomend to use low cost but much better performance chip like ESP series.

Embedded Libraries

• ArxTypeTraits v0.2.3
• ArxContainer v0.4.0
• DebugLog v0.6.6
• TeensyDirtySTLErrorSolution v0.1.0

Used Inside of

• MsgPacketizer

APIs

// reserve internal buffer
void reserve_buffer(const size_t size);

// variable sized serializer for any type
template <typename First, typename ...Rest>
void serialize(const First& first, Rest&&... rest);
template <typename T>
void serialize(const arr_size_t& arr_size, Args&&... args);
template <typename ...Args>
void serialize(const map_size_t& map_size, Args&&... args);
template <size_t N>
void serialize(const StaticJsonDocument<N>& doc, const size_t num_max_string_type = 32);
void serialize(const DynamicJsonDocument& doc, const size_t num_max_string_type = 32);
void serialize_arduinojson(const JsonDocument& doc, const size_t num_max_string_type = 32);

// variable sized serializer to array or map for any type
template <typename ...Args>
void to_array(Args&&... args);
template <typename ...Args>
void to_map(Args&&... args);

// single arg packer for any type
template <typename T>
void pack<T>(const T& t);
template <typename T>
void pack<T>(const T* ptr, const size_t size); // only for pointer types

// accesor and utility for serialized binary data
const bin_t<uint8_t>& packet() const;
const uint8_t* data() const;
size_t size() const;
size_t indices() const;
void clear();

// abstract serializer for msgpack formats
// serialize() and pack() are wrapper for these methods
void packInteger(const T& value); // accept both uint and int
void packFloat(const T& value);
void packString(const T& str);
void packString(const T& str, const size_t len);
void packBinary(const uint8_t* bin, const size_t size);
void packArraySize(const size_t size);
void packMapSize(const size_t size);
void packFixExt(const int8_t type, const T value);
void packFixExt(const int8_t type, const uint64_t value_h, const uint64_t value_l);
void packFixExt(const int8_t type, const uint8_t* ptr, const uint8_t size);
void packFixExt(const int8_t type, const uint16_t* ptr, const uint8_t size);
void packFixExt(const int8_t type, const uint32_t* ptr, const uint8_t size);
void packFixExt(const int8_t type, const uint64_t* ptr, const uint8_t size);
void packExt(const int8_t type, const T* ptr, const U size);
void packExt(const object::ext& e);
void packTimestamp(const object::timespec& time);

// serializer for detailed msgpack format
// serialize() and pack() are wrapper for these methods
void packNil();
void packNil(const object::nil_t& n);
void packBool(const bool b);
void packUInt7(const uint8_t value);
void packUInt8(const uint8_t value);
void packUInt16(const uint16_t value);
void packUInt32(const uint32_t value);
void packUInt64(const uint64_t value);
void packInt5(const int8_t value);
void packInt8(const int8_t value);
void packInt16(const int16_t value);
void packInt32(const int32_t value);
void packInt64(const int64_t value);
void packFloat32(const float value);
void packFloat64(const double value);
void packString5(const str_t& str);
void packString5(const str_t& str, const size_t len);
void packString5(const char* value);
void packString5(const char* value, const size_t len);
void packString8(const str_t& str);
void packString8(const str_t& str, const size_t len);
void packString8(const char* value);
void packString8(const char* value, const size_t len);
void packString16(const str_t& str);
void packString16(const str_t& str, const size_t len);
void packString16(const char* value);
void packString16(const char* value, const size_t len);
void packString32(const str_t& str);
void packString32(const str_t& str, const size_t len);
void packString32(const char* value);
void packString32(const char* value, const size_t len);
void packString5(const __FlashStringHelper* str);
void packString5(const __FlashStringHelper* str, const size_t len);
void packString8(const __FlashStringHelper* str);
void packString8(const __FlashStringHelper* str, const size_t len);
void packString16(const __FlashStringHelper* str);
void packString16(const __FlashStringHelper* str, const size_t len);
void packString32(const __FlashStringHelper* str);
void packString32(const __FlashStringHelper* str, const size_t len);
void packBinary8(const uint8_t* value, const uint8_t size);
void packBinary16(const uint8_t* value, const uint16_t size);
void packBinary32(const uint8_t* value, const uint32_t size);
void packArraySize4(const uint8_t value);
void packArraySize16(const uint16_t value);
void packArraySize32(const uint32_t value);
void packMapSize4(const uint8_t value);
void packMapSize16(const uint16_t value);
void packMapSize32(const uint32_t value);
void packFixExt1(const int8_t type, const uint8_t value);
void packFixExt2(const int8_t type, const uint16_t value);
void packFixExt2(const int8_t type, const uint8_t* ptr);
void packFixExt2(const int8_t type, const uint16_t* ptr);
void packFixExt4(const int8_t type, const uint32_t value);
void packFixExt4(const int8_t type, const uint8_t* ptr);
void packFixExt4(const int8_t type, const uint32_t* ptr);
void packFixExt8(const int8_t type, const uint64_t value);
void packFixExt8(const int8_t type, const uint8_t* ptr);
void packFixExt8(const int8_t type, const uint64_t* ptr);
void packFixExt16(const int8_t type, const uint64_t value_h, const uint64_t value_l);
void packFixExt16(const int8_t type, const uint8_t* ptr);
void packFixExt16(const int8_t type, const uint64_t* ptr);
void packExtSize8(const int8_t type, const uint8_t size);
void packExtSize16(const int8_t type, const uint16_t size);
void packExtSize32(const int8_t type, const uint32_t size);
void packTimestamp32(const uint32_t unix_time_sec);
void packTimestamp64(const uint64_t unix_time);
void packTimestamp64(const uint64_t unix_time_sec, const uint32_t unix_time_nsec);
void packTimestamp96(const int64_t unix_time_sec, const uint32_t unix_time_nsec);

// reserve internal buffer for indices
void reserve_indices(const size_t size);

// feed data to deserialize
bool feed(const uint8_t* data, size_t size);

// variable sized deserializer
template <typename First, typename ...Rest>
bool deserialize(First& first, Rest&&... rest);
template <size_t N>
bool deserialize(StaticJsonDocument<N>& doc);
bool deserialize(DynamicJsonDocument& doc);

// varibale sized desrializer for array and map
template <typename ...Args>
bool from_array(Args&&... args);
template <typename ...Args>
bool from_map(Args&&... args);

// single arg deserializer
template <typename T>
bool unpack(T& value);

// check if next arg can be deserialized to value
template <typename T>
bool unpackable(const T& value) const;

// accesor and utility for deserialized msgpack data
bool decode_ready() const;
bool decoded() const;
size_t size() const;
void index(const size_t i);
size_t index() const;
void clear();

// abstract deserializer for msgpack formats
// deserialize() and unpack() are wrapper for these methods
T unpackUInt();
T unpackInt();
T unpackFloat();
str_t unpackString();
bin_t<T> unpackBinary();
bin_t<T> unpackBinary();
size_t unpackArraySize();
size_t unpackMapSize();
object::ext unpackExt();
object::timespec unpackTimestamp();

// deserializer for detailed msgpack format
// these methods check deserialize index overflow and type mismatch
// deserialize() and unpack() are wrapper for these methods
bool unpackNil();
bool unpackBool();
uint8_t unpackUInt7();
uint8_t unpackUInt8();
uint16_t unpackUInt16();
uint32_t unpackUInt32();
uint64_t unpackUInt64();
int8_t unpackInt5();
int8_t unpackInt8();
int16_t unpackInt16();
int32_t unpackInt32();
int64_t unpackInt64();
float unpackFloat32();
double unpackFloat64();
str_t unpackString5();
str_t unpackString8();
str_t unpackString16();
str_t unpackString32();
bin_t<T> unpackBinary8();
bin_t<T> unpackBinary16();
bin_t<T> unpackBinary32();
std::array<T, N> unpackBinary8();
std::array<T, N> unpackBinary16();
std::array<T, N> unpackBinary32();
size_t unpackArraySize4();
size_t unpackArraySize16();
size_t unpackArraySize32();
size_t unpackMapSize4();
size_t unpackMapSize16();
size_t unpackMapSize32();
object::ext unpackFixExt1();
object::ext unpackFixExt2();
object::ext unpackFixExt4();
object::ext unpackFixExt8();
object::ext unpackFixExt16();
object::ext unpackExt8();
object::ext unpackExt16();
object::ext unpackExt32();
object::timespec unpackTimestamp32();
object::timespec unpackTimestamp64();
object::timespec unpackTimestamp96();

// deserializer for detailed msgpack format
// these methods does NOT check index overflow and type mismatch
bool unpackNilUnchecked();
bool unpackBoolUnchecked();
uint8_t unpackUIntUnchecked7();
uint8_t unpackUIntUnchecked8();
uint16_t unpackUIntUnchecked16();
uint32_t unpackUIntUnchecked32();
uint64_t unpackUIntUnchecked64();
int8_t unpackIntUnchecked5();
int8_t unpackIntUnchecked8();
int16_t unpackIntUnchecked16();
int32_t unpackIntUnchecked32();
int64_t unpackIntUnchecked64();
float unpackFloatUnchecked32();
double unpackFloatUnchecked64();
str_t unpackStringUnchecked5();
str_t unpackStringUnchecked8();
str_t unpackStringUnchecked16();
str_t unpackStringUnchecked32();
bin_t<T> unpackBinaryUnchecked8();
bin_t<T> unpackBinaryUnchecked16();
bin_t<T> unpackBinaryUnchecked32();
std::array<T, N> unpackBinaryUnchecked8();
std::array<T, N> unpackBinaryUnchecked16();
std::array<T, N> unpackBinaryUnchecked32();
size_t unpackArraySizeUnchecked4();
size_t unpackArraySizeUnchecked16();
size_t unpackArraySizeUnchecked32();
size_t unpackMapSizeUnchecked4();
size_t unpackMapSizeUnchecked16();
size_t unpackMapSizeUnchecked32();
object::ext unpackFixExtUnchecked1();
object::ext unpackFixExtUnchecked2();
object::ext unpackFixExtUnchecked4();
object::ext unpackFixExtUnchecked8();
object::ext unpackFixExtUnchecked16();
object::ext unpackExtUnchecked8();
object::ext unpackExtUnchecked16();
object::ext unpackExtUnchecked32();
object::timespec unpackTimestampUnchecked32();
object::timespec unpackTimestampUnchecked64();
object::timespec unpackTimestampUnchecked96();

// checks types of next msgpack object
bool isNil() const;
bool isBool() const;
bool isUInt7() const;
bool isUInt8() const;
bool isUInt16() const;
bool isUInt32() const;
bool isUInt64() const;
bool isUInt() const;
bool isInt5() const;
bool isInt8() const;
bool isInt16() const;
bool isInt32() const;
bool isInt64() const;
bool isInt() const;
bool isFloat32() const;
bool isFloat64() const;
bool isFloat() const;
bool isStr5() const;
bool isStr8() const;
bool isStr16() const;
bool isStr32() const;
bool isStr() const;
bool isBin8() const;
bool isBin16() const;
bool isBin32() const;
bool isBin() const;
bool isArray4() const;
bool isArray16() const;
bool isArray32() const;
bool isArray() const;
bool isMap4() const;
bool isMap16() const;
bool isMap32() const;
bool isMap() const;
bool isFixExt1() const;
bool isFixExt2() const;
bool isFixExt4() const;
bool isFixExt8() const;
bool isFixExt16() const;
bool isFixExt() const;
bool isExt8() const;
bool isExt16() const;
bool isExt32() const;
bool isExt() const;
bool isTimestamp32() const;
bool isTimestamp64() const;
bool isTimestamp96() const;
bool isTimestamp() const;
MsgPack::Type getType() const

template <typename T>
inline size_t estimate_size(const T& msg);

namespace file {
    template <typename FsType, typename T>
    inline bool save_as_json(FsType& fs, const String& path, const T& value, JsonDocument& doc);
    template <size_t N, typename FsType, typename T>
    inline bool save_as_json_static(FsType& fs, const String& path, const T& value);
    template <typename FsType, typename T>
    inline bool save_as_json_dynamic(FsType& fs, const String& path, const T& value, const size_t json_size = 512);

    template <typename FsType, typename T>
    inline bool load_from_json(FsType& fs, const String& path, T& value, JsonDocument& doc, const size_t num_max_string_type = 32);
    template <size_t N, typename FsType, typename T>
    inline bool load_from_json_static(FsType& fs, const String& path, T& value);
    template <typename FsType, typename T>
    inline bool load_from_json_dynamic(FsType& fs, const String& path, T& value, const size_t json_size = 512);
}

namespace eeprom {
    template <typename T>
    inline bool save(const T& value, const size_t index_offset = 0);
    template <typename T>
    inline bool load(T& value, const size_t index_offset = 0);
    template <typename T>
    inline void clear(const T& value, const size_t index_offset = 0);
    inline void clear_size(const size_t size, const size_t index_offset = 0);
}

enum class Type : uint8_t {
    NA          = 0xC1, // never used
    NIL         = 0xC0,
    BOOL        = 0xC2,
    UINT7       = 0x00, // same as POSITIVE_FIXINT
    UINT8       = 0xCC,
    UINT16      = 0xCD,
    UINT32      = 0xCE,
    UINT64      = 0xCF,
    INT5        = 0xE0, // same as NEGATIVE_FIXINT
    INT8        = 0xD0,
    INT16       = 0xD1,
    INT32       = 0xD2,
    INT64       = 0xD3,
    FLOAT32     = 0xCA,
    FLOAT64     = 0xCB,
    STR5        = 0xA0, // same as FIXSTR
    STR8        = 0xD9,
    STR16       = 0xDA,
    STR32       = 0xDB,
    BIN8        = 0xC4,
    BIN16       = 0xC5,
    BIN32       = 0xC6,
    ARRAY4      = 0x90, // same as FIXARRAY
    ARRAY16     = 0xDC,
    ARRAY32     = 0xDD,
    MAP4        = 0x80, // same as FIXMAP
    MAP16       = 0xDE,
    MAP32       = 0xDF,
    FIXEXT1     = 0xD4,
    FIXEXT2     = 0xD5,
    FIXEXT4     = 0xD6,
    FIXEXT8     = 0xD7,
    FIXEXT16    = 0xD8,
    EXT8        = 0xC7,
    EXT16       = 0xC8,
    EXT32       = 0xC9,
    TIMESTAMP32 = 0xD6,
    TIMESTAMP64 = 0xD7,
    TIMESTAMP96 = 0xC7,

    POSITIVE_FIXINT = 0x00,
    NEGATIVE_FIXINT = 0xE0,
    FIXSTR          = 0xA0,
    FIXARRAY        = 0x90,
    FIXMAP          = 0x80,
};

Reference

• MessagePack Specification
• msgpack adaptor
• msgpack object
• msgpack-c wiki

License

MIT

arduino_msgpack

This Arduino library provides a light weight serializer and parser for messagepack.

Install

Download the zip, and import it with your Arduino IDE: Sketch>Include Library>Add .zip library

Usage

See the either the .h file, or the examples (led_controller and test_uno_writer).

In short:

• functions like msgpck_what_next(Stream * s); watch the next type of data without reading it (without advancing the buffer of Stream s).
• functions like msgpck_read_bool(Stream * s, bool *b) read a value from Stream s.
• functions like msgpck_write_bool(Stream * s, bool b) write a value on Stream s.

Notes:

• Stream are used as much as possible in order not to add to much overhead with buffers. Therefore you should be able to store the minimum number of value at a given time.
• Map and Array related functions concern only their headers. Ex: If you want to write an array containing two elements you should write the array header, then write the two elements.

Limitations

Currently the library does not support:

• 8 bytes float (Only 4 bytes floats are supported by default on every Arduino and floats are anyway not recommended on Arduino)
• 2^32 char long (or longer) strings
• 2^32 byte long (or longer) bins
• extention types.

CMP

CMP is a C implementation of the MessagePack serialization format. It currently implements version 5 of the MessagePack Spec.

CMP's goal is to be lightweight and straightforward, forcing nothing on the programmer.

License

While I'm a big believer in the GPL, I license CMP under the MIT license.

Example Usage

The following examples use a file as the backend, and are modeled after the examples included with the msgpack-c project.

#include <stdio.h>
#include <stdlib.h>

#include "cmp.h"

static bool read_bytes(void *data, size_t sz, FILE *fh) {
    return fread(data, sizeof(uint8_t), sz, fh) == (sz * sizeof(uint8_t));
}

static bool file_reader(cmp_ctx_t *ctx, void *data, size_t limit) {
    return read_bytes(data, limit, (FILE *)ctx->buf);
}

static bool file_skipper(cmp_ctx_t *ctx, size_t count) {
    return fseek((FILE *)ctx->buf, count, SEEK_CUR);
}

static size_t file_writer(cmp_ctx_t *ctx, const void *data, size_t count) {
    return fwrite(data, sizeof(uint8_t), count, (FILE *)ctx->buf);
}

static void error_and_exit(const char *msg) {
    fprintf(stderr, "%s\n\n", msg);
    exit(EXIT_FAILURE);
}

int main(void) {
    FILE *fh = NULL;
    cmp_ctx_t cmp = {0};
    uint32_t array_size = 0;
    uint32_t str_size = 0;
    char hello[6] = {0};
    char message_pack[12] = {0};

    fh = fopen("cmp_data.dat", "w+b");

    if (fh == NULL) {
        error_and_exit("Error opening data.dat");
    }

    cmp_init(&cmp, fh, file_reader, file_skipper, file_writer);

    if (!cmp_write_array(&cmp, 2)) {
        error_and_exit(cmp_strerror(&cmp));
    }

    if (!cmp_write_str(&cmp, "Hello", 5)) {
        error_and_exit(cmp_strerror(&cmp));
    }

    if (!cmp_write_str(&cmp, "MessagePack", 11)) {
        error_and_exit(cmp_strerror(&cmp));
    }

    rewind(fh);

    if (!cmp_read_array(&cmp, &array_size)) {
        error_and_exit(cmp_strerror(&cmp));
    }

    /* You can read the str byte size and then read str bytes... */

    if (!cmp_read_str_size(&cmp, &str_size)) {
        error_and_exit(cmp_strerror(&cmp));
    }

    if (str_size > (sizeof(hello) - 1)) {
        error_and_exit("Packed 'hello' length too long\n");
    }

    if (!read_bytes(hello, str_size, fh)) {
        error_and_exit(cmp_strerror(&cmp));
    }

    /*
     * ...or you can set the maximum number of bytes to read and do it all in
     * one call
     */

    str_size = sizeof(message_pack);
    if (!cmp_read_str(&cmp, message_pack, &str_size)) {
        error_and_exit(cmp_strerror(&cmp));
    }

    printf("Array Length: %u.\n", array_size);
    printf("[\"%s\", \"%s\"]\n", hello, message_pack);

    fclose(fh);

    return EXIT_SUCCESS;
}

Advanced Usage

See the examples folder.

Fast, Lightweight, Flexible, and Robust

CMP uses no internal buffers; conversions, encoding and decoding are done on the fly.

CMP's source and header file together are ~4k LOC.

CMP makes no heap allocations.

CMP uses standardized types rather than declaring its own, and it depends only on stdbool.h, stdint.h and string.h.

CMP is written using C89 (ANSI C), aside, of course, from its use of fixed-width integer types and bool.

On the other hand, CMP's test suite requires C99.

CMP only requires the programmer supply a read function, a write function, and an optional skip function. In this way, the programmer can use CMP on memory, files, sockets, etc.

CMP is portable. It uses fixed-width integer types, and checks the endianness of the machine at runtime before swapping bytes (MessagePack is big-endian).

CMP provides a fairly comprehensive error reporting mechanism modeled after errno and strerror.

CMP is thread aware; while contexts cannot be shared between threads, each thread may use its own context freely.

CMP is tested using the MessagePack test suite as well as a large set of custom test cases. Its small test program is compiled with clang using -Wall -Werror -Wextra ... along with several other flags, and generates no compilation errors in either clang or GCC.

CMP's source is written as readably as possible, using explicit, descriptive variable names and a consistent, clear style.

CMP's source is written to be as secure as possible. Its testing suite checks for invalid values, and data is always treated as suspect before it passes validation.

CMP's API is designed to be clear, convenient and unsurprising. Strings are null-terminated, binary data is not, error codes are clear, and so on.

CMP provides optional backwards compatibility for use with other MessagePack implementations that only implement version 4 of the spec.

Building

There is no build system for CMP. The programmer can drop cmp.c and cmp.h in their source tree and modify as necessary. No special compiler settings are required to build it, and it generates no compilation errors in either clang or gcc.

Versioning

CMP's versions are single integers. I don't use semantic versioning because I don't guarantee that any version is completely compatible with any other. In general, semantic versioning provides a false sense of security. You should be evaluating compatibility yourself, not relying on some stranger's versioning convention.

Stability

I only guarantee stability for versions released on the releases page. While rare, both master and develop branches may have errors or mismatched versions.

Backwards Compatibility

Version 4 of the MessagePack spec has no BIN type, and provides no STR8 marker. In order to remain backwards compatible with version 4 of MessagePack, do the following:

Avoid these functions:

• cmp_write_bin
• cmp_write_bin_marker
• cmp_write_str8_marker
• cmp_write_str8
• cmp_write_bin8_marker
• cmp_write_bin8
• cmp_write_bin16_marker
• cmp_write_bin16
• cmp_write_bin32_marker
• cmp_write_bin32

Use these functions in lieu of their v5 counterparts:

• cmp_write_str_marker_v4 instead of cmp_write_str_marker
• cmp_write_str_v4 instead of cmp_write_str
• cmp_write_object_v4 instead of cmp_write_object

Disabling Floating Point Operations

Thanks to tdragon it's possible to disable floating point operations in CMP by defining CMP_NO_FLOAT. No floating point functionality will be included. Fair warning: this changes the ABI.

Setting Endianness at Compile Time

CMP will honor WORDS_BIGENDIAN. If defined to 0 it will convert data to/from little-endian format when writing/reading. If defined to 1 it won't. If not defined, CMP will check at runtime.

Introduction

MPack is a C implementation of an encoder and decoder for the MessagePack serialization format. It is:

• Simple and easy to use
• Secure against untrusted data
• Lightweight, suitable for embedded
• Extensively documented
• Extremely fast

The core of MPack contains a buffered reader and writer, and a tree-style parser that decodes into a tree of dynamically typed nodes. Helper functions can be enabled to read values of expected type, to work with files, to grow buffers or allocate strings automatically, to check UTF-8 encoding, and more.

The MPack code is small enough to be embedded directly into your codebase. Simply download the amalgamation package and add mpack.h and mpack.c to your project.

MPack supports all modern compilers, all desktop and smartphone OSes, WebAssembly, inside the Linux kernel, and even 8-bit microcontrollers such as Arduino. The MPack featureset can be customized at compile-time to set which features, components and debug checks are compiled, and what dependencies are available.

Build Status

The Node API

The Node API parses a chunk of MessagePack data into an immutable tree of dynamically-typed nodes. A series of helper functions can be used to extract data of specific types from each node.

// parse a file into a node tree
mpack_tree_t tree;
mpack_tree_init_filename(&tree, "homepage-example.mp", 0);
mpack_tree_parse(&tree);
mpack_node_t root = mpack_tree_root(&tree);

// extract the example data on the msgpack homepage
bool compact = mpack_node_bool(mpack_node_map_cstr(root, "compact"));
int schema = mpack_node_i32(mpack_node_map_cstr(root, "schema"));

// clean up and check for errors
if (mpack_tree_destroy(&tree) != mpack_ok) {
    fprintf(stderr, "An error occurred decoding the data!\n");
    return;
}

Note that no additional error handling is needed in the above code. If the file is missing or corrupt, if map keys are missing or if nodes are not in the expected types, special "nil" nodes and false/zero values are returned and the tree is placed in an error state. An error check is only needed before using the data.

The above example allocates nodes automatically. A fixed node pool can be provided to the parser instead in memory-constrained environments. For maximum performance and minimal memory usage, the Expect API can be used to parse data of a predefined schema.

The Write API

The Write API encodes structured data to MessagePack.

// encode to memory buffer
char* data;
size_t size;
mpack_writer_t writer;
mpack_writer_init_growable(&writer, &data, &size);

// write the example on the msgpack homepage
mpack_build_map(&writer);
mpack_write_cstr(&writer, "compact");
mpack_write_bool(&writer, true);
mpack_write_cstr(&writer, "schema");
mpack_write_uint(&writer, 0);
mpack_complete_map(&writer);

// finish writing
if (mpack_writer_destroy(&writer) != mpack_ok) {
    fprintf(stderr, "An error occurred encoding the data!\n");
    return;
}

// use the data
do_something_with_data(data, size);
free(data);

In the above example, we encode to a growable memory buffer. The writer can instead write to a pre-allocated or stack-allocated buffer (with up-front sizes for compound types), avoiding the need for memory allocation. The writer can also be provided with a flush function (such as a file or socket write function) to call when the buffer is full or when writing is done.

If any error occurs, the writer is placed in an error state. The writer will flag an error if too much data is written, if the wrong number of elements are written, if an allocation failure occurs, if the data could not be flushed, etc. No additional error handling is needed in the above code; any subsequent writes are ignored when the writer is in an error state, so you don't need to check every write for errors.

The above example uses mpack_build_map() to automatically determine the number of key-value pairs contained. If you know up-front the number of elements needed, you can pass it to mpack_start_map() instead. In that case the corresponding mpack_finish_map() will assert in debug mode that the expected number of elements were actually written, which is something that other MessagePack C/C++ libraries may not do.

Comparison With Other Parsers

MPack is rich in features while maintaining very high performance and a small code footprint. Here's a short feature table comparing it to other C parsers:

A larger feature comparison table is available here which includes descriptions of the various entries in the table.

This benchmarking suite compares the performance of MPack to other implementations of schemaless serialization formats. MPack outperforms all JSON and MessagePack libraries (except CWPack), and in some tests MPack is several times faster than RapidJSON for equivalent data.

Why Not Just Use JSON?

Conceptually, MessagePack stores data similarly to JSON: they are both composed of simple values such as numbers and strings, stored hierarchically in maps and arrays. So why not just use JSON instead? The main reason is that JSON is designed to be human-readable, so it is not as efficient as a binary serialization format:

• Compound types such as strings, maps and arrays are delimited, so appropriate storage cannot be allocated upfront. The whole object must be parsed to determine its size.

• Strings are not stored in their native encoding. Special characters such as quotes and backslashes must be escaped when written and converted back when read.

• Numbers are particularly inefficient (especially when parsing back floats), making JSON inappropriate as a base format for structured data that contains lots of numbers.

• Binary data is not supported by JSON at all. Small binary blobs such as icons and thumbnails need to be Base64 encoded or passed out-of-band.

The above issues greatly increase the complexity of the decoder. Full-featured JSON decoders are quite large, and minimal decoders tend to leave out such features as string unescaping and float parsing, instead leaving these up to the user or platform. This can lead to hard-to-find platform-specific and locale-specific bugs, as well as a greater potential for security vulnerabilites. This also significantly decreases performance, making JSON unattractive for use in applications such as mobile games.

While the space inefficiencies of JSON can be partially mitigated through minification and compression, the performance inefficiencies cannot. More importantly, if you are minifying and compressing the data, then why use a human-readable format in the first place?

Testing MPack

The MPack build process does not build MPack into a library; it is used to build and run the unit tests. You do not need to build MPack or the unit testing suite to use MPack.

See test/README.md for information on how to test MPack.

CWPack

CWPack is a lightweight and yet complete implementation of the MessagePack serialization format version 5. It also supports the Timestamp extension type.

Excellent Performance

Together with MPack, CWPack is the fastest open-source messagepack implementation. Both totally outperform CMP and msgpack-c

Design

CWPack does no memory allocations and no file handling in its basic setup. All that is done outside of CWPack. Example extensions are included.

CWPack is working against memory buffers. User defined handlers are called when buffers are filled up (packing) or needs refill (unpack).

Containers (arrays, maps) are read/written in parts, first the item containing the size and then the contained items one by one. Exception to this is the cw_skip_items function which skip whole containers.

Example

Pack and unpack example from the MessagePack home page:

void example (void)
{
    cw_pack_context pc;
    char buffer[20];
    cw_pack_context_init (&pc, buffer, 20, 0);

    cw_pack_map_size (&pc, 2);
    cw_pack_str (&pc, "compact", 7);
    cw_pack_boolean (&pc, true);
    cw_pack_str (&pc, "schema", 6);
    cw_pack_unsigned (&pc, 0);

    if (pc.return_code != CWP_RC_OK)  ERROR;
    int length = pc.current - pc.start;
    if (length != 18) ERROR;

    cw_unpack_context uc;
    cw_unpack_context_init (&uc, pc.start, length, 0);

    if (cw_unpack_next_map_size(&uc) != 2) ERROR;
    if (cw_unpack_next_str_lengh(&uc) != 7) ERROR;
    if (strncmp("compact", uc.item.as.str.start, 7)) ERROR;
    if (cw_unpack_next_boolean(&uc) != true) ERROR;
    if (cw_unpack_next_str_lengh(&uc) != 6) ERROR;
    if (strncmp("schema", uc.item.as.str.start, 6)) ERROR;
    if (cw_unpack_next_signed32(&uc) != 0) ERROR;

    if (uc.return_code != CWP_RC_OK)  ERROR;
    cw_unpack_next(&uc);
    if (uc.return_code != CWP_RC_END_OF_INPUT)  ERROR;
}

In the examples folder there are more examples.

Backward compatibility

CWPack may be run in compatibility mode. It affects only packing; EXT & TIMESTAMP is considered illegal, BIN are transformed to STR and generation of STR8 is supressed.

Error handling

When an error is detected in a context, the context is stopped and all future calls to that context are immediatly returned without any actions. Thus it is possible to make some calls and delay error checking until all calls are done.

CWPack does not check for illegal values (e.g. in STR for illegal unicode characters).

Build

CWPack consists of a single src file and three header files. It is written in strict ansi C and the files are together ~ 1.4K lines. No separate build is neccesary, just include the files in your own build.

CWPack has no dependencies to other libraries.

Test

Included in the test folder are a module test and a performance test and shell scripts to run them.

Objective-C

CWPack also contains an Objective-C interface. The MessagePack home page example would look like:

CWPackContext *pc = [CWPackContext newWithContext:my_cw_pack_context];
[pc packObject:@{@"compact":@YES, @"schema":@0}];

CWUnpackContext *uc = [CWUnpackContext newWithContext:my_cw_unpack_context];
NSDictionary *dict = [uc unpackNextObject];

Swift

CWPack also contains a Swift interface. The MessagePack home page example would pack like:

let packer = CWDataPacker()
packer + DictionaryHeader(2) + "compact" + true + "schema" + 0
let data = packer.data

	```

MessagePack for CLI

CI Status

Configuration	Status
Release
Debug (.NET Core 2.0)
Debug (.NET Core 2.0)
Debug (.NET Framework 4.x)
Debug (Code DOM)]
Debug (miscs)]
Debug (.NET Framework 3.5)
Debug (.NET Framework 3.5 Code DOM)

What is it?

This is MessagePack serialization/deserialization for CLI (Common Language Infrastructure) implementations such as .NET Framework, Silverlight, Mono (including Moonlight.) This library can be used from ALL CLS compliant languages such as C#, F#, Visual Basic, Iron Python, Iron Ruby, PowerShell, C++/CLI or so.

Usage

You can serialize/deserialize objects as following:

1. Create serializer via MessagePackSerializer.Get generic method. This method creates dependent types serializers as well.
2. Invoke serializer as following:

• Pack method with destination Stream and target object for serialization.
• Unpack method with source Stream.

// Creates serializer.
var serializer = MessagePackSerializer.Get<T>();
// Pack obj to stream.
serializer.Pack(stream, obj);
// Unpack from stream.
var unpackedObject = serializer.Unpack(stream);

' Creates serializer.
Dim serializer = MessagePackSerializer.Get(Of T)()
' Pack obj to stream.
serializer.Pack(stream, obj)
' Unpack from stream.
Dim unpackedObject = serializer.Unpack(stream)

For production environment, you should instantiate own SerializationContext and manage its lifetime. It is good idea to treat it as singleton because SerializationContext is thread-safe.

Features

• Fast and interoperable binary format serialization with simple API.
• Generating pre-compiled assembly for rapid start up.
• Flexible MessagePackObject which represents MessagePack type system naturally.

Note: AOT support is limited yet. Use serializer pre-generation with mpu -s utility or API.
If you do not pre-generated serializers, MsgPack for CLI uses reflection in AOT environments, it is slower and it sometimes causes AOT related error (ExecutionEngineException for runtime JIT compilation). You also have to call MessagePackSerializer.PrepareType<T> and companions in advance to avoid AOT related error. See wiki for details.

Documentation

See wiki

Installation

• Binary files distributed via the NuGet package MsgPack.Cli.
• You can extract binary (DLL) file as following:
  1. Download *.zip file from GitHub Release page.
  2. Extract it.
  3. Under the bin directory, binaries are there!
  • For mono, you can use net461 or net35 drops as you run with.
  • For Unity, unity3d drop is suitable.

How to build

1. Install Visual Studio 2017 (Community edition is OK) and 2015 (for MsgPack.Windows.sln).

   • You must install .NET Framework 3.5, 4.x, .NET Core, and Xamarin dev tools to build all builds successfully. If you do not want to install options, edit <TargetFrameworks> element in *.csproj files to exclude platforms you want to exclude.

2. Install latest .NET Core SDK.

3. Run with Visual Studio Developer Command Prompt:

   msbuild MsgPack.sln /t:Restore msbuild MsgPack.sln

Or (for Unity 3D drops):

msbuild MsgPack.compats.sln /t:Restore
msbuild MsgPack.compats.sln

Or (for Windows Runtime/Phone drops and Silverlight 5 drops):

msbuild MsgPack.Windows.sln /t:Restore
msbuild MsgPack.Windows.sln

Or (for Xamarin unit testing, you must have Xamarin Business or upper license and Mac machine on the LAN to build on Windows):

msbuild MsgPack.Xamarin.sln /t:Restore
msbuild MsgPack.Xamarin.sln

Or open one of above solution files in your IDE and run build command in it.

1. Install latest Mono and .NET Core SDK.
2. Now, you can build MsgPack.sln and MsgPack.Xamarin.sln with above instructions and msbuild in latest Mono. Note that xbuild does not work because it does not support latest csproj format.

First of all, there are binary drops on github release page, you should use it to save your time.
Because we will not guarantee source code organization compatibilities, we might add/remove non-public types or members, which should break source code build.
If you want to import sources, you must include just only described on MsgPack.Unity3D.csproj.
If you want to use ".NET 2.0 Subset" settings, you must use just only described on MsgPack.Unity3D.CorLibOnly.csproj file, and define CORLIB_ONLY compiler constants.

If you run on Windows, it is recommended to use HXM instead of Hyper-V based emulator.
You can disable Hyper-V from priviledged (administrator) powershell as follows:

Disable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-Hypervisor

If you want to use Hyper-V again (such as for Docker for Windows etc.), you can do it by following in priviledged (administrator) powershell:

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V-Hypervisor

• Q: Javac shows compilation error.
  • A: Rebuild the test project and try it run again.

You must create provisoning profiles in your MacOS devices.
See Xamarin documents about provisining for details.

There are bundle IDs of current iOS tests:

• org.msgpack.msgpack-cli-xamarin-ios-test
• org.msgpack.msgpack-cli-xamarin-ios-test-packer
• org.msgpack.msgpack-cli-xamarin-ios-test-unpacker
• org.msgpack.msgpack-cli-xamarin-ios-test-unpacking
• org.msgpack.msgpack-cli-xamarin-ios-test-timestamp
• org.msgpack.msgpack-cli-xamarin-ios-test-arrayserialization
• org.msgpack.msgpack-cli-xamarin-ios-test-mapserialization

Note that some reflection based serializer tests failed with AOT related limitation.

See Xamarin's official trouble shooting docs first.

• Q: An error occurred while running unit test project.
  • A: Rebuild the project and rerun it. Or, login your Mac again, ant retry it.
• Q: It is hard to read English.
  • A: You can read localized Xamarin docs with putting {region}-{lang} as the first component of URL path such as https://developer.xamarin.com/ja-jp/guides/....

Maintenance

This solution contains Silverlight5 and (old) UWP project for backward compability. They are required Visual Studio 2015 to build and test.
You can download Visual Studio 2015 community edition from here.

You do not have to install Visual Studio 2015 as long as you don't edit/test/build Silverlight and/or old UWP project.

See also

• GitHub Page : http://cli.msgpack.org/
• Wiki (documentation) : https://github.com/msgpack/msgpack-cli/wiki
• API Reference : http://cli.msgpack.org/doc/top.html
• Issue tracker : https://github.com/msgpack/msgpack-cli/issues
• MSBuild reference : http://msdn.microsoft.com/en-us/library/0k6kkbsd.aspx
• Mono xbuild reference : http://www.mono-project.com/Microsoft.Build

SimpleMsgPack.Net

MessagePack implementation for C# / msgpack.org[C#]

Binary files distributed via the NuGet package SimpleMsgPack.

It's like JSON but small and fast.

unit Owner: D10.Mofen
contact:
       qq:185511468, 
    email:[email protected]
	homepage:www.diocp.org
if you find any bug, please contact me!

Works with

.NET Framework 4.x

    MsgPack msgpack = new MsgPack();
    msgpack.ForcePathObject("p.name").AsString = "张三";
    msgpack.ForcePathObject("p.age").AsInteger = 25;
    msgpack.ForcePathObject("p.datas").AsArray.Add(90);
    msgpack.ForcePathObject("p.datas").AsArray.Add(80);
    msgpack.ForcePathObject("p.datas").AsArray.Add("李四");
    msgpack.ForcePathObject("p.datas").AsArray.Add(3.1415926);

    // pack file
    msgpack.ForcePathObject("p.filedata").LoadFileAsBytes("C:\\a.png");

    // pack msgPack binary
    byte[] packData = msgpack.Encode2Bytes();

    MsgPack unpack_msgpack = new MsgPack();
	
    // unpack msgpack
    unpack_msgpack.DecodeFromBytes(packData);

    System.Console.WriteLine("name:{0}, age:{1}",
          unpack_msgpack.ForcePathObject("p.name").AsString,
          unpack_msgpack.ForcePathObject("p.age").AsInteger);

    Console.WriteLine("==================================");
    System.Console.WriteLine("use index property, Length{0}:{1}",
          unpack_msgpack.ForcePathObject("p.datas").AsArray.Length,
          unpack_msgpack.ForcePathObject("p.datas").AsArray[0].AsString
          );

    Console.WriteLine("==================================");
    Console.WriteLine("use foreach statement:");
    foreach (MsgPack item in unpack_msgpack.ForcePathObject("p.datas"))
    {
        Console.WriteLine(item.AsString);
    }

    // unpack filedata 
    unpack_msgpack.ForcePathObject("p.filedata").SaveBytesToFile("C:\\b.png");
    Console.Read();

MPack

This library is a lightweight implementation of the MessagePack binary serialization format. MessagePack is a 1-to-1 binary representation of JSON, and the official specification can be found here: https://github.com/msgpack....

(https://ci.appveyor.com/project/caesay/mpack)

Notes

• This library is designed to be super light weight.
• Its easiest to understand how this library works if you think in terms of json. The type MPackMap represents a dictionary, and the type MPackArray represents an array.
• Create MPack instances with the static method MPack.From(object);. You can pass any simple type (such as string, integer, etc), or any Array composed of a simple type. MPack also has implicit conversions from most of the basic types built in.
• Transform an MPack object back into a CLR type with the static method MPack.To<T>(); or MPack.To(type);. MPack also has explicit converions going back to most basic types, you can do string str = (string)mpack; for instance.
• MPack now supports native asynchrounous reading and cancellation tokens. It will not block a thread to wait on a stream.

NuGet

MPack is available as a NuGet package!

PM> Install-Package MPack

Usage

Create a object model that can be represented as MsgPack. Here we are creating a dictionary, but really it can be anything:

MPackMap dictionary = new MPackMap
{
    {
        "array1", MPack.From(new[]
        {
            "array1_value1",  // implicitly converted string
            MPack.From("array1_value2"),
            MPack.FromString("array1_value3"),
        })
    },
    {"bool1", MPack.From(true)}, //boolean
    {"double1", MPack.From(50.5)}, //single-precision float
    {"double2", MPack.From(15.2)},
    {"int1", 50505}, // implicitly converted integer
    {"int2", MPack.From(50)} // integer
};

Serialize the data to a byte array or to a stream to be saved, transmitted, etc:

byte[] encodedBytes = dictionary.EncodeToBytes();
// -- or --
dictionary.EncodeToStream(stream);

Parse the binary data back into a MPack object model (you can also cast back to an MPackMap or MPackArray after reading if you want dictionary/array methods):

var reconstructed = MPack.ParseFromBytes(encodedBytes);
// -- or --
var reconstructed = MPack.ParseFromStream(stream);

Turn MPack objects back into types that we understand with the generic To<>() method. Since we know the types of everything here we can just call To<bool>() to reconstruct our bool, but if you don't know you can access the instance enum MPack.ValueType to know what kind of value it is:

bool bool1 = reconstructed["bool1"].To<bool>();
var array1 = reconstructed["array1"] as MPackArray;
var array1_value1 = array1[0];
double double1 = reconstructed["double1"].To<double>();
//etc...

Credits

The following people/projects have made this possible:

0. Me: [caelantsayler]at[gmail]dot[com]
1. All of the people that make MessagePack happen: https://github.com/msgpack

LsMsgPack

MsgPack debugging and validation tool also usable as Fiddler plugin

More info about this application (and screenshots) can be found at: http://www.infotopie.nl/open-source/msgpack-explorer

Library Usage Example

Although the original was optimised for debugging and analysing, a lightweight version of the lib is included which does not keep track of all offsets and other overhead needed for debugging. It can be used in your code.

Add LsMsgPackL.dll as a reference.

public class MyClass
{
    public string Name { get; set; }
    public int Quantity { get; set; }
    public List<object> Anything { get; set; }
}

public void Test()
{
    MyClass message = new MyClass()
    {
        Name = "Test message",
        Quantity = 100,
        Anything = new List<object>(new object[] { "first", 2, false, null, 4.2d, "last" })
    };
    
    // Serialize
    byte[] buffer = MsgPackSerializer.Serialize(message);
    
    // Deserialize
    MyClass returnMsg = MsgPackSerializer.Deserialize<MyClass>(buffer);
}

I have never run any benchmarks so I have no idea how it will perform against other implementations.

Fiddler Integration

In order to use this tool as a Fiddler plugin, copy the following files to the Fiddler Inspectors directory (usually C:\Program Files\Fiddler2\Inspectors):

• MsgPackExplorer.exe
• LsMsgPackFiddlerInspector.dll
• LsMsgPack.dll

Restart fiddler and you should see a MsgPack option in the Inspectors list.

Source documentation

This module contains the "parser" and generator of MsgPack Packages. It breaks down the binary file into a hirarchical structure, keeping track of offsets and errors. And it can also be used to generate MsgPack files.

The main winforms executable, containing a MsgPackExplorer UserControl (so it can easily be integrated into other tools such as Fiddler).

A tiny wrapper enabling the use of MsgPack Explorer as a Fiddler Inspector.

Some unit tests on the core LsMsgPack.dll. No full coverage yet, but at least it's a start.

A light version of the "parser". The parsing and generating methods are almost identical to the LsMsgPack lib, but with allot of overhead removed that comes with keeping track of offsets, original types and other debugging info. I'm planning to use this version in my projects that use the MsgPack format. The LightUnitTests are the same as LsMsgPackUnitTests with some tests omitted.

MessagePack for C# (.NET, .NET Core, Unity, Xamarin)

The extremely fast MessagePack serializer for C#. It is 10x faster than MsgPack-Cli and outperforms other C# serializers. MessagePack for C# also ships with built-in support for LZ4 compression - an extremely fast compression algorithm. Performance is important, particularly in applications like games, distributed computing, microservices, or data caches.

MessagePack has a compact binary size and a full set of general purpose expressive data types. Please have a look at the comparison with JSON, protobuf, ZeroFormatter section and learn why MessagePack C# is the fastest.

Table of Contents

• Installation
  • NuGet packages
  • Unity
  • Migration notes from v1.x
• Quick Start
• Analyzer
• Built-in supported types
• Object Serialization
• DataContract compatibility
• Serializing readonly/immutable object members (SerializationConstructor)
• Serialization Callback
• Union
• Dynamic (Untyped) Deserialization
• Object Type Serialization
• Typeless
• Security
• Performance
  • Deserialization Performance for different options
  • String interning
• LZ4 Compression
  • Attributions
• Comparison with protobuf, JSON, ZeroFormatter
• Hints to achieve maximum performance when using MessagePack for C#
  • Use indexed keys instead of string keys (Contractless)
  • Create own custom composite resolver
  • Use native resolvers
  • Be careful when copying buffers
  • Choosing compression
• Extensions
• Experimental Features
• High-Level API (MessagePackSerializer)
  • Multiple MessagePack structures on a single Stream
• Low-Level API (IMessagePackFormatter<T>)
• Primitive API (MessagePackWriter, MessagePackReader)
  • MessagePackReader
  • MessagePackWriter
• Main Extension Point (IFormatterResolver)
• MessagePackFormatterAttribute
• IgnoreFormatter
• Reserved Extension Types
• Unity support
• AOT Code Generation (support for Unity/Xamarin)
• RPC
  • MagicOnion
  • StreamJsonRpc
• How to build
• Author Info

Installation

This library is distributed via NuGet. Special Unity support is available, too.

We target .NET Standard 2.0 with special optimizations for .NET Core 2.1+, making it compatible with most reasonably recent .NET runtimes such as Core 2.0 and later, Framework 4.6.1 and later, Mono 5.4 and later and Unity 2018.3 and later. The library code is pure C# (with Just-In-Time IL code generation on some platforms).

To install with NuGet, just install the MessagePack package:

Install-Package MessagePack

Install the optional C# analyzers package to get warnings about coding mistakes and automatic fix suggestions to save you time:

Install-Package MessagePackAnalyzer

There are also a range of official and third party Extension Packages available (learn more in our extensions section):

Install-Package MessagePack.ReactiveProperty
Install-Package MessagePack.UnityShims
Install-Package MessagePack.AspNetCoreMvcFormatter

For Unity projects, the releases page provides downloadable .unitypackage files. When using in Unity IL2CPP or Xamarin AOT environments, please carefully read the pre-code generation section.

If you were using MessagePack for C# v1.x, check out the "How to update to our new v2.x version" document.

Quick Start

Define the struct or class to be serialized and annotate it with a [MessagePackObject] attribute. Annotate members whose values should be serialized (fields as well as properties) with [Key] attributes.

[MessagePackObject]
public class MyClass
{
    // Key attributes take a serialization index (or string name)
    // The values must be unique and versioning has to be considered as well.
    // Keys are described in later sections in more detail.
    [Key(0)]
    public int Age { get; set; }

    [Key(1)]
    public string FirstName { get; set; }

    [Key(2)]
    public string LastName { get; set; }

    // All fields or properties that should not be serialized must be annotated with [IgnoreMember].
    [IgnoreMember]
    public string FullName { get { return FirstName + LastName; } }
}

Call MessagePackSerializer.Serialize<T>/Deserialize<T> to serialize/deserialize your object instance. You can use the ConvertToJson method to get a human readable representation of any MessagePack binary blob.

class Program
{
    static void Main(string[] args)
    {
        var mc = new MyClass
        {
            Age = 99,
            FirstName = "hoge",
            LastName = "huga",
        };

        // Call Serialize/Deserialize, that's all.
        byte[] bytes = MessagePackSerializer.Serialize(mc);
        MyClass mc2 = MessagePackSerializer.Deserialize<MyClass>(bytes);

        // You can dump MessagePack binary blobs to human readable json.
        // Using indexed keys (as opposed to string keys) will serialize to MessagePack arrays,
        // hence property names are not available.
        // [99,"hoge","huga"]
        var json = MessagePackSerializer.ConvertToJson(bytes);
        Console.WriteLine(json);
    }
}

By default, a MessagePackObject annotation is required. This can be made optional; see the Object Serialization section and the Formatter Resolver section for details.

Analyzer

The MessagePackAnalyzer package aids with:

1. Automating definitions for your serializable objects.
2. Produces compiler warnings upon incorrect attribute use, member accessibility, and more.

If you want to allow a specific custom type (for example, when registering a custom type), put MessagePackAnalyzer.json at the project root and change the Build Action to AdditionalFiles.

An example MessagePackAnalyzer.json:

[ "MyNamespace.FooClass", "MyNameSpace.BarStruct" ]

Built-in supported types

These types can serialize by default:

• Primitives (int, string, etc...), Enums, Nullable<>, Lazy<>
• TimeSpan, DateTime, DateTimeOffset
• Guid, Uri, Version, StringBuilder
• BigInteger, Complex, Half
• Array[], Array[,], Array[,,], Array[,,,], ArraySegment<>, BitArray
• KeyValuePair<,>, Tuple<,...>, ValueTuple<,...>
• ArrayList, Hashtable
• List<>, LinkedList<>, Queue<>, Stack<>, HashSet<>, ReadOnlyCollection<>, SortedList<,>
• IList<>, ICollection<>, IEnumerable<>, IReadOnlyCollection<>, IReadOnlyList<>
• Dictionary<,>, IDictionary<,>, SortedDictionary<,>, ILookup<,>, IGrouping<,>, ReadOnlyDictionary<,>, IReadOnlyDictionary<,>
• ObservableCollection<>, ReadOnlyObservableCollection<>
• ISet<>,
• ConcurrentBag<>, ConcurrentQueue<>, ConcurrentStack<>, ConcurrentDictionary<,>
• Immutable collections (ImmutableList<>, etc)
• Custom implementations of ICollection<> or IDictionary<,> with a parameterless constructor
• Custom implementations of IList or IDictionary with a parameterless constructor

You can add support for custom types, and there are some official/third-party extension packages for:

• ReactiveProperty
• for Unity (Vector3, Quaternion, etc...)
• F# (Record, FsList, Discriminated Unions, etc...)

Please see the extensions section.

MessagePack.Nil is the built-in type representing null/void in MessagePack for C#.

Object Serialization

MessagePack for C# can serialize your own public class or struct types. By default, serializable types must be annotated with the [MessagePackObject] attribute and members with the [Key] attribute. Keys can be either indexes (int) or arbitrary strings. If all keys are indexes, arrays are used for serialization, which offers advantages in performance and binary size. Otherwise, MessagePack maps (dictionaries) will be used.

If you use [MessagePackObject(keyAsPropertyName: true)], then members do not require explicit Key attributes, but string keys will be used.

[MessagePackObject]
public class Sample1
{
    [Key(0)]
    public int Foo { get; set; }
    [Key(1)]
    public int Bar { get; set; }
}

[MessagePackObject]
public class Sample2
{
    [Key("foo")]
    public int Foo { get; set; }
    [Key("bar")]
    public int Bar { get; set; }
}

[MessagePackObject(keyAsPropertyName: true)]
public class Sample3
{
    // No need for a Key attribute
    public int Foo { get; set; }

    // If want to ignore a public member, you can use the  IgnoreMember attribute
    [IgnoreMember]
    public int Bar { get; set; }
}

// [10,20]
Console.WriteLine(MessagePackSerializer.SerializeToJson(new Sample1 { Foo = 10, Bar = 20 }));

// {"foo":10,"bar":20}
Console.WriteLine(MessagePackSerializer.SerializeToJson(new Sample2 { Foo = 10, Bar = 20 }));

// {"Foo":10}
Console.WriteLine(MessagePackSerializer.SerializeToJson(new Sample3 { Foo = 10, Bar = 20 }));

All public instance members (fields as well as properties) will be serialized. If you want to ignore certain public members, annotate the member with a [IgnoreMember] attribute.

Please note that any serializable struct or class must have public accessibility; private and internal structs and classes cannot be serialized! The default of requiring MessagePackObject annotations is meant to enforce explicitness and therefore may help write more robust code.

Should you use an indexed (int) key or a string key? We recommend using indexed keys for faster serialization and a more compact binary representation than string keys. However, the additional information in the strings of string keys can be quite useful when debugging.

When classes change or are extended, be careful about versioning. MessagePackSerializer will initialize members to their default value if a key does not exist in the serialized binary blob, meaning members using reference types can be initialized to null. If you use indexed (int) keys, the keys should start at 0 and should be sequential. If a later version stops using certain members, you should keep the obsolete members (C# provides an Obsolete attribute to annotate such members) until all other clients had a chance to update and remove their uses of these members as well. Also, when the values of indexed keys "jump" a lot, leaving gaps in the sequence, it will negatively affect the binary size, as null placeholders will be inserted into the resulting arrays. However, you shouldn't reuse indexes of removed members to avoid compatibility issues between clients or when trying to deserialize legacy blobs.

Example of index gaps and resulting placeholders:

[MessagePackObject]
public class IntKeySample
{
    [Key(3)]
    public int A { get; set; }
    [Key(10)]
    public int B { get; set; }
}

// [null,null,null,0,null,null,null,null,null,null,0]
Console.WriteLine(MessagePackSerializer.SerializeToJson(new IntKeySample()));

If you do not want to explicitly annotate with the MessagePackObject/Key attributes and instead want to use MessagePack for C# more like e.g. Json.NET, you can make use of the contractless resolver.

public class ContractlessSample
{
    public int MyProperty1 { get; set; }
    public int MyProperty2 { get; set; }
}

var data = new ContractlessSample { MyProperty1 = 99, MyProperty2 = 9999 };
var bin = MessagePackSerializer.Serialize(
  data,
  MessagePack.Resolvers.ContractlessStandardResolver.Options);

// {"MyProperty1":99,"MyProperty2":9999}
Console.WriteLine(MessagePackSerializer.ConvertToJson(bin));

// You can also set ContractlessStandardResolver as the default.
// (Global state; Not recommended when writing library code)
MessagePackSerializer.DefaultOptions = MessagePack.Resolvers.ContractlessStandardResolver.Options;

// Now serializable...
var bin2 = MessagePackSerializer.Serialize(data);

If you want to serialize private members as well, you can use one of the *AllowPrivate resolvers.

[MessagePackObject]
public class PrivateSample
{
    [Key(0)]
    int x;

    public void SetX(int v)
    {
        x = v;
    }

    public int GetX()
    {
        return x;
    }
}

var data = new PrivateSample();
data.SetX(9999);

// You can choose either StandardResolverAllowPrivate
// or ContractlessStandardResolverAllowPrivate
var bin = MessagePackSerializer.Serialize(
  data,
  MessagePack.Resolvers.DynamicObjectResolverAllowPrivate.Options);

If you want to use MessagePack for C# more like a BinaryFormatter with a typeless serialization API, use the typeless resolver and helpers. Please consult the Typeless section.

Resolvers are the way to add specialized support for custom types to MessagePack for C#. Please refer to the Extension point section.

DataContract compatibility

You can use [DataContract] annotations instead of [MessagePackObject] ones. If type is annotated with DataContract, you can use [DataMember] annotations instead of [Key] ones and [IgnoreDataMember] instead of [IgnoreMember].

Then [DataMember(Order = int)] will behave the same as [Key(int)], [DataMember(Name = string)] the same as [Key(string)], and [DataMember] the same as [Key(nameof(member name)].

Using DataContract, e.g. in shared libraries, makes your classes/structs independent from MessagePack for C# serialization. However, it is not supported by the analyzers nor in code generation by the mpc tool. Also, features like UnionAttribute, MessagePackFormatter, SerializationConstructor, etc can not be used. Due to this, we recommend that you use the specific MessagePack for C# annotations when possible.

Serializing readonly/immutable object members (SerializationConstructor)

MessagePack for C# supports serialization of readonly/immutable objects/members. For example, this struct can be serialized and deserialized.

[MessagePackObject]
public struct Point
{
    [Key(0)]
    public readonly int X;
    [Key(1)]
    public readonly int Y;

    public Point(int x, int y)
    {
        this.X = x;
        this.Y = y;
    }
}

var data = new Point(99, 9999);
var bin = MessagePackSerializer.Serialize(data);

// Okay to deserialize immutable object
var point = MessagePackSerializer.Deserialize<Point>(bin);

MessagePackSerializer will choose the constructor with the best matched argument list, using argument indexes index for index keys, or parameter names for string keys. If it cannot determine an appropriate constructor, a MessagePackDynamicObjectResolverException: can't find matched constructor parameter exception will be thrown. You can specify which constructor to use manually with a [SerializationConstructor] annotation.

[MessagePackObject]
public struct Point
{
    [Key(0)]
    public readonly int X;
    [Key(1)]
    public readonly int Y;

    [SerializationConstructor]
    public Point(int x)
    {
        this.X = x;
        this.Y = -1;
    }

    // If not marked attribute, used this(most matched argument)
    public Point(int x, int y)
    {
        this.X = x;
        this.Y = y;
    }
}

C# 9.0 record with primary constructor is similar immutable object, also supports serialize/deserialize.

// use key as property name
[MessagePackObject(true)]public record Point(int X, int Y);

// use property: to set KeyAttribute
[MessagePackObject] public record Point([property:Key(0)] int X, [property: Key(1)] int Y);

// Or use explicit properties
[MessagePackObject]
public record Person
{
    [Key(0)]
    public string FirstName { get; init; }

    [Key(1)]
    public string LastName { get; init; }
}

When using init property setters in generic classes, a CLR bug prevents our most efficient code generation from invoking the property setter. As a result, you should avoid using init on property setters in generic classes when using the public-only DynamicObjectResolver/StandardResolver.

When using the DynamicObjectResolverAllowPrivate/StandardResolverAllowPrivate resolver the bug does not apply and you may use init without restriction.

Serialization Callback

Objects implementing the IMessagePackSerializationCallbackReceiver interface will received OnBeforeSerialize and OnAfterDeserialize calls during serialization/deserialization.

[MessagePackObject]
public class SampleCallback : IMessagePackSerializationCallbackReceiver
{
    [Key(0)]
    public int Key { get; set; }

    public void OnBeforeSerialize()
    {
        Console.WriteLine("OnBefore");
    }

    public void OnAfterDeserialize()
    {
        Console.WriteLine("OnAfter");
    }
}

Union

MessagePack for C# supports serializing interface-typed and abstract class-typed objects. It behaves like XmlInclude or ProtoInclude. In MessagePack for C# these are called Union. Only interfaces and abstracts classes are allowed to be annotated with Union attributes. Unique union keys are required.

// Annotate inheritance types
[MessagePack.Union(0, typeof(FooClass))]
[MessagePack.Union(1, typeof(BarClass))]
public interface IUnionSample
{
}

[MessagePackObject]
public class FooClass : IUnionSample
{
    [Key(0)]
    public int XYZ { get; set; }
}

[MessagePackObject]
public class BarClass : IUnionSample
{
    [Key(0)]
    public string OPQ { get; set; }
}

// ---

IUnionSample data = new FooClass() { XYZ = 999 };

// Serialize interface-typed object.
var bin = MessagePackSerializer.Serialize(data);

// Deserialize again.
var reData = MessagePackSerializer.Deserialize<IUnionSample>(bin);

// Use with e.g. type-switching in C# 7.0
switch (reData)
{
    case FooClass x:
        Console.WriteLine(x.XYZ);
        break;
    case BarClass x:
        Console.WriteLine(x.OPQ);
        break;
    default:
        break;
}

Unions are internally serialized to two-element arrays.

IUnionSample data = new BarClass { OPQ = "FooBar" };

var bin = MessagePackSerializer.Serialize(data);

// Union is serialized to two-length array, [key, object]
// [1,["FooBar"]]
Console.WriteLine(MessagePackSerializer.ConvertToJson(bin));

Using Union with abstract classes works the same way.

[Union(0, typeof(SubUnionType1))]
[Union(1, typeof(SubUnionType2))]
[MessagePackObject]
public abstract class ParentUnionType
{
    [Key(0)]
    public int MyProperty { get; set; }
}

[MessagePackObject]
public class SubUnionType1 : ParentUnionType
{
    [Key(1)]
    public int MyProperty1 { get; set; }
}

[MessagePackObject]
public class SubUnionType2 : ParentUnionType
{
    [Key(1)]
    public int MyProperty2 { get; set; }
}

Please be mindful that you cannot reuse the same keys in derived types that are already present in the parent type, as internally a single flat array or map will be used and thus cannot have duplicate indexes/keys.

Dynamic (Untyped) Deserialization

When calling MessagePackSerializer.Deserialize<object> or MessagePackSerializer.Deserialize<dynamic>, any values present in the blob will be converted to primitive values, i.e. bool, char, sbyte, byte, short, int, long, ushort, uint, ulong, float, double, DateTime, string, byte[], object[], IDictionary<object, object>.

// Sample blob.
var model = new DynamicModel { Name = "foobar", Items = new[] { 1, 10, 100, 1000 } };
var blob = MessagePackSerializer.Serialize(model, ContractlessStandardResolver.Options);

// Dynamic ("untyped")
var dynamicModel = MessagePackSerializer.Deserialize<dynamic>(blob, ContractlessStandardResolver.Options);

// You can access the data using array/dictionary indexers, as shown above
Console.WriteLine(dynamicModel["Name"]); // foobar
Console.WriteLine(dynamicModel["Items"][2]); // 100

Exploring object trees using the dictionary indexer syntax is the fastest option for untyped deserialization, but it is tedious to read and write. Where performance is not as important as code readability, consider deserializing with ExpandoObject.

Object Type Serialization

StandardResolver and ContractlessStandardResolver can serialize object/anonymous typed objects.

var objects = new object[] { 1, "aaa", new ObjectFieldType { Anything = 9999 } };
var bin = MessagePackSerializer.Serialize(objects);

// [1,"aaa",[9999]]
Console.WriteLine(MessagePackSerializer.ConvertToJson(bin));

// Support anonymous Type Serialize
var anonType = new { Foo = 100, Bar = "foobar" };
var bin2 = MessagePackSerializer.Serialize(anonType, MessagePack.Resolvers.ContractlessStandardResolver.Options);

// {"Foo":100,"Bar":"foobar"}
Console.WriteLine(MessagePackSerializer.ConvertToJson(bin2));

Unity supports is limited.

When deserializing, the behavior will be the same as Dynamic (Untyped) Deserialization.

Typeless

The typeless API is similar to BinaryFormatter, as it will embed type information into the blobs, so no types need to be specified explicitly when calling the API.

object mc = new Sandbox.MyClass()
{
    Age = 10,
    FirstName = "hoge",
    LastName = "huga"
};

// Serialize with the typeless API
var blob = MessagePackSerializer.Typeless.Serialize(mc);

// Blob has embedded type-assembly information.
// ["Sandbox.MyClass, Sandbox",10,"hoge","huga"]
Console.WriteLine(MessagePackSerializer.ConvertToJson(bin));

// You can deserialize to MyClass again with the typeless API
// Note that no type has to be specified explicitly in the Deserialize call
// as type information is embedded in the binary blob
var objModel = MessagePackSerializer.Typeless.Deserialize(bin) as MyClass;

Type information is represented by the MessagePack ext format, type code 100.

MessagePackSerializer.Typeless is a shortcut of Serialize/Deserialize<object>(TypelessContractlessStandardResolver.Instance). If you want to configure it as the default resolver, you can use MessagePackSerializer.Typeless.RegisterDefaultResolver.

TypelessFormatter can used standalone or combined with other resolvers.

// Replaced `object` uses the typeless resolver
var resolver = MessagePack.Resolvers.CompositeResolver.Create(
    new[] { MessagePack.Formatters.TypelessFormatter.Instance },
    new[] { MessagePack.Resolvers.StandardResolver.Instance });

public class Foo
{
    // use Typeless(this field only)
    [MessagePackFormatter(typeof(TypelessFormatter))]
    public object Bar;
}

If a type's name is changed later, you can no longer deserialize old blobs. But you can specify a fallback name in such cases, providing a TypelessFormatter.BindToType function of your own.

MessagePack.Formatters.TypelessFormatter.BindToType = typeName =>
{
    if (typeName.StartsWith("SomeNamespace"))
    {
        typeName = typeName.Replace("SomeNamespace", "AnotherNamespace");
    }

    return Type.GetType(typeName, false);
};

Security

Deserializing data from an untrusted source can introduce security vulnerabilities in your application. Depending on the settings used during deserialization, untrusted data may be able to execute arbitrary code or cause a denial of service attack. Untrusted data might come from over the network from an untrusted source (e.g. any and every networked client) or can be tampered with by an intermediary when transmitted over an unauthenticated connection, or from a local storage that might have been tampered with, or many other sources. MessagePack for C# does not provide any means to authenticate data or make it tamper-resistant. Please use an appropriate method of authenticating data before deserialization - such as a MAC .

Please be very mindful of these attack scenarios; many projects and companies, and serialization library users in general, have been bitten by untrusted user data deserialization in the past.

When deserializing untrusted data, put MessagePack into a more secure mode by configuring your MessagePackSerializerOptions.Security property:

var options = MessagePackSerializerOptions.Standard
    .WithSecurity(MessagePackSecurity.UntrustedData);

// Pass the options explicitly for the greatest control.
T object = MessagePackSerializer.Deserialize<T>(data, options);

// Or set the security level as the default.
MessagePackSerializer.DefaultOptions = options;

You should also avoid the Typeless serializer/formatters/resolvers for untrusted data as that opens the door for the untrusted data to potentially deserialize unanticipated types that can compromise security.

The UntrustedData mode merely hardens against some common attacks, but is no fully secure solution in itself.

Performance

Benchmarks comparing MessagePack For C# to other serializers were run on Windows 10 Pro x64 Intel Core i7-6700K 4.00GHz, 32GB RAM. Benchmark code is available here - and their version info. ZeroFormatter and FlatBuffers have infinitely fast deserializers, so ignore their deserialization performance.

MessagePack for C# uses many techniques to improve performance.

• The serializer uses IBufferWriter<byte> rather than System.IO.Stream to reduce memory overhead.
• Buffers are rented from pools to reduce allocations, keeping throughput high through reduced GC pressure.
• Don't create intermediate utility instances (*Writer/*Reader, *Context, etc...)
• Utilize dynamic code generation and JIT to avoid boxing value types. Use AOT generation on platforms that prohibit JITs.
• Cached generated formatters on static generic fields (don't use dictionary-cache because dictionary lookup is overhead). See Resolvers
• Heavily tuned dynamic IL code generation and JIT to avoid boxing value types. See DynamicObjectTypeBuilder. Use AOT generation on platforms that prohibit JIT.
• Call the Primitive API directly when IL code generation determines target types to be primitive.
• Reduce branching of variable length formats when IL code generation knows the target type (integer/string) ranges
• Don't use the IEnumerable<T> abstraction to iterate over collections when possible, see: CollectionFormatterBase and derived collection formatters
• Use pre-generated lookup tables to reduce checks of mgpack type constraints, see: MessagePackBinary
• Uses optimized type key dictionary for non-generic methods, see: ThreadsafeTypeKeyHashTable
• Avoid string key decoding for lookup maps (string key and use automata based name lookup with inlined IL code generation, see: AutomataDictionary
• To encode string keys, use pre-generated member name bytes and fixed sized byte array copies in IL, see: UnsafeMemory.cs

Before creating this library, I implemented a fast serializer with ZeroFormatter#Performance. This is a further evolved implementation. MessagePack for C# is always fast and optimized for all types (primitive, small struct, large object, any collections).

Performance varies depending on the options used. This is a micro benchmark with BenchmarkDotNet. The target object has 9 members (MyProperty1 ~ MyProperty9), values are zero.

ÌntKey, StringKey, Typeless_IntKey, Typeless_StringKey are MessagePack for C# options. All MessagePack for C# options achieve zero memory allocations in the deserialization process. JsonNetString/JilString is deserialized from strings. JsonNetStreamReader/JilStreamReader is deserialized from UTF-8 byte arrays using StreamReader. Deserialization is normally read from Stream. Thus, it will be restored from byte arrays (or Stream) instead of strings.

MessagePack for C# IntKey is the fastest. StringKey is slower than IntKey because matching the character string of property names is required. IntKey works by reading the array length, then for (array length) { binary decode }. StringKey works by reading map length, for (map length) { decode key, lookup key, binary decode }, so it requires an additional two steps (decoding of keys and lookups of keys).

String key is often a useful, contractless, simple replacement of JSON, interoperability with other languages, and more robust versioning. MessagePack for C# is also optimized for string keys as much a possible. First of all, it does not decode UTF-8 byte arrays to full string for matching with the member name; instead it will look up the byte arrays as it is (to avoid decoding costs and extra memory allocations).

And It will try to match each long type (per 8 character, if it is not enough, pad with 0) using automata and inline it when generating IL code.

This also avoids calculating the hash code of byte arrays, and the comparison can be made several times faster using the long type.

This is the sample of decompiled generated deserializer code, decompiled using ILSpy.

If the number of nodes is large, searches will use an embedded binary search.

Extra note, this is serialization benchmark result.

Of course, IntKey is fastest but StringKey also performs reasonably well.

The msgpack format does not provide for reusing strings in the data stream. This naturally leads the deserializer to create a new string object for every string encountered, even if it is equal to another string previously encountered.

When deserializing data that may contain the same strings repeatedly it can be worthwhile to have the deserializer take a little extra time to check whether it has seen a given string before and reuse it if it has.

To enable string interning on all string values, use a resolver that specifies StringInterningFormatter before any of the standard ones, like this:

var options = MessagePackSerializerOptions.Standard.WithResolver(
    CompositeResolver.Create(
        new IMessagePackFormatter[] { new StringInterningFormatter() },
        new IFormatterResolver[] { StandardResolver.Instance }));

MessagePackSerializer.Deserialize<ClassOfStrings>(data, options);

If you know which fields of a particular type are likely to contain duplicate strings, you can apply the string interning formatter to just those fields so the deserializer only pays for the interned string check where it matters most. Note that this technique requires a [MessagePackObject] or [DataContract] class.

[MessagePackObject]
public class ClassOfStrings
{
    [Key(0)]
    [MessagePackFormatter(typeof(StringInterningFormatter))]
    public string InternedString { get; set; }

    [Key(1)]
    public string OrdinaryString { get; set; }
}

If you are writing your own formatter for some type that contains strings, you can call on the StringInterningFormatter directly from your formatter as well for the strings.

LZ4 Compression

MessagePack is a fast and compact format but it is not compression. LZ4 is an extremely fast compression algorithm, and using it MessagePack for C# can achieve extremely fast performance as well as extremely compact binary sizes!

MessagePack for C# has built-in LZ4 support. You can activate it using a modified options object and passing it into an API like this:

var lz4Options = MessagePackSerializerOptions.Standard.WithCompression(MessagePackCompression.Lz4BlockArray);
MessagePackSerializer.Serialize(obj, lz4Options);

MessagePackCompression has two modes, Lz4Block and Lz4BlockArray. Neither is a simple binary LZ4 compression, but a special compression integrated into the serialization pipeline, using MessagePack ext code (Lz4BlockArray (98) or Lz4Block (99)). Therefore, it is not readily compatible with compression offered in other languages.

Lz4Block compresses an entire MessagePack sequence as a single LZ4 block. This is the simple compression that achieves best compression ratio, at the cost of copying the entire sequence when necessary to get contiguous memory.

Lz4BlockArray compresses an entire MessagePack sequence as a array of LZ4 blocks. Compressed/decompressed blocks are chunked and thus do not enter the GC's Large-Object-Heap, but the compression ratio is slightly worse.

We recommend to use Lz4BlockArray as the default when using compression. For compatibility with MessagePack v1.x, use Lz4Block.

Regardless of which LZ4 option is set at the deserialization, both methods can be deserialized. For example, when the Lz4BlockArray option was used, binary data using either Lz4Block and Lz4BlockArray can be deserialized. Neither can be decompressed and hence deserialized when the compression option is set to None.

LZ4 compression support is using Milosz Krajewski's lz4net code with some modifications.

Comparison with protobuf, JSON, ZeroFormatter

protobuf-net is major, widely used binary-format library on .NET. I love protobuf-net and respect their great work. But when you use protobuf-net as a general purpose serialization format, you may encounter an annoying issue.

[ProtoContract]
public class Parent
{
    [ProtoMember(1)]
    public int Primitive { get; set; }
    [ProtoMember(2)]
    public Child Prop { get; set; }
    [ProtoMember(3)]
    public int[] Array { get; set; }
}

[ProtoContract]
public class Child
{
    [ProtoMember(1)]
    public int Number { get; set; }
}

using (var ms = new MemoryStream())
{
    // serialize null.
    ProtoBuf.Serializer.Serialize<Parent>(ms, null);

    ms.Position = 0;
    var result = ProtoBuf.Serializer.Deserialize<Parent>(ms);

    Console.WriteLine(result != null); // True, not null. but all property are zero formatted.
    Console.WriteLine(result.Primitive); // 0
    Console.WriteLine(result.Prop); // null
    Console.WriteLine(result.Array); // null
}

using (var ms = new MemoryStream())
{
    // serialize empty array.
    ProtoBuf.Serializer.Serialize<Parent>(ms, new Parent { Array = System.Array.Empty<int>() });

    ms.Position = 0;
    var result = ProtoBuf.Serializer.Deserialize<Parent>(ms);

    Console.WriteLine(result.Array == null); // True, null!
}

protobuf(-net) cannot handle null and empty collection correctly, because protobuf has no null representation (see this SO answer from a protobuf-net author).

MessagePack's type system can correctly serialize the entire C# type system. This is a strong reason to recommend MessagePack over protobuf.

Protocol Buffers have good IDL and gRPC support. If you want to use IDL, I recommend Google.Protobuf over MessagePack.

JSON is good general-purpose format. It is simple, human-readable and thoroughly-enough specified. Utf8Json - which I created as well - adopts same architecture as MessagePack for C# and avoids encoding/decoding costs as much as possible just like this library does. If you want to know more about binary vs text formats, see Utf8Json/which serializer should be used.

ZeroFormatter is similar as FlatBuffers but specialized to C#, and special in that regard. Deserialization is infinitely fast but the produced binary size is larger. And ZeroFormatter's caching algorithm requires additional memory.

For many common uses, MessagePack for C# would be a better fit.

Hints to achieve maximum performance when using MessagePack for C#

MessagePack for C# prioritizes maximum performance by default. However, there are also some options that sacrifice performance for convenience.

The Deserialization Performance for different options section shows the results of indexed keys (IntKey) vs string keys (StringKey) performance. Indexed keys serialize the object graph as a MessagePack array. String keys serializes the object graph as a MessagePack map.

For example this type is serialized to

[MessagePackObject]
public class Person
{
    [Key(0)] or [Key("name")]
    public string Name { get; set;}
    [Key(1)] or [Key("age")]
    public int Age { get; set;}
}

new Person { Name = "foobar", Age = 999 }

• IntKey: ["foobar", 999]
• StringKey: {"name:"foobar","age":999}.

IntKey is always fast in both serialization and deserialization because it does not have to handle and lookup key names, and always has the smaller binary size.

StringKey is often a useful, contractless, simple replacement for JSON, interoperability with other languages with MessagePack support, and less error prone versioning. But to achieve maximum performance, use IntKey.

CompositeResolver.Create is an easy way to create composite resolvers. But formatter lookups have some overhead. If you create a custom resolver (or use StaticCompositeResolver.Instance), you can avoid this overhead.

public class MyApplicationResolver : IFormatterResolver
{
    public static readonly IFormatterResolver Instance = new MyApplicationResolver();

    // configure your custom resolvers.
    private static readonly IFormatterResolver[] Resolvers = new IFormatterResolver[]
    {
    };

    private MyApplicationResolver() { }

    public IMessagePackFormatter<T> GetFormatter<T>()
    {
        return Cache<T>.Formatter;
    }

    private static class Cache<T>
    {
        public static IMessagePackFormatter<T> Formatter;

        static Cache()
        {
            // configure your custom formatters.
            if (typeof(T) == typeof(XXX))
            {
                Formatter = new ICustomFormatter();
                return;
            }

            foreach (var resolver in Resolvers)
            {
                var f = resolver.GetFormatter<T>();
                if (f != null)
                {
                    Formatter = f;
                    return;
                }
            }
        }
    }
}

NOTE: If you are creating a library, recommend using the above custom resolver instead of CompositeResolver.Create. Also, libraries must not use StaticCompositeResolver - as it is global state - to avoid compatibility issues.

By default, MessagePack for C# serializes GUID as string. This is much slower than the native .NET format GUID. The same applies to Decimal. If your application makes heavy use of GUID or Decimal and you don't have to worry about interoperability with other languages, you can replace them with the native serializers NativeGuidResolver and NativeDecimalResolver respectively.

Also, DateTime is serialized using the MessagePack timestamp format. By using the NativeDateTimeResolver, it is possible to maintain Kind and perform faster serialization.

MessagePackSerializer.Serialize returns byte[] in default. The final byte[] is copied from an internal buffer pool. That is an extra cost. You can use IBufferWriter<T> or the Stream API to write to buffers directly. If you want to use a buffer pool outside of the serializer, you should implement custom IBufferWriter<byte> or use an existing one such as Sequence<T> from the Nerdbank.Streams package.

During deserialization, MessagePackSerializer.Deserialize(ReadOnlyMemory<byte> buffer) is better than the Deserialize(Stream stream) overload. This is because the Stream API version starts by reading the data, generating a ReadOnlySequence<byte>, and only then starts the deserialization.

Compression is generally effective when there is duplicate data. In MessagePack, arrays containing objects using string keys (Contractless) can be compressed efficiently because compression can be applied to many duplicate property names. Indexed keys compression is not as effectively compressed as string keys, but indexed keys are smaller in the first place.

This is some example benchmark performance data;

IntKey(Lz4) is not as effectively compressed, but performance is still somewhat degraded. On the other hand, StringKey can be expected to have a sufficient effect on the binary size. However, this is just an example. Compression can be quite effective depending on the data, too, or have little effect other than slowing down your program. There are also cases in which well-compressible data exists in the values (such as long strings, e.g. containing HTML data with many repeated HTML tags). It is important to verify the actual effects of compression on a case by case basis.

Extensions

MessagePack for C# has extension points that enable you to provide optimal serialization support for custom types. There are official extension support packages.

Install-Package MessagePack.ReactiveProperty
Install-Package MessagePack.UnityShims
Install-Package MessagePack.AspNetCoreMvcFormatter

The MessagePack.ReactiveProperty package adds support for types of the ReactiveProperty library. It adds ReactiveProperty<>, IReactiveProperty<>, IReadOnlyReactiveProperty<>, ReactiveCollection<>, Unit serialization support. It is useful for save viewmodel state.

The MessagePack.UnityShims package provides shims for Unity's standard structs (Vector2, Vector3, Vector4, Quaternion, Color, Bounds, Rect, AnimationCurve, Keyframe, Matrix4x4, Gradient, Color32, RectOffset, LayerMask, Vector2Int, Vector3Int, RangeInt, RectInt, BoundsInt) and corresponding formatters. It can enable proper communication between servers and Unity clients.

After installation, extension packages must be enabled, by creating composite resolvers. Here is an example showing how to enable all extensions.

// Set extensions to default resolver.
var resolver = MessagePack.Resolvers.CompositeResolver.Create(
    // enable extension packages first
    ReactivePropertyResolver.Instance,
    MessagePack.Unity.Extension.UnityBlitResolver.Instance,
    MessagePack.Unity.UnityResolver.Instance,

    // finally use standard (default) resolver
    StandardResolver.Instance
);
var options = MessagePackSerializerOptions.Standard.WithResolver(resolver);

// Pass options every time or set as default
MessagePackSerializer.DefaultOptions = options;

For configuration details, see: Extension Point section.

The MessagePack.AspNetCoreMvcFormatter is add-on for ASP.NET Core MVC's serialization to boost up performance. This is configuration sample.

public void ConfigureServices(IServiceCollection services)
{
    services.AddMvc().AddMvcOptions(option =>
    {
        option.OutputFormatters.Clear();
        option.OutputFormatters.Add(new MessagePackOutputFormatter(ContractlessStandardResolver.Options));
        option.InputFormatters.Clear();
        option.InputFormatters.Add(new MessagePackInputFormatter(ContractlessStandardResolver.Options));
    });
}

Other authors are creating extension packages, too.

• MagicOnion - gRPC based HTTP/2 RPC Streaming Framework
• MasterMemory - Embedded Readonly In-Memory Document Database

You can make your own extension serializers or integrate with frameworks. Let's create and share!

• MessagePack.FSharpExtensions - supports F# list, set, map, unit, option, discriminated union
• MessagePack.NodaTime - Support for NodaTime types to MessagePack C#
• WebApiContrib.Core.Formatter.MessagePack - supports ASP.NET Core MVC (details in blog post)
• MessagePack.MediaTypeFormatter - MessagePack MediaTypeFormatter

Experimental Features

MessagePack for C# has experimental features which provides you with very performant formatters. There is an official package.

Install-Package MessagePack.Experimental

For detailed information, see: Experimental.md

API

High-Level API (MessagePackSerializer)

The MessagePackSerializer class is the entry point of MessagePack for C#. Static methods make up the main API of MessagePack for C#.

The MessagePackSerializer.Typeless class offers most of the same APIs as above, but removes all type arguments from the API, forcing serialization to include the full type name of the root object. It uses the TypelessContractlessStandardResolver. Consider the result to be a .NET-specific MessagePack binary that isn't readily compatible with MessagePack deserializers in other runtimes.

MessagePack for C# fundamentally serializes using IBufferWriter<byte> and deserializes using ReadOnlySequence<byte> or Memory<byte>. Method overloads are provided to conveniently use it with common buffer types and the .NET Stream class, but some of these convenience overloads require copying buffers once and therefore have a certain overhead.

The high-level API uses a memory pool internally to avoid unnecessary memory allocation. If result size is under 64K, it allocates GC memory only for the return bytes.

Each serialize/deserialize method takes an optional MessagePackSerializerOptions parameter which can be used to specify a custom IFormatterResolver to use or to activate LZ4 compression support.

To deserialize a Stream that contains multiple consecutive MessagePack data structures, you can use the MessagePackStreamReader class to efficiently identify the ReadOnlySequence<byte> for each data structure and deserialize it. For example:

static async Task<List<T>> DeserializeListFromStreamAsync<T>(Stream stream, CancellationToken cancellationToken)
{
    var dataStructures = new List<T>();
    using (var streamReader = new MessagePackStreamReader(stream))
    {
        while (await streamReader.ReadAsync(cancellationToken) is ReadOnlySequence<byte> msgpack)
        {
            dataStructures.Add(MessagePackSerializer.Deserialize<T>(msgpack, cancellationToken: cancellationToken));
        }
    }

    return dataStructures;
}

Low-Level API (IMessagePackFormatter<T>)

The IMessagePackFormatter<T> interface is responsible for serializing a unique type. For example Int32Formatter : IMessagePackFormatter<Int32> represents Int32 MessagePack serializer.

public interface IMessagePackFormatter<T>
{
    void Serialize(ref MessagePackWriter writer, T value, MessagePackSerializerOptions options);
    T Deserialize(ref MessagePackReader reader, MessagePackSerializerOptions options);
}

Many built-in formatters exists under MessagePack.Formatters. Your custom types are usually automatically supported with the built-in type resolvers that generate new IMessagePackFormatter<T> types on-the-fly using dynamic code generation. See our AOT code generation support for platforms that do not support this.

However, some types - especially those provided by third party libraries or the runtime itself - cannot be appropriately annotated, and contractless serialization would produce inefficient or even wrong results. To take more control over the serialization of such custom types, write your own IMessagePackFormatter<T> implementation. Here is an example of such a custom formatter implementation. Note its use of the primitive API that is described in the next section.

/// <summary>Serializes a <see cref="FileInfo" /> by its full path as a string.</summary>
public class FileInfoFormatter : IMessagePackFormatter<FileInfo>
{
    public void Serialize(
      ref MessagePackWriter writer, FileInfo value, MessagePackSerializerOptions options)
    {
        if (value == null)
        {
            writer.WriteNil();
            return;
        }

        writer.WriteString(value.FullName);
    }

    public FileInfo Deserialize(
      ref MessagePackReader reader, MessagePackSerializerOptions options)
    {
        if (reader.TryReadNil())
        {
            return null;
        }

        options.Security.DepthStep(ref reader);

        var path = reader.ReadString();

        reader.Depth--;
        return new FileInfo(path);
    }
}

The DepthStep and Depth-- statements provide a level of security while deserializing untrusted data that might otherwise be able to execute a denial of service attack by sending MessagePack data that would deserialize into a very deep object graph leading to a StackOverflowException that would crash the process. This pair of statements should surround the bulk of any IMessagePackFormatter<T>.Deserialize method.

Important: A message pack formatter must read or write exactly one data structure. In the above example we just read/write a string. If you have more than one element to write out, you must precede it with a map or array header. You must read the entire map/array when deserializing. For example:

public class MySpecialObjectFormatter : IMessagePackFormatter<MySpecialObject>
{
    public void Serialize(
      ref MessagePackWriter writer, MySpecialObject value, MessagePackSerializerOptions options)
    {
        if (value == null)
        {
            writer.WriteNil();
            return;
        }

        writer.WriteArrayHeader(2);
        writer.WriteString(value.FullName);
        writer.WriteString(value.Age);
    }

    public MySpecialObject Deserialize(
      ref MessagePackReader reader, MessagePackSerializerOptions options)
    {
        if (reader.TryReadNil())
        {
            return null;
        }

        options.Security.DepthStep(ref reader);

        string fullName = null;
        int age = 0;

        // Loop over *all* array elements independently of how many we expect,
        // since if we're serializing an older/newer version of this object it might
        // vary in number of elements that were serialized, but the contract of the formatter
        // is that exactly one data structure must be read, regardless.
        // Alternatively, we could check that the size of the array/map is what we expect
        // and throw if it is not.
        int count = reader.ReadArrayHeader();
        for (int i = 0; i < count; i++)
        {
            switch (i)
            {
                case 0:
                    fullName = reader.ReadString();
                    break;
                case 1:
                    age = reader.ReadInt32();
                    break;
                default:
                    reader.Skip();
                    break;
            }
        }

        reader.Depth--;
        return new MySpecialObject(fullName, age);
    }
}

Your custom formatters must be discoverable via some IFormatterResolver. Learn more in our resolvers section.

You can see many other samples from builtin formatters.

Primitive API (MessagePackWriter, MessagePackReader)

The MessagePackWriter and MessagePackReader structs make up the lowest-level API. They read and write the primitives types defined in the MessagePack specification.

A MessagePackReader can efficiently read from ReadOnlyMemory<byte> or ReadOnlySequence<byte> without any allocations, except to allocate a new string as required by the ReadString() method. All other methods return either value structs or ReadOnlySequence<byte> slices for extensions/arrays. Reading directly from ReadOnlySequence<byte> means the reader can directly consume some modern high performance APIs such as PipeReader.

The MessagePackReader is capable of automatically interpreting both the old and new MessagePack spec.

A MessagePackWriter writes to a given instance of IBufferWriter<byte>. Several common implementations of this exist, allowing zero allocations and minimal buffer copies while writing directly to several I/O APIs including PipeWriter.

The MessagePackWriter writes the new MessagePack spec by default, but can write MessagePack compatible with the old spec by setting the OldSpec property to true.

DateTime is serialized to MessagePack Timestamp format, it serialize/deserialize UTC and loses Kind info and requires that MessagePackWriter.OldSpec == false. If you use the NativeDateTimeResolver, DateTime values will be serialized using .NET's native Int64 representation, which preserves Kind info but may not be interoperable with non-.NET platforms.

Main Extension Point (IFormatterResolver)

An IFormatterResolver is storage of typed serializers. The MessagePackSerializer API accepts a MessagePackSerializerOptions object which specifies the IFormatterResolver to use, allowing customization of the serialization of complex types.

Each instance of MessagePackSerializer accepts only a single resolver. Most object graphs will need more than one for serialization, so composing a single resolver made up of several is often required, and can be done with the CompositeResolver as shown below:

// Do this once and store it for reuse.
var resolver = MessagePack.Resolvers.CompositeResolver.Create(
    // resolver custom types first
    ReactivePropertyResolver.Instance,
    MessagePack.Unity.Extension.UnityBlitResolver.Instance,
    MessagePack.Unity.UnityResolver.Instance,

    // finally use standard resolver
    StandardResolver.Instance
);
var options = MessagePackSerializerOptions.Standard.WithResolver(resolver);

// Each time you serialize/deserialize, specify the options:
byte[] msgpackBytes = MessagePackSerializer.Serialize(myObject, options);
T myObject2 = MessagePackSerializer.Deserialize<MyObject>(msgpackBytes, options);

A resolver can be set as default with MessagePackSerializer.DefaultOptions = options, but WARNING: When developing an application where you control all MessagePack-related code it may be safe to rely on this mutable static to control behavior. For all other libraries or multi-purpose applications that use MessagePackSerializer you should explicitly specify the MessagePackSerializerOptions to use with each method invocation to guarantee your code behaves as you expect even when sharing an AppDomain or process with other MessagePack users that may change this static property.

Here is sample of use DynamicEnumAsStringResolver with DynamicContractlessObjectResolver (It is Json.NET-like lightweight setting.)

// composite same as StandardResolver
var resolver = MessagePack.Resolvers.CompositeResolver.Create(
    MessagePack.Resolvers.BuiltinResolver.Instance,
    MessagePack.Resolvers.AttributeFormatterResolver.Instance,

    // replace enum resolver
    MessagePack.Resolvers.DynamicEnumAsStringResolver.Instance,

    MessagePack.Resolvers.DynamicGenericResolver.Instance,
    MessagePack.Resolvers.DynamicUnionResolver.Instance,
    MessagePack.Resolvers.DynamicObjectResolver.Instance,

    MessagePack.Resolvers.PrimitiveObjectResolver.Instance,

    // final fallback(last priority)
    MessagePack.Resolvers.DynamicContractlessObjectResolver.Instance
);

If you want to make an extension package, you should write both a formatter and resolver for easier consumption. Here is sample of a resolver:

public class SampleCustomResolver : IFormatterResolver
{
    // Resolver should be singleton.
    public static readonly IFormatterResolver Instance = new SampleCustomResolver();

    private SampleCustomResolver()
    {
    }

    // GetFormatter<T>'s get cost should be minimized so use type cache.
    public IMessagePackFormatter<T> GetFormatter<T>()
    {
        return FormatterCache<T>.Formatter;
    }

    private static class FormatterCache<T>
    {
        public static readonly IMessagePackFormatter<T> Formatter;

        // generic's static constructor should be minimized for reduce type generation size!
        // use outer helper method.
        static FormatterCache()
        {
            Formatter = (IMessagePackFormatter<T>)SampleCustomResolverGetFormatterHelper.GetFormatter(typeof(T));
        }
    }
}

internal static class SampleCustomResolverGetFormatterHelper
{
    // If type is concrete type, use type-formatter map
    static readonly Dictionary<Type, object> formatterMap = new Dictionary<Type, object>()
    {
        {typeof(FileInfo), new FileInfoFormatter()}
        // add more your own custom serializers.
    };

    internal static object GetFormatter(Type t)
    {
        object formatter;
        if (formatterMap.TryGetValue(t, out formatter))
        {
            return formatter;
        }

        // If type can not get, must return null for fallback mechanism.
        return null;
    }
}

MessagePackFormatterAttribute

MessagePackFormatterAttribute is a lightweight extension point of class, struct, interface, enum and property/field. This is like Json.NET's JsonConverterAttribute. For example, serialize private field, serialize x10 formatter.

[MessagePackFormatter(typeof(CustomObjectFormatter))]
public class CustomObject
{
    string internalId;

    public CustomObject()
    {
        this.internalId = Guid.NewGuid().ToString();
    }

    // serialize/deserialize internal field.
    class CustomObjectFormatter : IMessagePackFormatter<CustomObject>
    {
        public void Serialize(ref MessagePackWriter writer, CustomObject value, MessagePackSerializerOptions options)
        {
            options.Resolver.GetFormatterWithVerify<string>().Serialize(ref writer, value.internalId, options);
        }

        public CustomObject Deserialize(ref MessagePackReader reader, MessagePackSerializerOptions options)
        {
            var id = options.Resolver.GetFormatterWithVerify<string>().Deserialize(ref reader, options);
            return new CustomObject { internalId = id };
        }
    }
}

// per field, member

public class Int_x10Formatter : IMessagePackFormatter<int>
{
    public int Deserialize(ref MessagePackReader reader, MessagePackSerializerOptions options)
    {
        return reader.ReadInt32() * 10;
    }

    public void Serialize(ref MessagePackWriter writer, int value, MessagePackSerializerOptions options)
    {
        writer.WriteInt32(value * 10);
    }
}

[MessagePackObject]
public class MyClass
{
    // You can attach custom formatter per member.
    [Key(0)]
    [MessagePackFormatter(typeof(Int_x10Formatter))]
    public int MyProperty1 { get; set; }
}

Formatter is retrieved by AttributeFormatterResolver, it is included in StandardResolver.

IgnoreFormatter

IgnoreFormatter<T> is lightweight extension point of class and struct. If there exists types that can't be serialized, you can register IgnoreFormatter<T> that serializes those to nil/null.

// CompositeResolver can set custom formatter.
var resolver = MessagePack.Resolvers.CompositeResolver.Create(
    new IMessagePackFormatter[]
    {
        // for example, register reflection infos (can not serialize)
        new IgnoreFormatter<MethodBase>(),
        new IgnoreFormatter<MethodInfo>(),
        new IgnoreFormatter<PropertyInfo>(),
        new IgnoreFormatter<FieldInfo>()
    },
    new IFormatterResolver[]
    {
        ContractlessStandardResolver.Instance
    });

Reserved Extension Types

MessagePack for C# already used some MessagePack extension type codes, be careful to avoid using the same ext code for other purposes.

This leaves the following ranges for your use:

• [0, 30)
• [120, 127]

Within the reserved ranges, this library defines or implements extensions that use these type codes:

Unity support

Unity lowest supported version is 2018.3, API Compatibility Level supports both .NET 4.x and .NET Standard 2.0.

You can install the unitypackage from the releases page. If your build targets .NET Framework 4.x and runs on mono, you can use it as is. But if your build targets IL2CPP, you can not use Dynamic***Resolver, so it is required to use pre-code generation. Please see pre-code generation section.

MessagePack for C# includes some additional System.*.dll libraries that originally provides in NuGet. They are located under Plugins. If other packages use these libraries (e.g. Unity Collections package using System.Runtime.CompilerServices.Unsafe.dll), to avoid conflicts, please delete the DLL under Plugins.

Currently CompositeResolver.Create does not work on IL2CPP, so it is recommended to use StaticCompositeResolver.Instance.Register instead.

In Unity, MessagePackSerializer can serialize Vector2, Vector3, Vector4, Quaternion, Color, Bounds, Rect, AnimationCurve, Keyframe, Matrix4x4, Gradient, Color32, RectOffset, LayerMask, Vector2Int, Vector3Int, RangeInt, RectInt, BoundsInt and their nullable, array and list types with the built-in extension UnityResolver. It is included in StandardResolver by default.

MessagePack for C# has an additional unsafe extension. UnsafeBlitResolver is special resolver for extremely fast but unsafe serialization/deserialization of struct arrays.

x20 faster Vector3[] serialization than native JsonUtility. If use UnsafeBlitResolver, serialization uses a special format (ext:typecode 30~39) for Vector2[], Vector3[], Quaternion[], Color[], Bounds[], Rect[]. If use UnityBlitWithPrimitiveArrayResolver, it supports int[], float[], double[] too. This special feature is useful for serializing Mesh (many Vector3[]) or many transform positions.

If you want to use unsafe resolver, register UnityBlitResolver or UnityBlitWithPrimitiveArrayResolver.

Here is sample of configuration.

StaticCompositeResolver.Instance.Register(
    MessagePack.Unity.UnityResolver.Instance,
    MessagePack.Unity.Extension.UnityBlitWithPrimitiveArrayResolver.Instance,
    MessagePack.Resolvers.StandardResolver.Instance
);

var options = MessagePackSerializerOptions.Standard.WithResolver(StaticCompositeResolver.Instance);
MessagePackSerializer.DefaultOptions = options;

The MessagePack.UnityShims NuGet package is for .NET server-side serialization support to communicate with Unity. It includes shims for Vector3 etc and the Safe/Unsafe serialization extension.

If you want to share a class between Unity and a server, you can use SharedProject or Reference as Link or a glob reference (with LinkBase), etc. Anyway, you need to share at source-code level. This is a sample project structure using a glob reference (recommended).

• ServerProject(.NET 4.6/.NET Core/.NET Standard)
  • [<Compile Include="..\UnityProject\Assets\Scripts\Shared\**\*.cs" LinkBase="Shared" />]
  • [MessagePack]
  • [MessagePack.UnityShims]
• UnityProject
  • [Concrete SharedCodes]
  • [MessagePack](not dll/NuGet, use MessagePack.Unity.unitypackage's sourcecode)

AOT Code Generation (support for Unity/Xamarin)

By default, MessagePack for C# serializes custom objects by generating IL on the fly at runtime to create custom, highly tuned formatters for each type. This code generation has a minor upfront performance cost. Because strict-AOT environments such as Xamarin and Unity IL2CPP forbid runtime code generation, MessagePack provides a way for you to run a code generator ahead of time as well.

Note: When using Unity, dynamic code generation only works when targeting .NET Framework 4.x + mono runtime. For all other Unity targets, AOT is required.

If you want to avoid the upfront dynamic generation cost or you need to run on Xamarin or Unity, you need AOT code generation. mpc (MessagePackCompiler) is the code generator of MessagePack for C#. mpc uses Roslyn to analyze source code.

First of all, mpc requires .NET Core 3 Runtime. The easiest way to acquire and run mpc is as a dotnet tool.

dotnet tool install --global MessagePack.Generator

Installing it as a local tool allows you to include the tools and versions that you use in your source control system. Run these commands in the root of your repo:

dotnet new tool-manifest
dotnet tool install MessagePack.Generator

Check in your .config\dotnet-tools.json file. On another machine you can "restore" your tool using the dotnet tool restore command.

Once you have the tool installed, simply invoke using dotnet mpc within your repo:

dotnet mpc --help

Alternatively, you can download mpc from the releases page, that includes platform native binaries (that don't require a separate dotnet runtime).

Usage: mpc [options...]

Options:
  -i, -input <String>                                Input path to MSBuild project file or the directory containing Unity source files. (Required)
  -o, -output <String>                               Output file path(.cs) or directory(multiple generate file). (Required)
  -c, -conditionalSymbol <String>                    Conditional compiler symbols, split with ','. (Default: null)
  -r, -resolverName <String>                         Set resolver name. (Default: GeneratedResolver)
  -n, -namespace <String>                            Set namespace root name. (Default: MessagePack)
  -m, -useMapMode <Boolean>                          Force use map mode serialization. (Default: False)
  -ms, -multipleIfDirectiveOutputSymbols <String>    Generate #if-- files by symbols, split with ','. (Default: null)

mpc targets C# code with [MessagePackObject] or [Union] annotations.

// Simple Sample:
dotnet mpc -i "..\src\Sandbox.Shared.csproj" -o "MessagePackGenerated.cs"

// Use force map simulate DynamicContractlessObjectResolver
dotnet mpc -i "..\src\Sandbox.Shared.csproj" -o "MessagePackGenerated.cs" -m

By default, mpc generates the resolver as MessagePack.Resolvers.GeneratedResolver and formatters asMessagePack.Formatters.*.

Here is the full sample code to register a generated resolver in Unity.

using MessagePack;
using MessagePack.Resolvers;
using UnityEngine;

public class Startup
{
    static bool serializerRegistered = false;

    [RuntimeInitializeOnLoadMethod(RuntimeInitializeLoadType.BeforeSceneLoad)]
    static void Initialize()
    {
        if (!serializerRegistered)
        {
            StaticCompositeResolver.Instance.Register(
                 MessagePack.Resolvers.GeneratedResolver.Instance,
                 MessagePack.Resolvers.StandardResolver.Instance
            );

            var option = MessagePackSerializerOptions.Standard.WithResolver(StaticCompositeResolver.Instance);

            MessagePackSerializer.DefaultOptions = option;
            serializerRegistered = true;
        }
    }

#if UNITY_EDITOR


    [UnityEditor.InitializeOnLoadMethod]
    static void EditorInitialize()
    {
        Initialize();
    }

#endif
}

In Unity, you can use MessagePack CodeGen windows at Windows -> MessagePack -> CodeGenerator.

Install the .NET Core runtime, install mpc (as a .NET Core Tool as described above), and execute dotnet mpc. Currently this tool is experimental so please tell me your opinion.

In Xamarin, you can install the the MessagePack.MSBuild.Tasks NuGet package into your projects to pre-compile fast serialization code and run in environments where JIT compilation is not allowed.

RPC

MessagePack advocated MessagePack RPC, but work on it has stopped and it is not widely used.

I've created a gRPC based MessagePack HTTP/2 RPC streaming framework called MagicOnion. gRPC usually communicates with Protocol Buffers using IDL. But MagicOnion uses MessagePack for C# and does not need IDL. When communicating C# to C#, schemaless (or rather C# classes as schema) is better than using IDL.

The StreamJsonRpc library is based on JSON-RPC and includes a pluggable formatter architecture and as of v2.3 includes MessagePack support.

How to build

See our contributor's guide.

Author Info

Yoshifumi Kawai (a.k.a. neuecc) is a software developer in Japan. He is the Director/CTO at Grani, Inc. Grani is a mobile game developer company in Japan and well known for using C#. He is awarding Microsoft MVP for Visual C# since 2011. He is known as the creator of UniRx (Reactive Extensions for Unity)

• Blog: https://medium.com/@neuecc (English)
• Blog: http://neue.cc/ (Japanese)
• Twitter: https://twitter.com/neuecc (Japanese)

MsgPack v5 implementation for C++ 11

Features

• std::streambuf serializer and deserializer
• hierarchy or token stream
• push and pull parser
• byte wise data flow control

Small Example

MsgPack::Serializer serializer(socket);  
std::vector<std::unique_ptr<MsgPack::Element>> arrayWithoutElements, arrayWith3Elements;
arrayWith3Elements.push_back(MsgPack::Factory(true));
arrayWith3Elements.push_back(MsgPack__Factory(Array(std::move(arrayWithoutElements))));
arrayWith3Elements.push_back(MsgPack::Factory("Hello World!"));  
serializer << MsgPack__Factory(Array(std::move(arrayWith3Elements)));

MsgPack::Deserializer deserializer(socket);  
deserializer.deserialize([](std::unique_ptr<MsgPack::Element> parsed) {
    std::cout << "Parsed: " << *parsed << "\n";
    return false;
}, true);

Read More

Tutorial

GoodForm

Form validation library. Includes MsgPack and JSON serializer/deserializer.

Building

GoodForm uses std::any, which requires c++17. When c++17 is not available, boost::any is expected and will be installed automatically when using cget.

cd goodform
cget install -f ./requirements.txt                      # Install dependencies locally.
mkdir build && cd build                                 # Create out of source build directory.
cmake -DCMAKE_TOOLCHAIN_FILE=../cget/cget/cget.cmake .. # Configure project with dependency paths.
make

MsgPack Usage

std::stringstream ss;
goodform::any var, var2;
var = goodform::object
  {
    {"compact", true},
    {"schema", 0}
  };

goodform::msgpack::serialize(var, ss);
goodform::msgpack::deserialize(ss, var2);

goodform::form form(var2);

struct
{
  bool compact;
  std::int32_t schema;
} mpack;

mpack.compact = form.at("compact").boolean().val();
mpack.schema = form.at("schema").int32().val();

if (form.is_good())
{
  std::cout << "{ \"compact\": " << std::boolalpha << mpack.compact << ", \"schema\": " << mpack.schema << " }" << std::endl;
}

JSON Usage

goodform::any var;
std::stringstream ss;
ss << "{" << std::endl
  << "\"first_name\":\"John\", // This is a comment" << std::endl
  << "\"last_name\":\"Smith\", " << std::endl
  << "\"age\": 23," << std::endl
  << "\"gpa\": 4.0," << std::endl
  << "\"email\":\"[email protected]\"," << std::endl
  << "\"password_hash\":\"5f4dcc3b5aa765d61d8327deb882cf99\"," << std::endl
  << "\"interests\": [\"sailing\",\"swimming\",\"yoga\"]" << std::endl
  << "}" << std::endl;

goodform::json::deserialize(ss, var);

goodform::form form(var);

struct
{
  std::string first_name;
  std::string last_name;
  std::uint8_t age;
  float gpa;
  std::string email;
  std::string password_hash;
  bool subscribe_to_email_marketing;
  std::list<std::string> interests;
} form_data;


form_data.first_name = form.at("first_name").string().match(std::regex("^[a-zA-Z ]{1,64}$")).val();
form_data.last_name = form.at("last_name").string().match(std::regex("^[a-zA-Z ]{1,64}$")).val();
form_data.age = form.at("age").uint8().val();
form_data.gpa = form.at("gpa").float32().gte(0).lte(4.0).val();
form_data.email = form.at("email").string().match(std::regex("^.{3,256}$")).val();
form_data.password_hash = form.at("password_hash").string().match(std::regex("^[a-fA-F0-9]{32}$")).val();
form_data.subscribe_to_email_marketing = form.at("subscribe_to_email_marketing", true).boolean().val(); // Optional field defaults to true.

form.at("interests").array().for_each([&form_data](goodform::sub_form& sf, std::size_t i)
{
  form_data.interests.push_back(sf.string().val());
});

if (form.is_good())
{
  // Use validated form_data.
}
else
{
  // Handle error.
}

What is msgpack11 ?

msgpack11 is a tiny MsgPack library for C++11, providing MsgPack parsing and serialization.
This library is inspired by json11.
The API of msgpack11 is designed to be similar with json11.

Installation

• Using CMake

    git clone [email protected]:ar90n/msgpack11.git
    mkdir build
    cd build
    cmake ../msgpack11
    make && make install
  

• Using Buck

    git clone [email protected]:ar90n/msgpack11.git
    cd msgpack11
    buck build :msgpack11
  

Example

MsgPack my_msgpack = MsgPack::object {
    { "key1", "value1" },
    { "key2", false },
    { "key3", MsgPack::array { 1, 2, 3 } },
};

//access to elements
std::cout << my_msgpack["key1"].string_value();

//serialize
std::string msgpack_bytes = my_msgpack.dump();

//deserialize
std::string err;
MsgPack des_msgpack = MsgPack::parse(msgpack_bytes, err);

There are more specific examples in example.cpp. Please see it.

Benchmark

Derived from schemaless-benchmarks

CPU : 2.6 GHz Intel Core i7
Memory : 16 GB 2133 MHz LPDDR3
Git revision : 6f6b4302b68b3c88312eb24367418b7fce81298c

Feature

• Support serialization and deserialization.

Acknowledgement

• json11
• msgpack-c
• schemaless-benchmarks

License

This software is released under the MIT License, see LICENSE.txt.

cppack

A modern (c++17 required) implementation of the msgpack spec.

Msgpack is a binary serialization specification. It allows you to save and load application objects like classes and structs over networks, to files, and between programs and even different languages.

Check out this blog for my rational creating this library.

Features

• Fast and compact
• Full test coverage
• Easy to use
• Automatic type handling
• Open source MIT license
• Easy error handling

Want to use this library? Just #include the header and you're good to go. Its less than 1000 lines of code.

Easily pack objects into byte arrays using a pack free function:

struct Person {
  std::string name;
  uint16_t age;
  std::vector<std::string> aliases;

  template<class T>
  void msgpack(T &pack) {
    pack(name, age, aliases);
  }
};

int main() {
    auto person = Person{"John", 22, {"Ripper", "Silverhand"}};

    auto data = msgpack::pack(person); // Pack your object
    auto john = msgpack::unpack<Person>(data.data()); // Unpack it
}

More Examples

• Support for extension types
  • The msgpack spec allows for additional types to be enumerated as Extensions. If reasonable use cases come about for this feature then it may be added.
• Name/value pairs
  • The msgpack spec uses the 'map' type differently than this library. This library implements maps in which key/value pairs must all have the same value types.
• Endian conversion shortcuts
  • On platforms that already hold types in big endian, the serialization could be optimized using type traits.

msgpack for C/C++