MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON. But it's faster and smaller. Small integers are encoded into a single byte, and typical short strings require only one extra byte in addition to the strings themselves.
Next:
MessagePack is supported by over 50 programming languages and environments. See list of implementations.
Redis scripting has support for MessagePack because it is a fast and compact serialization format with a simple to implement specification. I liked it so much that I implemented a MessagePack C extension for Lua just to include it into Redis.
Salvatore Sanfilippo, creator of Redis
Fluentd uses MessagePack for all internal data representation. It's crazy fast because of zero-copy optimization of msgpack-ruby. Now MessagePack is an essential component of Fluentd to achieve high performance and flexibility at the same time.
Sadayuki Furuhashi, creator of Fluentd
Treasure Data built a multi-tenant database optimized for analytical queries using MessagePack. The schemaless database is growing by billions of records every month. We also use MessagePack as a glue between components. Actually we just wanted a fast replacement of JSON, and MessagePack is simply useful.
Kazuki Ohta, CTO
MessagePack has been simply invaluable to us. We use MessagePack + Memcache to cache many of our feeds on Pinterest. These feeds are compressed and very quick to unpack thanks to MessagePack while Memcache gives us fast atomic pushes.
MessagePack for Actionscript3 (Flash, Flex and AIR).
as3-msgpack was designed to work with the interfaces IDataInput and IDataOutput, thus the API might be easily connected with the native classes that handle binary data (such as ByteArray, Socket, FileStream and URLStream).
Moreover, as3-msgpack is capable of decoding data from binary streams.
Get started: http://loteixeira.github.io/lib/2013/08/19/as3-msgpack/
Basic usage (encoding/decoding):
// create messagepack objectvar msgpack:MsgPack =new MsgPack();// encode an arrayvarbytes:ByteArray= msgpack.write([1, 2, 3, 4, 5]);// rewind the bufferbytes.position=0;// print the decoded objecttrace(msgpack.read(bytes));
This library is only for serialize / deserialize.
To send / receive serialized data with Stream class, please use MsgPacketizer.
#include<MsgPack.h>// input to msgpackint i = 123;
float f = 1.23;
MsgPack::str_t s = "str"; // std::string or String
MsgPack::arr_t<int> v {1, 2, 3}; // std::vector or arx::vector
MsgPack::map_t<String, float> m {{"one", 1.1}, {"two", 2.2}, {"three", 3.3}}; // std::map or arx::map// output from msgpackint ri;
float rf;
MsgPack::str_t rs;
MsgPack::arr_t<int> rv;
MsgPack::map_t<String, float> rm;
voidsetup() {
delay(2000);
Serial.begin(115200);
Serial.println("msgpack test start");
// serialize to msgpack
MsgPack::Packer packer;
packer.serialize(i, f, s, v, m);
// deserialize from msgpack
MsgPack::Unpacker unpacker;
unpacker.feed(packer.data(), packer.size());
unpacker.deserialize(ri, rf, rs, rv, rm);
if (i != ri) Serial.println("failed: int");
if (f != rf) Serial.println("failed: float");
if (s != rs) Serial.println("failed: string");
if (v != rv) Serial.println("failed: vector<int>");
if (m != rm) Serial.println("failed: map<string, int>");
Serial.println("msgpack test success");
}
voidloop() {}
Encode / Decode to Collections without Container
In msgpack, there are two collection types: Array and Map.
C++ containers will be converted to one of them but you can do that from individual parameters.
To pack / unpack values as such collections in a simple way, please use these functions.
packer.to_array(i, f, s); // becoms array format [i, f, s];
unpacker.from_array(ii, ff, ss); // unpack from array format to ii, ff, ss
packer.to_map("i", i, "f", f); // becoms {"i":i, "f":f}
unpacker.from_map(ki, ii, kf, ff); // unpack from map to ii, ff, ss
The same conversion can be achieved using serialize and deserialize.
packer.serialize(MsgPack::arr_size_t(3), i, f, s); // [i, f, s]
unpacker.deserialize(MsgPack::arr_size_t(3), ii, ff, ss);
packer.serialize(MsgPack::map_size_t(2), "i", i, "f", f); // {"i":i, "f":f}
unpacker.deserialize(MsgPack::map_size_t(2), ki, ii, kf, ff);
Here, MsgPack::arr_size_t and MsgPack::map_size_t are used to identify the size of Array and Map format in serialize or deserialize.
This way is expandable to pack and unpack complex data structure because it can be nested.
// {"i":i, "arr":[ii, iii]}
packer.serialize(MsgPack::map_size_t(2), "i", i, "arr", MsgPack::arr_size_t(2), ii, iii);
unpacker.deserialize(MsgPack::map_size_t(2), ki, i, karr, MsgPack::arr_size_t(2), ii, iii);
Custom Class Adaptation
To serialize / deserialize custom type you defined, please use MSGPACK_DEFINE() macro inside of your class. This macro enables you to convert your custom class to Array format.
In other languages like JavaScript, Python and etc. has also library for msgpack.
But some libraries can NOT convert msgpack in "plain" style.
They always wrap them into collections like Array or Map by default.
For example, you can't convert "plain" format in other languages.
packer.serialize(i, f, s); // "plain" format is NOT unpackable
packer.serialize(arr_size_t(3), i, f, s); // unpackable if you wrap that into Array
It is because the msgpack is used as based on JSON (I think).
So you need to use Array format for JSON array, and Map for Json Object.
To achieve that, there are several ways.
use to_array or to_map to convert to simple structure
use serialize() or deserialize() with arr_size_t / map_size_t for complex structure
use custom class as JSON array / object which is wrapped into Array / Map
use custom class nest recursively for more complex structure
use ArduinoJson for more flexible handling of JSON
Use MsgPack with ArduinoJson
you can [serialize / deserialize] StaticJsonDocument<N> and DynamicJsonDocument directly
You can directly save/load to/from JSON file with this library. SD, SdFat, SD_MMC, SPIFFS, etc. are available for the target file system. Please see save_load_as_json_file example for more details.
#include<SD.h>
#include<MsgPack.h>structMyConfig {
Meta meta;
Data data;
MSGPACK_DEFINE(meta, data);
};
MyConfig config;
voidsetup() {
SD.begin();
// load json data from /config.txt to config struct directly
MsgPack::file::load_from_json_static<256>(SD, "/config.txt", config);
// change your configuration...// save config data from config struct to /config.txt as json directly
MsgPack::file::save_as_json_static<256>(SD, "/config.txt", config);
}
Save/Load to/from EEPROM with MsgPack
In Arduino, you can use the MsgPack utility to save/load to/from EEPROM. Following code shows how to use them. Please see save_load_eeprom example for more details.
structMyConfig {
Meta meta;
Data data;
MSGPACK_DEFINE(meta, data);
};
MyConfig config;
voidsetup() {
EEPROM.begin();
// load current configMsgPack::eeprom::load(config);
// change your configuration...// saveMsgPack::eeprom::save(config);
EEPROM.end();
}
Supported Type Adaptors
These are the lists of types which can be serialize and deserialize.
You can also pack() or unpack() variable one by one.
NIL
MsgPack::object::nil_t
Bool
bool
Integer
char (signed/unsigned)
ints (signed/unsigned)
Float
float
double
Str
char*
char[]
std::string or String(Arduino) (MsgPack::str_t)
Bin
unsigned char* (need to serialize(ptr, size) or pack(ptr, size))
unsigned char[] (need to serialize(ptr, size) or pack(ptr, size))
STL is used to handle packet data by default, but for following boards/architectures, ArxContainer is used to store the packet data because STL can not be used for such boards.
The storage size of such boards for max packet binary size and number of msgpack objects are limited.
AVR
megaAVR
SAMD
Memory Management (for NO-STL Boards)
As mentioned above, for such boards like Arduino Uno, the storage sizes are limited.
And of course you can manage them by defining following macros.
But these default values are optimized for such boards, please be careful not to excess your boards storage/memory.
// msgpack serialized binary size
#defineMSGPACK_MAX_PACKET_BYTE_SIZE128// max size of MsgPack::arr_t
#defineMSGPACK_MAX_ARRAY_SIZE8// max size of MsgPack::map_t
#defineMSGPACK_MAX_MAP_SIZE8// msgpack objects size in one packet
#defineMSGPACK_MAX_OBJECT_SIZE24
These macros have no effect for STL enabled boards.
In addtion for such boards, type aliases for following types are different from others.
MsgPack::str_t = String
MsgPack::bin_t<T> = arx::vector<T, N = MSGPACK_MAX_PACKET_BYTE_SIZE>
MsgPack::arr_t<T> = arx::vector<T, N = MSGPACK_MAX_ARRAY_SIZE>
MsgPack::map_t<T, U> = arx::map<T, U, N = MSGPACK_MAX_MAP_SIZE>
Please see "Memory Management" section and ArxContainer for detail.
STL library for Arduino Support
For such boards, there are several STL libraries, like ArduinoSTL, StandardCPlusPlus, and so on.
But such libraries are mainly based on uClibc++ and it has many lack of function.
I considered to support them but I won't support them unless uClibc++ becomes much better compatibility to standard C++ library.
I reccomend to use low cost but much better performance chip like ESP series.
This Arduino library provides a light weight serializer and parser for messagepack.
Install
Download the zip, and import it with your Arduino IDE: Sketch>Include Library>Add .zip library
Usage
See the either the .h file, or the examples (led_controller and test_uno_writer).
In short:
functions like msgpck_what_next(Stream * s); watch the next type of data without reading it (without advancing the buffer of Stream s).
functions like msgpck_read_bool(Stream * s, bool *b) read a value from Stream s.
functions like msgpck_write_bool(Stream * s, bool b) write a value on Stream s.
Notes:
Stream are used as much as possible in order not to add to much overhead with buffers. Therefore you should be able to store the minimum number of value at a given time.
Map and Array related functions concern only their headers. Ex: If you want to write an array containing two elements you should write the array header, then write the two elements.
Limitations
Currently the library does not support:
8 bytes float (Only 4 bytes floats are supported by default on every Arduino and floats are anyway not recommended on Arduino)
CMP is a C implementation of the MessagePack serialization format. It
currently implements version 5 of the MessagePack
Spec.
CMP's goal is to be lightweight and straightforward, forcing nothing on the
programmer.
License
While I'm a big believer in the GPL, I license CMP under the MIT license.
Example Usage
The following examples use a file as the backend, and are modeled after the
examples included with the msgpack-c project.
#include<stdio.h>
#include<stdlib.h>
#include"cmp.h"staticboolread_bytes(void *data, size_t sz, FILE *fh) {
returnfread(data, sizeof(uint8_t), sz, fh) == (sz * sizeof(uint8_t));
}
staticboolfile_reader(cmp_ctx_t *ctx, void *data, size_t limit) {
returnread_bytes(data, limit, (FILE *)ctx->buf);
}
staticboolfile_skipper(cmp_ctx_t *ctx, size_t count) {
returnfseek((FILE *)ctx->buf, count, SEEK_CUR);
}
staticsize_tfile_writer(cmp_ctx_t *ctx, constvoid *data, size_t count) {
returnfwrite(data, sizeof(uint8_t), count, (FILE *)ctx->buf);
}
staticvoiderror_and_exit(constchar *msg) {
fprintf(stderr, "%s\n\n", msg);
exit(EXIT_FAILURE);
}
intmain(void) {
FILE *fh = NULL;
cmp_ctx_t cmp = {0};
uint32_t array_size = 0;
uint32_t str_size = 0;
char hello[6] = {0};
char message_pack[12] = {0};
fh = fopen("cmp_data.dat", "w+b");
if (fh == NULL) {
error_and_exit("Error opening data.dat");
}
cmp_init(&cmp, fh, file_reader, file_skipper, file_writer);
if (!cmp_write_array(&cmp, 2)) {
error_and_exit(cmp_strerror(&cmp));
}
if (!cmp_write_str(&cmp, "Hello", 5)) {
error_and_exit(cmp_strerror(&cmp));
}
if (!cmp_write_str(&cmp, "MessagePack", 11)) {
error_and_exit(cmp_strerror(&cmp));
}
rewind(fh);
if (!cmp_read_array(&cmp, &array_size)) {
error_and_exit(cmp_strerror(&cmp));
}
/* You can read the str byte size and then read str bytes... */if (!cmp_read_str_size(&cmp, &str_size)) {
error_and_exit(cmp_strerror(&cmp));
}
if (str_size > (sizeof(hello) - 1)) {
error_and_exit("Packed 'hello' length too long\n");
}
if (!read_bytes(hello, str_size, fh)) {
error_and_exit(cmp_strerror(&cmp));
}
/* * ...or you can set the maximum number of bytes to read and do it all in * one call*/
str_size = sizeof(message_pack);
if (!cmp_read_str(&cmp, message_pack, &str_size)) {
error_and_exit(cmp_strerror(&cmp));
}
printf("Array Length: %u.\n", array_size);
printf("[\"%s\", \"%s\"]\n", hello, message_pack);
fclose(fh);
return EXIT_SUCCESS;
}
Advanced Usage
See the examples folder.
Fast, Lightweight, Flexible, and Robust
CMP uses no internal buffers; conversions, encoding and decoding are done on
the fly.
CMP's source and header file together are ~4k LOC.
CMP makes no heap allocations.
CMP uses standardized types rather than declaring its own, and it depends only
on stdbool.h, stdint.h and string.h.
CMP is written using C89 (ANSI C), aside, of course, from its use of
fixed-width integer types and bool.
On the other hand, CMP's test suite requires C99.
CMP only requires the programmer supply a read function, a write function, and
an optional skip function. In this way, the programmer can use CMP on memory,
files, sockets, etc.
CMP is portable. It uses fixed-width integer types, and checks the endianness
of the machine at runtime before swapping bytes (MessagePack is big-endian).
CMP provides a fairly comprehensive error reporting mechanism modeled after
errno and strerror.
CMP is thread aware; while contexts cannot be shared between threads, each
thread may use its own context freely.
CMP is tested using the MessagePack test suite as well as a large set of custom
test cases. Its small test program is compiled with clang using -Wall -Werror -Wextra ... along with several other flags, and generates no compilation
errors in either clang or GCC.
CMP's source is written as readably as possible, using explicit, descriptive
variable names and a consistent, clear style.
CMP's source is written to be as secure as possible. Its testing suite checks
for invalid values, and data is always treated as suspect before it passes
validation.
CMP's API is designed to be clear, convenient and unsurprising. Strings are
null-terminated, binary data is not, error codes are clear, and so on.
CMP provides optional backwards compatibility for use with other MessagePack
implementations that only implement version 4 of the spec.
Building
There is no build system for CMP. The programmer can drop cmp.c and cmp.h
in their source tree and modify as necessary. No special compiler settings are
required to build it, and it generates no compilation errors in either clang or
gcc.
Versioning
CMP's versions are single integers. I don't use semantic versioning because
I don't guarantee that any version is completely compatible with any other. In
general, semantic versioning provides a false sense of security. You should be
evaluating compatibility yourself, not relying on some stranger's versioning
convention.
Stability
I only guarantee stability for versions released on
the releases page. While rare, both master and develop
branches may have errors or mismatched versions.
Backwards Compatibility
Version 4 of the MessagePack spec has no BIN type, and provides no STR8
marker. In order to remain backwards compatible with version 4 of MessagePack,
do the following:
Avoid these functions:
cmp_write_bin
cmp_write_bin_marker
cmp_write_str8_marker
cmp_write_str8
cmp_write_bin8_marker
cmp_write_bin8
cmp_write_bin16_marker
cmp_write_bin16
cmp_write_bin32_marker
cmp_write_bin32
Use these functions in lieu of their v5 counterparts:
cmp_write_str_marker_v4 instead of cmp_write_str_marker
cmp_write_str_v4 instead of cmp_write_str
cmp_write_object_v4 instead of cmp_write_object
Disabling Floating Point Operations
Thanks to tdragon it's possible to disable
floating point operations in CMP by defining CMP_NO_FLOAT. No floating point
functionality will be included. Fair warning: this changes the ABI.
Setting Endianness at Compile Time
CMP will honor WORDS_BIGENDIAN. If defined to 0 it will convert data
to/from little-endian format when writing/reading. If defined to 1 it won't.
If not defined, CMP will check at runtime.
The core of MPack contains a buffered reader and writer, and a tree-style parser that decodes into a tree of dynamically typed nodes. Helper functions can be enabled to read values of expected type, to work with files, to grow buffers or allocate strings automatically, to check UTF-8 encoding, and more.
The MPack code is small enough to be embedded directly into your codebase. Simply download the amalgamation package and add mpack.h and mpack.c to your project.
MPack supports all modern compilers, all desktop and smartphone OSes, WebAssembly, inside the Linux kernel, and even 8-bit microcontrollers such as Arduino. The MPack featureset can be customized at compile-time to set which features, components and debug checks are compiled, and what dependencies are available.
Build Status
The Node API
The Node API parses a chunk of MessagePack data into an immutable tree of dynamically-typed nodes. A series of helper functions can be used to extract data of specific types from each node.
// parse a file into a node treempack_tree_t tree;
mpack_tree_init_filename(&tree, "homepage-example.mp", 0);
mpack_tree_parse(&tree);
mpack_node_t root = mpack_tree_root(&tree);
// extract the example data on the msgpack homepagebool compact = mpack_node_bool(mpack_node_map_cstr(root, "compact"));
int schema = mpack_node_i32(mpack_node_map_cstr(root, "schema"));
// clean up and check for errorsif (mpack_tree_destroy(&tree) != mpack_ok) {
fprintf(stderr, "An error occurred decoding the data!\n");
return;
}
Note that no additional error handling is needed in the above code. If the file is missing or corrupt, if map keys are missing or if nodes are not in the expected types, special "nil" nodes and false/zero values are returned and the tree is placed in an error state. An error check is only needed before using the data.
The above example allocates nodes automatically. A fixed node pool can be provided to the parser instead in memory-constrained environments. For maximum performance and minimal memory usage, the Expect API can be used to parse data of a predefined schema.
The Write API
The Write API encodes structured data to MessagePack.
// encode to memory bufferchar* data;
size_t size;
mpack_writer_t writer;
mpack_writer_init_growable(&writer, &data, &size);
// write the example on the msgpack homepagempack_build_map(&writer);
mpack_write_cstr(&writer, "compact");
mpack_write_bool(&writer, true);
mpack_write_cstr(&writer, "schema");
mpack_write_uint(&writer, 0);
mpack_complete_map(&writer);
// finish writingif (mpack_writer_destroy(&writer) != mpack_ok) {
fprintf(stderr, "An error occurred encoding the data!\n");
return;
}
// use the datado_something_with_data(data, size);
free(data);
In the above example, we encode to a growable memory buffer. The writer can instead write to a pre-allocated or stack-allocated buffer (with up-front sizes for compound types), avoiding the need for memory allocation. The writer can also be provided with a flush function (such as a file or socket write function) to call when the buffer is full or when writing is done.
If any error occurs, the writer is placed in an error state. The writer will flag an error if too much data is written, if the wrong number of elements are written, if an allocation failure occurs, if the data could not be flushed, etc. No additional error handling is needed in the above code; any subsequent writes are ignored when the writer is in an error state, so you don't need to check every write for errors.
The above example uses mpack_build_map() to automatically determine the number of key-value pairs contained. If you know up-front the number of elements needed, you can pass it to mpack_start_map() instead. In that case the corresponding mpack_finish_map() will assert in debug mode that the expected number of elements were actually written, which is something that other MessagePack C/C++ libraries may not do.
Comparison With Other Parsers
MPack is rich in features while maintaining very high performance and a small code footprint. Here's a short feature table comparing it to other C parsers:
A larger feature comparison table is available here which includes descriptions of the various entries in the table.
This benchmarking suite compares the performance of MPack to other implementations of schemaless serialization formats. MPack outperforms all JSON and MessagePack libraries (except CWPack), and in some tests MPack is several times faster than RapidJSON for equivalent data.
Why Not Just Use JSON?
Conceptually, MessagePack stores data similarly to JSON: they are both composed of simple values such as numbers and strings, stored hierarchically in maps and arrays. So why not just use JSON instead? The main reason is that JSON is designed to be human-readable, so it is not as efficient as a binary serialization format:
Compound types such as strings, maps and arrays are delimited, so appropriate storage cannot be allocated upfront. The whole object must be parsed to determine its size.
Strings are not stored in their native encoding. Special characters such as quotes and backslashes must be escaped when written and converted back when read.
Numbers are particularly inefficient (especially when parsing back floats), making JSON inappropriate as a base format for structured data that contains lots of numbers.
Binary data is not supported by JSON at all. Small binary blobs such as icons and thumbnails need to be Base64 encoded or passed out-of-band.
The above issues greatly increase the complexity of the decoder. Full-featured JSON decoders are quite large, and minimal decoders tend to leave out such features as string unescaping and float parsing, instead leaving these up to the user or platform. This can lead to hard-to-find platform-specific and locale-specific bugs, as well as a greater potential for security vulnerabilites. This also significantly decreases performance, making JSON unattractive for use in applications such as mobile games.
While the space inefficiencies of JSON can be partially mitigated through minification and compression, the performance inefficiencies cannot. More importantly, if you are minifying and compressing the data, then why use a human-readable format in the first place?
Testing MPack
The MPack build process does not build MPack into a library; it is used to build and run the unit tests. You do not need to build MPack or the unit testing suite to use MPack.
See test/README.md for information on how to test MPack.
CWPack is a lightweight and yet complete implementation of the
MessagePack serialization format
version 5.
It also supports the Timestamp extension type.
Excellent Performance
Together with MPack, CWPack is the fastest open-source messagepack implementation. Both totally outperform
CMP and msgpack-c
Design
CWPack does no memory allocations and no file handling in its basic setup. All that is done outside of CWPack. Example extensions are included.
CWPack is working against memory buffers. User defined handlers are called when buffers are filled up (packing) or needs refill (unpack).
Containers (arrays, maps) are read/written in parts, first the item containing the size and then the contained items one by one. Exception to this is the cw_skip_items function which skip whole containers.
Example
Pack and unpack example from the MessagePack home page:
voidexample (void)
{
cw_pack_context pc;
char buffer[20];
cw_pack_context_init (&pc, buffer, 20, 0);
cw_pack_map_size (&pc, 2);
cw_pack_str (&pc, "compact", 7);
cw_pack_boolean (&pc, true);
cw_pack_str (&pc, "schema", 6);
cw_pack_unsigned (&pc, 0);
if (pc.return_code != CWP_RC_OK) ERROR;
int length = pc.current - pc.start;
if (length != 18) ERROR;
cw_unpack_context uc;
cw_unpack_context_init (&uc, pc.start, length, 0);
if (cw_unpack_next_map_size(&uc) != 2) ERROR;
if (cw_unpack_next_str_lengh(&uc) != 7) ERROR;
if (strncmp("compact", uc.item.as.str.start, 7)) ERROR;
if (cw_unpack_next_boolean(&uc) != true) ERROR;
if (cw_unpack_next_str_lengh(&uc) != 6) ERROR;
if (strncmp("schema", uc.item.as.str.start, 6)) ERROR;
if (cw_unpack_next_signed32(&uc) != 0) ERROR;
if (uc.return_code != CWP_RC_OK) ERROR;
cw_unpack_next(&uc);
if (uc.return_code != CWP_RC_END_OF_INPUT) ERROR;
}
In the examples folder there are more examples.
Backward compatibility
CWPack may be run in compatibility mode. It affects only packing; EXT & TIMESTAMP is considered illegal, BIN are transformed to STR and generation of STR8 is supressed.
Error handling
When an error is detected in a context, the context is stopped and all future calls to that context are immediatly returned without any actions. Thus it is possible to make some calls and delay error checking until all calls are done.
CWPack does not check for illegal values (e.g. in STR for illegal unicode characters).
Build
CWPack consists of a single src file and three header files. It is written in strict ansi C and the files are together ~ 1.4K lines. No separate build is neccesary, just include the files in your own build.
CWPack has no dependencies to other libraries.
Test
Included in the test folder are a module test and a performance test and shell scripts to run them.
Objective-C
CWPack also contains an Objective-C interface. The MessagePack home page example would look like:
This is MessagePack serialization/deserialization for CLI (Common Language Infrastructure) implementations such as .NET Framework, Silverlight, Mono (including Moonlight.)
This library can be used from ALL CLS compliant languages such as C#, F#, Visual Basic, Iron Python, Iron Ruby, PowerShell, C++/CLI or so.
Usage
You can serialize/deserialize objects as following:
Create serializer via MessagePackSerializer.Get generic method. This method creates dependent types serializers as well.
Invoke serializer as following:
Pack method with destination Stream and target object for serialization.
Unpack method with source Stream.
// Creates serializer.varserializer=MessagePackSerializer.Get<T>();
// Pack obj to stream.serializer.Pack(stream, obj);
// Unpack from stream.varunpackedObject=serializer.Unpack(stream);
' Creates serializer.Dimserializer=MessagePackSerializer.Get(OfT)()' Pack obj to stream.serializer.Pack(stream,obj)' Unpack from stream.DimunpackedObject=serializer.Unpack(stream)
For production environment, you should instantiate own SerializationContext and manage its lifetime. It is good idea to treat it as singleton because SerializationContext is thread-safe.
Features
Fast and interoperable binary format serialization with simple API.
Generating pre-compiled assembly for rapid start up.
Flexible MessagePackObject which represents MessagePack type system naturally.
Note: AOT support is limited yet. Use serializer pre-generation with mpu -s utility or API.
If you do not pre-generated serializers, MsgPack for CLI uses reflection in AOT environments, it is slower and it sometimes causes AOT related error (ExecutionEngineException for runtime JIT compilation). You also have to call MessagePackSerializer.PrepareType<T> and companions in advance to avoid AOT related error. See wiki for details.
For mono, you can use net461 or net35 drops as you run with.
For Unity, unity3d drop is suitable.
How to build
For .NET Framework
Install Visual Studio 2017 (Community edition is OK) and 2015 (for MsgPack.Windows.sln).
You must install .NET Framework 3.5, 4.x, .NET Core, and Xamarin dev tools to build all builds successfully.
If you do not want to install options, edit <TargetFrameworks> element in *.csproj files to exclude platforms you want to exclude.
Or open one of above solution files in your IDE and run build command in it.
For Mono
Install latest Mono and .NET Core SDK.
Now, you can build MsgPack.sln and MsgPack.Xamarin.sln with above instructions and msbuild in latest Mono. Note that xbuild does not work because it does not support latest csproj format.
Own Unity 3D Build
First of all, there are binary drops on github release page, you should use it to save your time.
Because we will not guarantee source code organization compatibilities, we might add/remove non-public types or members, which should break source code build.
If you want to import sources, you must include just only described on MsgPack.Unity3D.csproj.
If you want to use ".NET 2.0 Subset" settings, you must use just only described on MsgPack.Unity3D.CorLibOnly.csproj file, and define CORLIB_ONLY compiler constants.
Xamarin Android testing
If you run on Windows, it is recommended to use HXM instead of Hyper-V based emulator.
You can disable Hyper-V from priviledged (administrator) powershell as follows:
Q: An error occurred while running unit test project.
A: Rebuild the project and rerun it. Or, login your Mac again, ant retry it.
Q: It is hard to read English.
A: You can read localized Xamarin docs with putting {region}-{lang} as the first component of URL path such as https://developer.xamarin.com/ja-jp/guides/....
Maintenance
MsgPack.Windows.sln
This solution contains Silverlight5 and (old) UWP project for backward compability. They are required Visual Studio 2015 to build and test.
You can download Visual Studio 2015 community edition from here.
You do not have to install Visual Studio 2015 as long as you don't edit/test/build Silverlight and/or old UWP project.
This library is a lightweight implementation of the MessagePack binary serialization format. MessagePack is a 1-to-1 binary representation of JSON, and the official specification can be found here: https://github.com/msgpack....
This library is designed to be super light weight.
Its easiest to understand how this library works if you think in terms of json. The type MPackMap represents a dictionary, and the type MPackArray represents an array.
Create MPack instances with the static method MPack.From(object);. You can pass any simple type (such as string, integer, etc), or any Array composed of a simple type. MPack also has implicit conversions from most of the basic types built in.
Transform an MPack object back into a CLR type with the static method MPack.To<T>(); or MPack.To(type);. MPack also has explicit converions going back to most basic types, you can do string str = (string)mpack; for instance.
MPack now supports native asynchrounous reading and cancellation tokens. It will not block a thread to wait on a stream.
NuGet
MPack is available as a NuGet package!
PM> Install-Package MPack
Usage
Create a object model that can be represented as MsgPack. Here we are creating a dictionary, but really it can be anything:
Serialize the data to a byte array or to a stream to be saved, transmitted, etc:
byte[] encodedBytes=dictionary.EncodeToBytes();
// -- or --dictionary.EncodeToStream(stream);
Parse the binary data back into a MPack object model (you can also cast back to an MPackMap or MPackArray after reading if you want dictionary/array methods):
varreconstructed=MPack.ParseFromBytes(encodedBytes);
// -- or --varreconstructed=MPack.ParseFromStream(stream);
Turn MPack objects back into types that we understand with the generic To<>() method. Since we know the types of everything here we can just call To<bool>() to reconstruct our bool, but if you don't know you can access the instance enum MPack.ValueType to know what kind of value it is:
Although the original was optimised for debugging and analysing, a lightweight version of the lib is included which does not keep track of all offsets and other overhead needed for debugging. It can be used in your code.
I have never run any benchmarks so I have no idea how it will perform against other implementations.
Fiddler Integration
In order to use this tool as a Fiddler plugin, copy the following files to the Fiddler Inspectors directory (usually C:\Program Files\Fiddler2\Inspectors):
MsgPackExplorer.exe
LsMsgPackFiddlerInspector.dll
LsMsgPack.dll
Restart fiddler and you should see a MsgPack option in the Inspectors list.
Source documentation
ModulesLsMsgPack.dll
This module contains the "parser" and generator of MsgPack Packages. It breaks down the binary file into a hirarchical structure, keeping track of offsets and errors. And it can also be used to generate MsgPack files.
MsgPackExplorer.exe
The main winforms executable, containing a MsgPackExplorer UserControl (so it can easily be integrated into other tools such as Fiddler).
LsMsgPackFiddlerInspector.dll
A tiny wrapper enabling the use of MsgPack Explorer as a Fiddler Inspector.
LsMsgPackUnitTests.dll
Some unit tests on the core LsMsgPack.dll. No full coverage yet, but at least it's a start.
LsMsgPackL.dll & LightUnitTests.dll
A light version of the "parser". The parsing and generating methods are almost identical to the LsMsgPack lib, but with allot of overhead removed that comes with keeping track of offsets, original types and other debugging info. I'm planning to use this version in my projects that use the MsgPack format.
The LightUnitTests are the same as LsMsgPackUnitTests with some tests omitted.
MessagePack for C# (.NET, .NET Core, Unity, Xamarin)
The extremely fast MessagePack serializer for C#.
It is 10x faster than MsgPack-Cli and outperforms other C# serializers. MessagePack for C# also ships with built-in support for LZ4 compression - an extremely fast compression algorithm. Performance is important, particularly in applications like games, distributed computing, microservices, or data caches.
This library is distributed via NuGet. Special Unity support is available, too.
We target .NET Standard 2.0 with special optimizations for .NET Core 2.1+, making it compatible with most reasonably recent .NET runtimes such as Core 2.0 and later, Framework 4.6.1 and later, Mono 5.4 and later and Unity 2018.3 and later.
The library code is pure C# (with Just-In-Time IL code generation on some platforms).
NuGet packages
To install with NuGet, just install the MessagePack package:
Install-Package MessagePack
Install the optional C# analyzers package to get warnings about coding mistakes and automatic fix suggestions to save you time:
Install-Package MessagePackAnalyzer
There are also a range of official and third party Extension Packages available (learn more in our extensions section):
For Unity projects, the releases page provides downloadable .unitypackage files. When using in Unity IL2CPP or Xamarin AOT environments, please carefully read the pre-code generation section.
Define the struct or class to be serialized and annotate it with a [MessagePackObject] attribute.
Annotate members whose values should be serialized (fields as well as properties) with [Key] attributes.
[MessagePackObject]
publicclassMyClass
{
// Key attributes take a serialization index (or string name)// The values must be unique and versioning has to be considered as well.// Keys are described in later sections in more detail.
[Key(0)]
publicintAge { get; set; }
[Key(1)]
publicstringFirstName { get; set; }
[Key(2)]
publicstringLastName { get; set; }
// All fields or properties that should not be serialized must be annotated with [IgnoreMember].
[IgnoreMember]
publicstringFullName { get { returnFirstName+LastName; } }
}
Call MessagePackSerializer.Serialize<T>/Deserialize<T> to serialize/deserialize your object instance.
You can use the ConvertToJson method to get a human readable representation of any MessagePack binary blob.
classProgram
{
staticvoidMain(string[] args)
{
varmc=newMyClass
{
Age=99,
FirstName="hoge",
LastName="huga",
};
// Call Serialize/Deserialize, that's all.byte[] bytes=MessagePackSerializer.Serialize(mc);
MyClassmc2=MessagePackSerializer.Deserialize<MyClass>(bytes);
// You can dump MessagePack binary blobs to human readable json.// Using indexed keys (as opposed to string keys) will serialize to MessagePack arrays,// hence property names are not available.// [99,"hoge","huga"]varjson=MessagePackSerializer.ConvertToJson(bytes);
Console.WriteLine(json);
}
}
Automating definitions for your serializable objects.
Produces compiler warnings upon incorrect attribute use, member accessibility, and more.
If you want to allow a specific custom type (for example, when registering a custom type), put MessagePackAnalyzer.json at the project root and change the Build Action to AdditionalFiles.
MessagePack.Nil is the built-in type representing null/void in MessagePack for C#.
Object Serialization
MessagePack for C# can serialize your own public class or struct types. By default, serializable types must be annotated with the [MessagePackObject] attribute and members with the [Key] attribute. Keys can be either indexes (int) or arbitrary strings. If all keys are indexes, arrays are used for serialization, which offers advantages in performance and binary size. Otherwise, MessagePack maps (dictionaries) will be used.
If you use [MessagePackObject(keyAsPropertyName: true)], then members do not require explicit Key attributes, but string keys will be used.
[MessagePackObject]
publicclassSample1
{
[Key(0)]
publicintFoo { get; set; }
[Key(1)]
publicintBar { get; set; }
}
[MessagePackObject]
publicclassSample2
{
[Key("foo")]
publicintFoo { get; set; }
[Key("bar")]
publicintBar { get; set; }
}
[MessagePackObject(keyAsPropertyName: true)]
publicclassSample3
{
// No need for a Key attributepublicintFoo { get; set; }
// If want to ignore a public member, you can use the IgnoreMember attribute
[IgnoreMember]
publicintBar { get; set; }
}
// [10,20]Console.WriteLine(MessagePackSerializer.SerializeToJson(newSample1 { Foo=10, Bar=20 }));
// {"foo":10,"bar":20}Console.WriteLine(MessagePackSerializer.SerializeToJson(newSample2 { Foo=10, Bar=20 }));
// {"Foo":10}Console.WriteLine(MessagePackSerializer.SerializeToJson(newSample3 { Foo=10, Bar=20 }));
All public instance members (fields as well as properties) will be serialized. If you want to ignore certain public members, annotate the member with a [IgnoreMember] attribute.
Please note that any serializable struct or class must have public accessibility; private and internal structs and classes cannot be serialized!
The default of requiring MessagePackObject annotations is meant to enforce explicitness and therefore may help write more robust code.
Should you use an indexed (int) key or a string key?
We recommend using indexed keys for faster serialization and a more compact binary representation than string keys.
However, the additional information in the strings of string keys can be quite useful when debugging.
When classes change or are extended, be careful about versioning. MessagePackSerializer will initialize members to their default value if a key does not exist in the serialized binary blob, meaning members using reference types can be initialized to null.
If you use indexed (int) keys, the keys should start at 0 and should be sequential. If a later version stops using certain members, you should keep the obsolete members (C# provides an Obsolete attribute to annotate such members) until all other clients had a chance to update and remove their uses of these members as well. Also, when the values of indexed keys "jump" a lot, leaving gaps in the sequence, it will negatively affect the binary size, as null placeholders will be inserted into the resulting arrays. However, you shouldn't reuse indexes of removed members to avoid compatibility issues between clients or when trying to deserialize legacy blobs.
If you do not want to explicitly annotate with the MessagePackObject/Key attributes and instead want to use MessagePack for C# more like e.g. Json.NET, you can make use of the contractless resolver.
publicclassContractlessSample
{
publicintMyProperty1 { get; set; }
publicintMyProperty2 { get; set; }
}
vardata=newContractlessSample { MyProperty1=99, MyProperty2=9999 };
varbin=MessagePackSerializer.Serialize(
data,
MessagePack.Resolvers.ContractlessStandardResolver.Options);
// {"MyProperty1":99,"MyProperty2":9999}Console.WriteLine(MessagePackSerializer.ConvertToJson(bin));
// You can also set ContractlessStandardResolver as the default.// (Global state; Not recommended when writing library code)MessagePackSerializer.DefaultOptions=MessagePack.Resolvers.ContractlessStandardResolver.Options;
// Now serializable...varbin2=MessagePackSerializer.Serialize(data);
If you want to serialize private members as well, you can use one of the *AllowPrivate resolvers.
[MessagePackObject]
publicclassPrivateSample
{
[Key(0)]
intx;
publicvoidSetX(intv)
{
x=v;
}
publicintGetX()
{
returnx;
}
}
vardata=newPrivateSample();
data.SetX(9999);
// You can choose either StandardResolverAllowPrivate// or ContractlessStandardResolverAllowPrivatevarbin=MessagePackSerializer.Serialize(
data,
MessagePack.Resolvers.DynamicObjectResolverAllowPrivate.Options);
If you want to use MessagePack for C# more like a BinaryFormatter with a typeless serialization API, use the typeless resolver and helpers. Please consult the Typeless section.
Resolvers are the way to add specialized support for custom types to MessagePack for C#. Please refer to the Extension point section.
DataContract compatibility
You can use [DataContract] annotations instead of [MessagePackObject] ones. If type is annotated with DataContract, you can use [DataMember] annotations instead of [Key] ones and [IgnoreDataMember] instead of [IgnoreMember].
Then [DataMember(Order = int)] will behave the same as [Key(int)], [DataMember(Name = string)] the same as [Key(string)], and [DataMember] the same as [Key(nameof(member name)].
Using DataContract, e.g. in shared libraries, makes your classes/structs independent from MessagePack for C# serialization. However, it is not supported by the analyzers nor in code generation by the mpc tool. Also, features like UnionAttribute, MessagePackFormatter, SerializationConstructor, etc can not be used. Due to this, we recommend that you use the specific MessagePack for C# annotations when possible.
Serializing readonly/immutable object members (SerializationConstructor)
MessagePack for C# supports serialization of readonly/immutable objects/members. For example, this struct can be serialized and deserialized.
MessagePackSerializer will choose the constructor with the best matched argument list, using argument indexes index for index keys, or parameter names for string keys. If it cannot determine an appropriate constructor, a MessagePackDynamicObjectResolverException: can't find matched constructor parameter exception will be thrown.
You can specify which constructor to use manually with a [SerializationConstructor] annotation.
[MessagePackObject]
publicstructPoint
{
[Key(0)]
publicreadonlyintX;
[Key(1)]
publicreadonlyintY;
[SerializationConstructor]
publicPoint(intx)
{
this.X=x;
this.Y=-1;
}
// If not marked attribute, used this(most matched argument)publicPoint(intx, inty)
{
this.X=x;
this.Y=y;
}
}
C# 9 record types
C# 9.0 record with primary constructor is similar immutable object, also supports serialize/deserialize.
// use key as property name
[MessagePackObject(true)]publicrecordPoint(int X, int Y);
// use property: to set KeyAttribute
[MessagePackObject] public record Point([property:Key(0)] intX, [property: Key(1)] intY);
// Or use explicit properties
[MessagePackObject]
publicrecordPerson
{
[Key(0)]
publicstringFirstName { get; init; }
[Key(1)]
publicstringLastName { get; init; }
}
C# 9 init property setter limitations
When using init property setters in generic classes, a CLR bug prevents our most efficient code generation from invoking the property setter.
As a result, you should avoid using init on property setters in generic classes when using the public-only DynamicObjectResolver/StandardResolver.
When using the DynamicObjectResolverAllowPrivate/StandardResolverAllowPrivate resolver the bug does not apply and you may use init without restriction.
Serialization Callback
Objects implementing the IMessagePackSerializationCallbackReceiver interface will received OnBeforeSerialize and OnAfterDeserialize calls during serialization/deserialization.
MessagePack for C# supports serializing interface-typed and abstract class-typed objects. It behaves like XmlInclude or ProtoInclude. In MessagePack for C# these are called Union. Only interfaces and abstracts classes are allowed to be annotated with Union attributes. Unique union keys are required.
Unions are internally serialized to two-element arrays.
IUnionSampledata=newBarClass { OPQ="FooBar" };
varbin=MessagePackSerializer.Serialize(data);
// Union is serialized to two-length array, [key, object]// [1,["FooBar"]]Console.WriteLine(MessagePackSerializer.ConvertToJson(bin));
Using Union with abstract classes works the same way.
Please be mindful that you cannot reuse the same keys in derived types that are already present in the parent type, as internally a single flat array or map will be used and thus cannot have duplicate indexes/keys.
Dynamic (Untyped) Deserialization
When calling MessagePackSerializer.Deserialize<object> or MessagePackSerializer.Deserialize<dynamic>, any values present in the blob will be converted to primitive values, i.e. bool, char, sbyte, byte, short, int, long, ushort, uint, ulong, float, double, DateTime, string, byte[], object[], IDictionary<object, object>.
// Sample blob.varmodel=newDynamicModel { Name="foobar", Items=new[] { 1, 10, 100, 1000 } };
varblob=MessagePackSerializer.Serialize(model, ContractlessStandardResolver.Options);
// Dynamic ("untyped")vardynamicModel=MessagePackSerializer.Deserialize<dynamic>(blob, ContractlessStandardResolver.Options);
// You can access the data using array/dictionary indexers, as shown aboveConsole.WriteLine(dynamicModel["Name"]); // foobarConsole.WriteLine(dynamicModel["Items"][2]); // 100
Exploring object trees using the dictionary indexer syntax is the fastest option for untyped deserialization, but it is tedious to read and write.
Where performance is not as important as code readability, consider deserializing with ExpandoObject.
Object Type Serialization
StandardResolver and ContractlessStandardResolver can serialize object/anonymous typed objects.
When deserializing, the behavior will be the same as Dynamic (Untyped) Deserialization.
Typeless
The typeless API is similar to BinaryFormatter, as it will embed type information into the blobs, so no types need to be specified explicitly when calling the API.
objectmc=newSandbox.MyClass()
{
Age=10,
FirstName="hoge",
LastName="huga"
};
// Serialize with the typeless APIvarblob=MessagePackSerializer.Typeless.Serialize(mc);
// Blob has embedded type-assembly information.// ["Sandbox.MyClass, Sandbox",10,"hoge","huga"]Console.WriteLine(MessagePackSerializer.ConvertToJson(bin));
// You can deserialize to MyClass again with the typeless API// Note that no type has to be specified explicitly in the Deserialize call// as type information is embedded in the binary blobvarobjModel=MessagePackSerializer.Typeless.Deserialize(bin) asMyClass;
Type information is represented by the MessagePack ext format, type code 100.
MessagePackSerializer.Typeless is a shortcut of Serialize/Deserialize<object>(TypelessContractlessStandardResolver.Instance).
If you want to configure it as the default resolver, you can use MessagePackSerializer.Typeless.RegisterDefaultResolver.
TypelessFormatter can used standalone or combined with other resolvers.
// Replaced `object` uses the typeless resolvervarresolver=MessagePack.Resolvers.CompositeResolver.Create(
new[] { MessagePack.Formatters.TypelessFormatter.Instance },
new[] { MessagePack.Resolvers.StandardResolver.Instance });
publicclassFoo
{
// use Typeless(this field only)
[MessagePackFormatter(typeof(TypelessFormatter))]
publicobjectBar;
}
If a type's name is changed later, you can no longer deserialize old blobs. But you can specify a fallback name in such cases, providing a TypelessFormatter.BindToType function of your own.
Deserializing data from an untrusted source can introduce security vulnerabilities in your application.
Depending on the settings used during deserialization, untrusted data may be able to execute arbitrary code or cause a denial of service attack.
Untrusted data might come from over the network from an untrusted source (e.g. any and every networked client) or can be tampered with by an intermediary when transmitted over an unauthenticated connection, or from a local storage that might have been tampered with, or many other sources. MessagePack for C# does not provide any means to authenticate data or make it tamper-resistant. Please use an appropriate method of authenticating data before deserialization - such as a MAC .
Please be very mindful of these attack scenarios; many projects and companies, and serialization library users in general, have been bitten by untrusted user data deserialization in the past.
When deserializing untrusted data, put MessagePack into a more secure mode by configuring your MessagePackSerializerOptions.Security property:
varoptions=MessagePackSerializerOptions.Standard
.WithSecurity(MessagePackSecurity.UntrustedData);
// Pass the options explicitly for the greatest control.Tobject=MessagePackSerializer.Deserialize<T>(data, options);
// Or set the security level as the default.MessagePackSerializer.DefaultOptions=options;
You should also avoid the Typeless serializer/formatters/resolvers for untrusted data as that opens the door for the untrusted data to potentially deserialize unanticipated types that can compromise security.
The UntrustedData mode merely hardens against some common attacks, but is no fully secure solution in itself.
Performance
Benchmarks comparing MessagePack For C# to other serializers were run on Windows 10 Pro x64 Intel Core i7-6700K 4.00GHz, 32GB RAM. Benchmark code is available here - and their version info.
ZeroFormatter and FlatBuffers have infinitely fast deserializers, so ignore their deserialization performance.
MessagePack for C# uses many techniques to improve performance.
The serializer uses IBufferWriter<byte> rather than System.IO.Stream to reduce memory overhead.
Buffers are rented from pools to reduce allocations, keeping throughput high through reduced GC pressure.
Utilize dynamic code generation and JIT to avoid boxing value types. Use AOT generation on platforms that prohibit JITs.
Cached generated formatters on static generic fields (don't use dictionary-cache because dictionary lookup is overhead). See Resolvers
Heavily tuned dynamic IL code generation and JIT to avoid boxing value types. See DynamicObjectTypeBuilder. Use AOT generation on platforms that prohibit JIT.
Call the Primitive API directly when IL code generation determines target types to be primitive.
Reduce branching of variable length formats when IL code generation knows the target type (integer/string) ranges
Don't use the IEnumerable<T> abstraction to iterate over collections when possible, see: CollectionFormatterBase and derived collection formatters
Use pre-generated lookup tables to reduce checks of mgpack type constraints, see: MessagePackBinary
Avoid string key decoding for lookup maps (string key and use automata based name lookup with inlined IL code generation, see: AutomataDictionary
To encode string keys, use pre-generated member name bytes and fixed sized byte array copies in IL, see: UnsafeMemory.cs
Before creating this library, I implemented a fast serializer with ZeroFormatter#Performance. This is a further evolved implementation. MessagePack for C# is always fast and optimized for all types (primitive, small struct, large object, any collections).
Deserialization Performance for different options
Performance varies depending on the options used. This is a micro benchmark with BenchmarkDotNet. The target object has 9 members (MyProperty1 ~ MyProperty9), values are zero.
Method
Mean
Error
Scaled
Gen 0
Allocated
M IntKey
72.67 ns
NA
1.00
0.0132
56 B
M StringKey
217.95 ns
NA
3.00
0.0131
56 B
M Typeless_IntKey
176.71 ns
NA
2.43
0.0131
56 B
M Typeless_StringKey
378.64 ns
NA
5.21
0.0129
56 B
MsgPackCliMap
1,355.26 ns
NA
18.65
0.1431
608 B
MsgPackCliArray
455.28 ns
NA
6.26
0.0415
176 B
ProtobufNet
265.85 ns
NA
3.66
0.0319
136 B
Hyperion
366.47 ns
NA
5.04
0.0949
400 B
JsonNetString
2,783.39 ns
NA
38.30
0.6790
2864 B
JsonNetStreamReader
3,297.90 ns
NA
45.38
1.4267
6000 B
JilString
553.65 ns
NA
7.62
0.0362
152 B
JilStreamReader
1,408.46 ns
NA
19.38
0.8450
3552 B
ÌntKey, StringKey, Typeless_IntKey, Typeless_StringKey are MessagePack for C# options. All MessagePack for C# options achieve zero memory allocations in the deserialization process. JsonNetString/JilString is deserialized from strings. JsonNetStreamReader/JilStreamReader is deserialized from UTF-8 byte arrays using StreamReader. Deserialization is normally read from Stream. Thus, it will be restored from byte arrays (or Stream) instead of strings.
MessagePack for C# IntKey is the fastest. StringKey is slower than IntKey because matching the character string of property names is required. IntKey works by reading the array length, then for (array length) { binary decode }. StringKey works by reading map length, for (map length) { decode key, lookup key, binary decode }, so it requires an additional two steps (decoding of keys and lookups of keys).
String key is often a useful, contractless, simple replacement of JSON, interoperability with other languages, and more robust versioning. MessagePack for C# is also optimized for string keys as much a possible. First of all, it does not decode UTF-8 byte arrays to full string for matching with the member name; instead it will look up the byte arrays as it is (to avoid decoding costs and extra memory allocations).
And It will try to match each long type (per 8 character, if it is not enough, pad with 0) using automata and inline it when generating IL code.
This also avoids calculating the hash code of byte arrays, and the comparison can be made several times faster using the long type.
This is the sample of decompiled generated deserializer code, decompiled using ILSpy.
If the number of nodes is large, searches will use an embedded binary search.
Extra note, this is serialization benchmark result.
Method
Mean
Error
Scaled
Gen 0
Allocated
IntKey
84.11 ns
NA
1.00
0.0094
40 B
StringKey
126.75 ns
NA
1.51
0.0341
144 B
Typeless_IntKey
183.31 ns
NA
2.18
0.0265
112 B
Typeless_StringKey
193.95 ns
NA
2.31
0.0513
216 B
MsgPackCliMap
967.68 ns
NA
11.51
0.1297
552 B
MsgPackCliArray
284.20 ns
NA
3.38
0.1006
424 B
ProtobufNet
176.43 ns
NA
2.10
0.0665
280 B
Hyperion
280.14 ns
NA
3.33
0.1674
704 B
ZeroFormatter
149.95 ns
NA
1.78
0.1009
424 B
JsonNetString
1,432.55 ns
NA
17.03
0.4616
1944 B
JsonNetStreamWriter
1,775.72 ns
NA
21.11
1.5526
6522 B
JilString
547.51 ns
NA
6.51
0.3481
1464 B
JilStreamWriter
778.78 ns
NA
9.26
1.4448
6066 B
Of course, IntKey is fastest but StringKey also performs reasonably well.
String interning
The msgpack format does not provide for reusing strings in the data stream.
This naturally leads the deserializer to create a new string object for every string encountered,
even if it is equal to another string previously encountered.
When deserializing data that may contain the same strings repeatedly it can be worthwhile
to have the deserializer take a little extra time to check whether it has seen a given string before
and reuse it if it has.
To enable string interning on all string values, use a resolver that specifies StringInterningFormatter
before any of the standard ones, like this:
If you know which fields of a particular type are likely to contain duplicate strings,
you can apply the string interning formatter to just those fields so the deserializer only pays
for the interned string check where it matters most.
Note that this technique requires a [MessagePackObject] or [DataContract] class.
If you are writing your own formatter for some type that contains strings,
you can call on the StringInterningFormatter directly from your formatter as well for the strings.
LZ4 Compression
MessagePack is a fast and compact format but it is not compression. LZ4 is an extremely fast compression algorithm, and using it MessagePack for C# can achieve extremely fast performance as well as extremely compact binary sizes!
MessagePack for C# has built-in LZ4 support. You can activate it using a modified options object and passing it into an API like this:
MessagePackCompression has two modes, Lz4Block and Lz4BlockArray. Neither is a simple binary LZ4 compression, but a special compression integrated into the serialization pipeline, using MessagePack ext code (Lz4BlockArray (98) or Lz4Block (99)). Therefore, it is not readily compatible with compression offered in other languages.
Lz4Block compresses an entire MessagePack sequence as a single LZ4 block. This is the simple compression that achieves best compression ratio, at the cost of copying the entire sequence when necessary to get contiguous memory.
Lz4BlockArray compresses an entire MessagePack sequence as a array of LZ4 blocks. Compressed/decompressed blocks are chunked and thus do not enter the GC's Large-Object-Heap, but the compression ratio is slightly worse.
We recommend to use Lz4BlockArray as the default when using compression.
For compatibility with MessagePack v1.x, use Lz4Block.
Regardless of which LZ4 option is set at the deserialization, both methods can be deserialized. For example, when the Lz4BlockArray option was used, binary data using either Lz4Block and Lz4BlockArray can be deserialized. Neither can be decompressed and hence deserialized when the compression option is set to None.
Attributions
LZ4 compression support is using Milosz Krajewski's lz4net code with some modifications.
Comparison with protobuf, JSON, ZeroFormatter
protobuf-net is major, widely used binary-format library on .NET. I love protobuf-net and respect their great work. But when you use protobuf-net as a general purpose serialization format, you may encounter an annoying issue.
MessagePack's type system can correctly serialize the entire C# type system. This is a strong reason to recommend MessagePack over protobuf.
Protocol Buffers have good IDL and gRPC support. If you want to use IDL, I recommend Google.Protobuf over MessagePack.
JSON is good general-purpose format. It is simple, human-readable and thoroughly-enough specified. Utf8Json - which I created as well - adopts same architecture as MessagePack for C# and avoids encoding/decoding costs as much as possible just like this library does. If you want to know more about binary vs text formats, see Utf8Json/which serializer should be used.
ZeroFormatter is similar as FlatBuffers but specialized to C#, and special in that regard. Deserialization is infinitely fast but the produced binary size is larger. And ZeroFormatter's caching algorithm requires additional memory.
For many common uses, MessagePack for C# would be a better fit.
Hints to achieve maximum performance when using MessagePack for C#
MessagePack for C# prioritizes maximum performance by default. However, there are also some options that sacrifice performance for convenience.
Use indexed keys instead of string keys (Contractless)
The Deserialization Performance for different options section shows the results of indexed keys (IntKey) vs string keys (StringKey) performance. Indexed keys serialize the object graph as a MessagePack array. String keys serializes the object graph as a MessagePack map.
IntKey is always fast in both serialization and deserialization because it does not have to handle and lookup key names, and always has the smaller binary size.
StringKey is often a useful, contractless, simple replacement for JSON, interoperability with other languages with MessagePack support, and less error prone versioning. But to achieve maximum performance, use IntKey.
Create own custom composite resolver
CompositeResolver.Create is an easy way to create composite resolvers. But formatter lookups have some overhead. If you create a custom resolver (or use StaticCompositeResolver.Instance), you can avoid this overhead.
NOTE: If you are creating a library, recommend using the above custom resolver instead of CompositeResolver.Create. Also, libraries must not use StaticCompositeResolver - as it is global state - to avoid compatibility issues.
Use native resolvers
By default, MessagePack for C# serializes GUID as string. This is much slower than the native .NET format GUID. The same applies to Decimal. If your application makes heavy use of GUID or Decimal and you don't have to worry about interoperability with other languages, you can replace them with the native serializers NativeGuidResolver and NativeDecimalResolver respectively.
Also, DateTime is serialized using the MessagePack timestamp format. By using the NativeDateTimeResolver, it is possible to maintain Kind and perform faster serialization.
Be careful when copying buffers
MessagePackSerializer.Serialize returns byte[] in default. The final byte[] is copied from an internal buffer pool. That is an extra cost. You can use IBufferWriter<T> or the Stream API to write to buffers directly. If you want to use a buffer pool outside of the serializer, you should implement custom IBufferWriter<byte> or use an existing one such as Sequence<T> from the Nerdbank.Streams package.
During deserialization, MessagePackSerializer.Deserialize(ReadOnlyMemory<byte> buffer) is better than the Deserialize(Stream stream) overload. This is because the Stream API version starts by reading the data, generating a ReadOnlySequence<byte>, and only then starts the deserialization.
Choosing compression
Compression is generally effective when there is duplicate data. In MessagePack, arrays containing objects using string keys (Contractless) can be compressed efficiently because compression can be applied to many duplicate property names. Indexed keys compression is not as effectively compressed as string keys, but indexed keys are smaller in the first place.
This is some example benchmark performance data;
Serializer
Mean
DataSize
IntKey
2.941 us
469.00 B
IntKey(Lz4)
3.449 us
451.00 B
StringKey
4.340 us
1023.00 B
StringKey(Lz4)
5.469 us
868.00 B
IntKey(Lz4) is not as effectively compressed, but performance is still somewhat degraded. On the other hand, StringKey can be expected to have a sufficient effect on the binary size. However, this is just an example. Compression can be quite effective depending on the data, too, or have little effect other than slowing down your program. There are also cases in which well-compressible data exists in the values (such as long strings, e.g. containing HTML data with many repeated HTML tags). It is important to verify the actual effects of compression on a case by case basis.
Extensions
MessagePack for C# has extension points that enable you to provide optimal serialization support for custom types. There are official extension support packages.
The MessagePack.ReactiveProperty package adds support for types of the ReactiveProperty library. It adds ReactiveProperty<>, IReactiveProperty<>, IReadOnlyReactiveProperty<>, ReactiveCollection<>, Unit serialization support. It is useful for save viewmodel state.
The MessagePack.UnityShims package provides shims for Unity's standard structs (Vector2, Vector3, Vector4, Quaternion, Color, Bounds, Rect, AnimationCurve, Keyframe, Matrix4x4, Gradient, Color32, RectOffset, LayerMask, Vector2Int, Vector3Int, RangeInt, RectInt, BoundsInt) and corresponding formatters. It can enable proper communication between servers and Unity clients.
After installation, extension packages must be enabled, by creating composite resolvers. Here is an example showing how to enable all extensions.
// Set extensions to default resolver.varresolver=MessagePack.Resolvers.CompositeResolver.Create(
// enable extension packages firstReactivePropertyResolver.Instance,
MessagePack.Unity.Extension.UnityBlitResolver.Instance,
MessagePack.Unity.UnityResolver.Instance,
// finally use standard (default) resolverStandardResolver.Instance
);
varoptions=MessagePackSerializerOptions.Standard.WithResolver(resolver);
// Pass options every time or set as defaultMessagePackSerializer.DefaultOptions=options;
The MessagePackSerializer class is the entry point of MessagePack for C#. Static methods make up the main API of MessagePack for C#.
API
Description
Serialize<T>
Serializes an object graph to a MessagePack binary blob. Async variant for Stream available. Non-generic overloads available.
Deserialize<T>
Deserializes a MessagePack binary to an object graph. Async variant for Stream available. Non-generic overloads available.
SerializeToJson
Serialize a MessagePack-compatible object graph to JSON instead of MessagePack. Useful for debugging.
ConvertToJson
Convert MessagePack binary to JSON. Useful for debugging.
ConvertFromJson
Convert JSON to a MessagePack binary.
The MessagePackSerializer.Typeless class offers most of the same APIs as above, but removes all type arguments from the API, forcing serialization to include the full type name of the root object. It uses the TypelessContractlessStandardResolver. Consider the result to be a .NET-specific MessagePack binary that isn't readily compatible with MessagePack deserializers in other runtimes.
MessagePack for C# fundamentally serializes using IBufferWriter<byte> and deserializes using ReadOnlySequence<byte> or Memory<byte>. Method overloads are provided to conveniently use it with common buffer types and the .NET Stream class, but some of these convenience overloads require copying buffers once and therefore have a certain overhead.
The high-level API uses a memory pool internally to avoid unnecessary memory allocation. If result size is under 64K, it allocates GC memory only for the return bytes.
Each serialize/deserialize method takes an optional MessagePackSerializerOptions parameter which can be used to specify a custom IFormatterResolver to use or to activate LZ4 compression support.
Multiple MessagePack structures on a single Stream
To deserialize a Stream that contains multiple consecutive MessagePack data structures,
you can use the MessagePackStreamReader class to efficiently identify the ReadOnlySequence<byte>
for each data structure and deserialize it. For example:
staticasyncTask<List<T>> DeserializeListFromStreamAsync<T>(Streamstream, CancellationTokencancellationToken)
{
vardataStructures=newList<T>();
using (varstreamReader=newMessagePackStreamReader(stream))
{
while (awaitstreamReader.ReadAsync(cancellationToken) isReadOnlySequence<byte> msgpack)
{
dataStructures.Add(MessagePackSerializer.Deserialize<T>(msgpack, cancellationToken: cancellationToken));
}
}
returndataStructures;
}
Low-Level API (IMessagePackFormatter<T>)
The IMessagePackFormatter<T> interface is responsible for serializing a unique type. For example Int32Formatter : IMessagePackFormatter<Int32> represents Int32 MessagePack serializer.
Many built-in formatters exists under MessagePack.Formatters. Your custom types are usually automatically supported with the built-in type resolvers that generate new IMessagePackFormatter<T> types on-the-fly using dynamic code generation. See our AOT code generation support for platforms that do not support this.
However, some types - especially those provided by third party libraries or the runtime itself - cannot be appropriately annotated, and contractless serialization would produce inefficient or even wrong results.
To take more control over the serialization of such custom types, write your own IMessagePackFormatter<T> implementation.
Here is an example of such a custom formatter implementation. Note its use of the primitive API that is described in the next section.
/// <summary>Serializes a <seecref="FileInfo" /> by its full path as a string.</summary>publicclassFileInfoFormatter : IMessagePackFormatter<FileInfo>
{
publicvoidSerialize(
refMessagePackWriterwriter, FileInfovalue, MessagePackSerializerOptionsoptions)
{
if (value==null)
{
writer.WriteNil();
return;
}
writer.WriteString(value.FullName);
}
publicFileInfoDeserialize(
refMessagePackReaderreader, MessagePackSerializerOptionsoptions)
{
if (reader.TryReadNil())
{
returnnull;
}
options.Security.DepthStep(refreader);
varpath=reader.ReadString();
reader.Depth--;
returnnewFileInfo(path);
}
}
The DepthStep and Depth-- statements provide a level of security while deserializing untrusted data
that might otherwise be able to execute a denial of service attack by sending MessagePack data that would
deserialize into a very deep object graph leading to a StackOverflowException that would crash the process.
This pair of statements should surround the bulk of any IMessagePackFormatter<T>.Deserialize method.
Important: A message pack formatter must read or write exactly one data structure.
In the above example we just read/write a string. If you have more than one element to write out,
you must precede it with a map or array header. You must read the entire map/array when deserializing.
For example:
publicclassMySpecialObjectFormatter : IMessagePackFormatter<MySpecialObject>
{
publicvoidSerialize(
refMessagePackWriterwriter, MySpecialObjectvalue, MessagePackSerializerOptionsoptions)
{
if (value==null)
{
writer.WriteNil();
return;
}
writer.WriteArrayHeader(2);
writer.WriteString(value.FullName);
writer.WriteString(value.Age);
}
publicMySpecialObjectDeserialize(
refMessagePackReaderreader, MessagePackSerializerOptionsoptions)
{
if (reader.TryReadNil())
{
returnnull;
}
options.Security.DepthStep(refreader);
stringfullName=null;
intage=0;
// Loop over *all* array elements independently of how many we expect,// since if we're serializing an older/newer version of this object it might// vary in number of elements that were serialized, but the contract of the formatter// is that exactly one data structure must be read, regardless.// Alternatively, we could check that the size of the array/map is what we expect// and throw if it is not.intcount=reader.ReadArrayHeader();
for (inti=0; i<count; i++)
{
switch (i)
{
case0:
fullName=reader.ReadString();
break;
case1:
age=reader.ReadInt32();
break;
default:
reader.Skip();
break;
}
}
reader.Depth--;
returnnewMySpecialObject(fullName, age);
}
}
Your custom formatters must be discoverable via some IFormatterResolver. Learn more in our resolvers section.
Primitive API (MessagePackWriter, MessagePackReader)
The MessagePackWriter and MessagePackReader structs make up the lowest-level API. They read and write the primitives types defined in the MessagePack specification.
MessagePackReader
A MessagePackReader can efficiently read from ReadOnlyMemory<byte> or ReadOnlySequence<byte> without any allocations, except to allocate a new string as required by the ReadString() method. All other methods return either value structs or ReadOnlySequence<byte> slices for extensions/arrays.
Reading directly from ReadOnlySequence<byte> means the reader can directly consume some modern high performance APIs such as PipeReader.
Method
Description
Skip
Advances the reader's position past the current value. If the value is complex (e.g. map, array) the entire structure is skipped.
Read*
Read and return a value whose type is named by the method name from the current reader position. Throws if the expected type does not match the actual type. When reading numbers, the type need not match the binary-specified type exactly. The numeric value will be coerced into the desired type or throw if the integer type is too small for a large value.
TryReadNil
Advances beyond the current value if the current value is nil and returns true; otherwise leaves the reader's position unchanged and returns false.
ReadBytes
Returns a slice of the input sequence representing the contents of a byte[], and advances the reader.
ReadStringSequence
Returns a slice of the input sequence representing the contents of a string without decoding it, and advances the reader.
Clone
Creates a new MessagePackReader with the specified input sequence and the same settings as the original reader.
CreatePeekReader
Creates a new reader with the same position as this one, allowing the caller to "read ahead" without impacting the original reader's position.
NextCode
Reads the low-level MessagePack byte that describes the type of the next value. Does not advance the reader. See MessagePack format of first byte. Its static class has ToMessagePackType and ToFormatName utility methods. MessagePackRange means Min-Max fix range of MessagePack format.
Other methods and properties as described by the .xml doc comment file and Intellisense.
The MessagePackReader is capable of automatically interpreting both the old and new MessagePack spec.
MessagePackWriter
A MessagePackWriter writes to a given instance of IBufferWriter<byte>. Several common implementations of this exist, allowing zero allocations and minimal buffer copies while writing directly to several I/O APIs including PipeWriter.
The MessagePackWriter writes the new MessagePack spec by default, but can write MessagePack compatible with the old spec by setting the OldSpec property to true.
Method
Description
Clone
Creates a new MessagePackWriter with the specified underlying IBufferWriter<byte> and the same settings as the original writer.
Flush
Writes any buffered bytes to the underlying IBufferWriter<byte>.
WriteNil
Writes the MessagePack equivalent of .NET's null value.
Write
Writes any MessagePack primitive value in the most compact form possible. Has overloads for every primitive type defined by the MessagePack spec.
Write*IntType*
Writes an integer value in exactly the MessagePack type specified, even if a more compact format exists.
WriteMapHeader
Introduces a map by specifying the number of key=value pairs it contains.
WriteArrayHeader
Introduces an array by specifying the number of elements it contains.
WriteExtensionFormat
Writes the full content of an extension value including length, type code and content.
WriteExtensionFormatHeader
Writes just the header (length and type code) of an extension value.
WriteRaw
Copies the specified bytes directly to the underlying IBufferWriter<byte> without any validation.
(others)
Other methods and properties as described by the .xml doc comment file and Intellisense.
DateTime is serialized to MessagePack Timestamp format, it serialize/deserialize UTC and loses Kind info and requires that MessagePackWriter.OldSpec == false.
If you use the NativeDateTimeResolver, DateTime values will be serialized using .NET's native Int64 representation, which preserves Kind info but may not be interoperable with non-.NET platforms.
Main Extension Point (IFormatterResolver)
An IFormatterResolver is storage of typed serializers. The MessagePackSerializer API accepts a MessagePackSerializerOptions object which specifies the IFormatterResolver to use, allowing customization of the serialization of complex types.
Resolver Name
Description
BuiltinResolver
Builtin primitive and standard classes resolver. It includes primitive(int, bool, string...) and there nullable, array and list. and some extra builtin types(Guid, Uri, BigInteger, etc...).
StandardResolver
Composited resolver. It resolves in the following order builtin -> attribute -> dynamic enum -> dynamic generic -> dynamic union -> dynamic object -> dynamic object fallback. This is the default of MessagePackSerializer.
Same as StandardResolver but allow serialize/deserialize private members.
ContractlessStandardResolverAllowPrivate
Same as ContractlessStandardResolver but allow serialize/deserialize private members.
PrimitiveObjectResolver
MessagePack primitive object resolver. It is used fallback in object type and supports bool, char, sbyte, byte, short, int, long, ushort, uint, ulong, float, double, DateTime, string, byte[], ICollection, IDictionary.
DynamicObjectTypeFallbackResolver
Serialize is used type in from object type, deserialize is used PrimitiveObjectResolver.
AttributeFormatterResolver
Get formatter from [MessagePackFormatter] attribute.
CompositeResolver
Composes several resolvers and/or formatters together in an ordered list, allowing reuse and overriding of behaviors of existing resolvers and formatters.
NativeDateTimeResolver
Serialize by .NET native DateTime binary format. It keeps DateTime.Kind that loses by standard(MessagePack timestamp) format.
NativeGuidResolver
Serialize by .NET native Guid binary representation. It is faster than standard(string) representation.
NativeDecimalResolver
Serialize by .NET native decimal binary representation. It is faster than standard(string) representation.
DynamicEnumResolver
Resolver of enum and there nullable, serialize there underlying type. It uses dynamic code generation to avoid boxing and boostup performance serialize there name.
DynamicEnumAsStringResolver
Resolver of enum and there nullable. It uses reflection call for resolve nullable at first time.
DynamicGenericResolver
Resolver of generic type(Tuple<>, List<>, Dictionary<,>, Array, etc). It uses reflection call for resolve generic argument at first time.
DynamicUnionResolver
Resolver of interface marked by UnionAttribute. It uses dynamic code generation to create dynamic formatter.
DynamicObjectResolver
Resolver of class and struct made by MessagePackObjectAttribute. It uses dynamic code generation to create dynamic formatter.
DynamicContractlessObjectResolver
Resolver of all classes and structs. It does not needs MessagePackObjectAttribute and serialized key as string(same as marked [MessagePackObject(true)]).
DynamicObjectResolverAllowPrivate
Same as DynamicObjectResolver but allow serialize/deserialize private members.
DynamicContractlessObjectResolverAllowPrivate
Same as DynamicContractlessObjectResolver but allow serialize/deserialize private members.
TypelessObjectResolver
Used for object, embed .NET type in binary by ext(100) format so no need to pass type in deserialization.
TypelessContractlessStandardResolver
Composited resolver. It resolves in the following order nativedatetime -> builtin -> attribute -> dynamic enum -> dynamic generic -> dynamic union -> dynamic object -> dynamiccontractless -> typeless. This is the default of MessagePackSerializer.Typeless
Each instance of MessagePackSerializer accepts only a single resolver. Most object graphs will need more than one for serialization, so composing a single resolver made up of several is often required, and can be done with the CompositeResolver as shown below:
// Do this once and store it for reuse.varresolver=MessagePack.Resolvers.CompositeResolver.Create(
// resolver custom types firstReactivePropertyResolver.Instance,
MessagePack.Unity.Extension.UnityBlitResolver.Instance,
MessagePack.Unity.UnityResolver.Instance,
// finally use standard resolverStandardResolver.Instance
);
varoptions=MessagePackSerializerOptions.Standard.WithResolver(resolver);
// Each time you serialize/deserialize, specify the options:byte[] msgpackBytes=MessagePackSerializer.Serialize(myObject, options);
TmyObject2=MessagePackSerializer.Deserialize<MyObject>(msgpackBytes, options);
A resolver can be set as default with MessagePackSerializer.DefaultOptions = options, but WARNING:
When developing an application where you control all MessagePack-related code it may be safe to rely on this mutable static to control behavior.
For all other libraries or multi-purpose applications that use MessagePackSerializer you should explicitly specify the MessagePackSerializerOptions to use with each method invocation to guarantee your code behaves as you expect even when sharing an AppDomain or process with other MessagePack users that may change this static property.
Here is sample of use DynamicEnumAsStringResolver with DynamicContractlessObjectResolver (It is Json.NET-like lightweight setting.)
// composite same as StandardResolvervarresolver=MessagePack.Resolvers.CompositeResolver.Create(
MessagePack.Resolvers.BuiltinResolver.Instance,
MessagePack.Resolvers.AttributeFormatterResolver.Instance,
// replace enum resolverMessagePack.Resolvers.DynamicEnumAsStringResolver.Instance,
MessagePack.Resolvers.DynamicGenericResolver.Instance,
MessagePack.Resolvers.DynamicUnionResolver.Instance,
MessagePack.Resolvers.DynamicObjectResolver.Instance,
MessagePack.Resolvers.PrimitiveObjectResolver.Instance,
// final fallback(last priority)MessagePack.Resolvers.DynamicContractlessObjectResolver.Instance
);
If you want to make an extension package, you should write both a formatter and resolver
for easier consumption.
Here is sample of a resolver:
publicclassSampleCustomResolver : IFormatterResolver
{
// Resolver should be singleton.publicstaticreadonlyIFormatterResolverInstance=newSampleCustomResolver();
privateSampleCustomResolver()
{
}
// GetFormatter<T>'s get cost should be minimized so use type cache.publicIMessagePackFormatter<T> GetFormatter<T>()
{
returnFormatterCache<T>.Formatter;
}
privatestaticclassFormatterCache<T>
{
publicstaticreadonlyIMessagePackFormatter<T> Formatter;
// generic's static constructor should be minimized for reduce type generation size!// use outer helper method.staticFormatterCache()
{
Formatter= (IMessagePackFormatter<T>)SampleCustomResolverGetFormatterHelper.GetFormatter(typeof(T));
}
}
}
internalstaticclassSampleCustomResolverGetFormatterHelper
{
// If type is concrete type, use type-formatter mapstaticreadonlyDictionary<Type, object> formatterMap=newDictionary<Type, object>()
{
{typeof(FileInfo), newFileInfoFormatter()}
// add more your own custom serializers.
};
internalstaticobjectGetFormatter(Typet)
{
objectformatter;
if (formatterMap.TryGetValue(t, outformatter))
{
returnformatter;
}
// If type can not get, must return null for fallback mechanism.returnnull;
}
}
MessagePackFormatterAttribute
MessagePackFormatterAttribute is a lightweight extension point of class, struct, interface, enum and property/field. This is like Json.NET's JsonConverterAttribute. For example, serialize private field, serialize x10 formatter.
Formatter is retrieved by AttributeFormatterResolver, it is included in StandardResolver.
IgnoreFormatter
IgnoreFormatter<T> is lightweight extension point of class and struct. If there exists types that can't be serialized, you can register IgnoreFormatter<T> that serializes those to nil/null.
// CompositeResolver can set custom formatter.varresolver=MessagePack.Resolvers.CompositeResolver.Create(
newIMessagePackFormatter[]
{
// for example, register reflection infos (can not serialize)newIgnoreFormatter<MethodBase>(),
newIgnoreFormatter<MethodInfo>(),
newIgnoreFormatter<PropertyInfo>(),
newIgnoreFormatter<FieldInfo>()
},
newIFormatterResolver[]
{
ContractlessStandardResolver.Instance
});
Reserved Extension Types
MessagePack for C# already used some MessagePack extension type codes, be careful to avoid using the same ext code for other purposes.
Range
Reserved for
[-128, -1]
Reserved by the msgpack spec for predefined types
[30, 120)
Reserved for this library's use to support common types in .NET
This leaves the following ranges for your use:
[0, 30)
[120, 127]
Within the reserved ranges, this library defines or implements extensions that use these type codes:
Code
Type
Use by
-1
DateTime
MessagePack-spec reserved for timestamp
30
Vector2[]
for Unity, UnsafeBlitFormatter
31
Vector3[]
for Unity, UnsafeBlitFormatter
32
Vector4[]
for Unity, UnsafeBlitFormatter
33
Quaternion[]
for Unity, UnsafeBlitFormatter
34
Color[]
for Unity, UnsafeBlitFormatter
35
Bounds[]
for Unity, UnsafeBlitFormatter
36
Rect[]
for Unity, UnsafeBlitFormatter
37
Int[]
for Unity, UnsafeBlitFormatter
38
Float[]
for Unity, UnsafeBlitFormatter
39
Double[]
for Unity, UnsafeBlitFormatter
98
All
MessagePackCompression.Lz4BlockArray
99
All
MessagePackCompression.Lz4Block
100
object
TypelessFormatter
Unity support
Unity lowest supported version is 2018.3, API Compatibility Level supports both .NET 4.x and .NET Standard 2.0.
You can install the unitypackage from the releases page.
If your build targets .NET Framework 4.x and runs on mono, you can use it as is.
But if your build targets IL2CPP, you can not use Dynamic***Resolver, so it is required to use pre-code generation. Please see pre-code generation section.
MessagePack for C# includes some additional System.*.dll libraries that originally provides in NuGet. They are located under Plugins. If other packages use these libraries (e.g. Unity Collections package using System.Runtime.CompilerServices.Unsafe.dll), to avoid conflicts, please delete the DLL under Plugins.
Currently CompositeResolver.Create does not work on IL2CPP, so it is recommended to use StaticCompositeResolver.Instance.Register instead.
In Unity, MessagePackSerializer can serialize Vector2, Vector3, Vector4, Quaternion, Color, Bounds, Rect, AnimationCurve, Keyframe, Matrix4x4, Gradient, Color32, RectOffset, LayerMask, Vector2Int, Vector3Int, RangeInt, RectInt, BoundsInt and their nullable, array and list types with the built-in extension UnityResolver. It is included in StandardResolver by default.
MessagePack for C# has an additional unsafe extension. UnsafeBlitResolver is special resolver for extremely fast but unsafe serialization/deserialization of struct arrays.
x20 faster Vector3[] serialization than native JsonUtility. If use UnsafeBlitResolver, serialization uses a special format (ext:typecode 30~39) for Vector2[], Vector3[], Quaternion[], Color[], Bounds[], Rect[]. If use UnityBlitWithPrimitiveArrayResolver, it supports int[], float[], double[] too. This special feature is useful for serializing Mesh (many Vector3[]) or many transform positions.
If you want to use unsafe resolver, register UnityBlitResolver or UnityBlitWithPrimitiveArrayResolver.
The MessagePack.UnityShims NuGet package is for .NET server-side serialization support to communicate with Unity. It includes shims for Vector3 etc and the Safe/Unsafe serialization extension.
If you want to share a class between Unity and a server, you can use SharedProject or Reference as Link or a glob reference (with LinkBase), etc. Anyway, you need to share at source-code level. This is a sample project structure using a glob reference (recommended).
[MessagePack](not dll/NuGet, use MessagePack.Unity.unitypackage's sourcecode)
AOT Code Generation (support for Unity/Xamarin)
By default, MessagePack for C# serializes custom objects by generating IL on the fly at runtime to create custom, highly tuned formatters for each type. This code generation has a minor upfront performance cost.
Because strict-AOT environments such as Xamarin and Unity IL2CPP forbid runtime code generation, MessagePack provides a way for you to run a code generator ahead of time as well.
Note: When using Unity, dynamic code generation only works when targeting .NET Framework 4.x + mono runtime.
For all other Unity targets, AOT is required.
If you want to avoid the upfront dynamic generation cost or you need to run on Xamarin or Unity, you need AOT code generation. mpc (MessagePackCompiler) is the code generator of MessagePack for C#. mpc uses Roslyn to analyze source code.
First of all, mpc requires .NET Core 3 Runtime. The easiest way to acquire and run mpc is as a dotnet tool.
Installing it as a local tool allows you to include the tools and versions that you use in your source control system. Run these commands in the root of your repo:
dotnet new tool-manifest
dotnet tool install MessagePack.Generator
Check in your .config\dotnet-tools.json file. On another machine you can "restore" your tool using the dotnet tool restore command.
Once you have the tool installed, simply invoke using dotnet mpc within your repo:
dotnet mpc --help
Alternatively, you can download mpc from the releases page, that includes platform native binaries (that don't require a separate dotnet runtime).
Usage: mpc [options...]
Options:
-i, -input <String> Input path to MSBuild project file or the directory containing Unity source files. (Required)
-o, -output <String> Output file path(.cs) or directory(multiple generate file). (Required)
-c, -conditionalSymbol <String> Conditional compiler symbols, split with ','. (Default: null)
-r, -resolverName <String> Set resolver name. (Default: GeneratedResolver)
-n, -namespace <String> Set namespace root name. (Default: MessagePack)
-m, -useMapMode <Boolean> Force use map mode serialization. (Default: False)
-ms, -multipleIfDirectiveOutputSymbols <String> Generate #if-- files by symbols, split with ','. (Default: null)
mpc targets C# code with [MessagePackObject] or [Union] annotations.
In Unity, you can use MessagePack CodeGen windows at Windows -> MessagePack -> CodeGenerator.
Install the .NET Core runtime, install mpc (as a .NET Core Tool as described above), and execute dotnet mpc. Currently this tool is experimental so please tell me your opinion.
In Xamarin, you can install the the MessagePack.MSBuild.Tasks NuGet package into your projects to pre-compile fast serialization code and run in environments where JIT compilation is not allowed.
RPC
MessagePack advocated MessagePack RPC, but work on it has stopped and it is not widely used.
MagicOnion
I've created a gRPC based MessagePack HTTP/2 RPC streaming framework called MagicOnion. gRPC usually communicates with Protocol Buffers using IDL. But MagicOnion uses MessagePack for C# and does not need IDL. When communicating C# to C#, schemaless (or rather C# classes as schema) is better than using IDL.
Yoshifumi Kawai (a.k.a. neuecc) is a software developer in Japan.
He is the Director/CTO at Grani, Inc.
Grani is a mobile game developer company in Japan and well known for using C#.
He is awarding Microsoft MVP for Visual C# since 2011.
He is known as the creator of UniRx (Reactive Extensions for Unity)
msgpack11 is a tiny MsgPack library for C++11, providing MsgPack parsing and serialization.
This library is inspired by json11.
The API of msgpack11 is designed to be similar with json11.
Installation
Using CMake
git clone [email protected]:ar90n/msgpack11.git
mkdir build
cd build
cmake ../msgpack11
make && make install
Using Buck
git clone [email protected]:ar90n/msgpack11.git
cd msgpack11
buck build :msgpack11
A modern (c++17 required) implementation of the msgpack spec.
Msgpack is a binary serialization specification. It allows you to save and load application objects like classes and structs over networks, to files, and between programs and even different languages.
Check out this blog for my rational creating this library.
Features
Fast and compact
Full test coverage
Easy to use
Automatic type handling
Open source MIT license
Easy error handling
Single Header only template library
Want to use this library? Just #include the header and you're good to go. Its less than 1000 lines of code.
Cereal style packaging
Easily pack objects into byte arrays using a pack free function:
structPerson {
std::string name;
uint16_t age;
std::vector<std::string> aliases;
template<classT>
voidmsgpack(T &pack) {
pack(name, age, aliases);
}
};
intmain() {
auto person = Person{"John", 22, {"Ripper", "Silverhand"}};
auto data = msgpack::pack(person); // Pack your objectauto john = msgpack::unpack<Person>(data.data()); // Unpack it
}
The msgpack spec allows for additional types to be enumerated as Extensions. If reasonable use cases come about for this feature then it may be added.
Name/value pairs
The msgpack spec uses the 'map' type differently than this library. This library implements maps in which key/value pairs must all have the same value types.
Endian conversion shortcuts
On platforms that already hold types in big endian, the serialization could be optimized using type traits.
MessagePack is an efficient binary serialization
format, which lets you exchange data among multiple languages like JSON,
except that it's faster and smaller. Small integers are encoded into a
single byte and short strings require only one extra byte in
addition to the strings themselves.
clojure-msgpack is a lightweight and simple library for converting
between native Clojure data structures and MessagePack byte formats.
clojure-msgpack only depends on Clojure itself; it has no third-party
dependencies.
Installation
Usage
Basic
pack: Serialize object as a sequence of java.lang.Bytes.
clojure-msgpack provides a streaming API for situations where it is more
convenient or efficient to work with byte streams instead of fixed byte arrays
(e.g. size of object is not known ahead of time).
The streaming counterpart to msgpack.core/pack is msgpack.core/pack-stream
which returns nil and accepts either
java.io.OutputStream
or
java.io.DataOutput
as an additional argument.
Serializing a value of unrecognized type will fail with IllegalArgumentException. See Application types if you want to register your own types.
Clojure types
Some native Clojure types don't have an obvious MessagePack counterpart. We can
serialize them as Extended types. To enable automatic conversion of these
types, load the clojure-extensions library.
(msg/pack:hello)
; => IllegalArgumentException No implementation of method: :pack-stream of; protocol: #'msgpack.core/Packable found for class: clojure.lang.Keyword; clojure.core/-cache-protocol-fn (core _deftype.clj:544)
Note: No error is thrown if an unpacked value is reserved under the old spec
but defined under the new spec. We always deserialize something if we can
regardless of compatibility-mode.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
MessagePack is a binary-based JSON-like serialization library.
MessagePack for D is a pure D implementation of MessagePack.
Features
Small size and High performance
Zero copy serialization / deserialization
Streaming deserializer for non-contiguous IO situation
Supports D features (Ranges, Tuples, real type)
Note: The real type is only supported in D.
Don't use the real type when communicating with other programming languages.
Note that Unpacker will raise an exception if a loss of precision occurs.
Current Limitations
No circular references support
If you want to use the LDC compiler, you need at least version 0.15.2 beta2
Install
Use dub to add it as a dependency:
% dub install msgpack-d
Usage
Example code can be found in the example directory.
msgpack-d is very simple to use. Use pack for serialization, and unpack for deserialization:
importstd.file;
import msgpack;
structS { int x; float y; string z; }
voidmain()
{
S input = S(10, 25.5, "message");
// serialize dataubyte[] inData = pack(input);
// write data to a file
write("file.dat", inData);
// read data from a fileubyte[] outData = cast(ubyte[])read("file.dat");
// unserialize the data
S target = outData.unpack!S();
// verify data is the sameassert(target.x == input.x);
assert(target.y == input.y);
assert(target.z == input.z);
}
Feature: Skip serialization/deserialization of a specific field.
Use the @nonPacked attribute:
structUser
{
string name;
@nonPacked int level; // pack / unpack will ignore the 'level' field
}
Feature: Use your own serialization/deserialization routines for custom class and struct types.
msgpack-d provides the functions registerPackHandler / registerUnpackHandler to allow you
to use custom routines during the serialization or deserialization of user-defined class and struct types.
This feature is especially useful when serializing a derived class object when that object is statically
typed as a base class object.
For example:
classDocument { }
classXmlDocument : Document
{
this() { }
this(string name) { this.name = name; }
string name;
}
voidxmlPackHandler(ref Packer p, ref XmlDocument xml)
{
p.pack(xml.name);
}
voidxmlUnpackHandler(ref Unpacker u, ref XmlDocument xml)
{
u.unpack(xml.name);
}
voidmain()
{
/// Register the 'xmlPackHandler' and 'xmlUnpackHandler' routines for/// XmlDocument object instances.
registerPackHandler!(XmlDocument, xmlPackHandler);
registerUnpackHandler!(XmlDocument, xmlUnpackHandler);
/// Now we can serialize/deserialize XmlDocument object instances via a/// base class reference.Document doc = new XmlDocument("test.xml");
auto data = pack(doc);
XmlDocument xml = unpack!XmlDocument(data);
assert(xml.name =="test.xml"); // xml.name is "test.xml"
}
In addition, here is also a method using @serializedAs attribute:
QMsgPack is a simple and powerful Delphi & C++ Builder implementation for messagepack protocol.
QMsgPack is a part of QDAC 3.0,Source code hosted in Sourceforge(http://sourceforge.net/p/qdac3).
Feathers
· Full types support,include messagepack extension type
· Full open source,free for used in ANY PURPOSE
· Quick and simple interface
· RTTI support include
Install
QMsgPack is not a desgin time package.So just place QMsgPack files into search path and add to your project.
Only in packing. Atoms are packed as binaries. Default value is pack.
Otherwise, any term including atoms throws badarg.
{known_atoms, [atom()]}
Both in packing and unpacking. In packing, if an atom is in this list
a binary is encoded as a binary. In unpacking, msgpacked binaries are
decoded as atoms with erlang:binary_to_existing_atom/2 with encoding
utf8. Default value is an empty list.
Even if allow_atom is none, known atoms are packed.
{unpack_str, as_binary|as_list}
A switch to choose decoded term style of str type when unpacking.
Only available at new spec. Default is as_list.
mode as_binary as_list
-----------+------------+-------
bin binary() binary()
str binary() string()
{validate_string, boolean()}
Only in unpacking, UTF-8 validation at unpacking from str type will
be enabled. Default value is false.
{pack_str, from_binary|from_list|none}
A switch to choose packing of string() when packing. Only available
at new spec. Default is from_list for symmetry with unpack_str
option.
mode from_list from_binary none
-----------+------------+--------------+-----------------
binary() bin str*/bin bin
string() str*/array array of int array of int
list() array array array
But the default option pays the cost of performance for symmetry. If
the overhead of UTF-8 validation is unacceptable, choosing none as
the option would be the best.
* Tries to pack as str if it is a valid string().
{map_format, map|jiffy|jsx}
Both at packing and unpacking. Default value is map.
At both. The default behaviour in case of facing ext data at decoding
is to ignore them as its length is known.
Now msgpack-erlang supports ext type. Now you can serialize everything
with your original (de)serializer. That will enable us to handle
erlang- native types like pid(), ref() contained in tuple(). See
test/msgpack_ext_example_tests.erl for example code.
The Float type of Message Pack represents IEEE 754 floating point number, so it includes Nan and Infinity.
In unpacking, msgpack-erlang returns nan, positive_infinity and negative_infinity.
License
Apache License 2.0
Release Notes
0.7.0
Support nan, positive_infinity and negative_infinity
0.6.0
Support OTP 19.0
0.5.0
Renewed optional arguments to pack/unpack interface. This is
incompatible change from 0.4 series.
0.4.0
Deprecate nil
Moved to rebar3
Promote default map unpacker as default format when OTP is >= 17
Added QuickCheck tests
Since this version OTP older than R16B03-1 are no more supported
0.3.5 / 0.3.4
0.3 series will be the last versions that supports R16B or older
versions of OTP.
OTP 18.0 support
Promote default map unpacker as default format when OTP is >= 18
0.3.3
Add OTP 17 series to Travis-CI tests
Fix wrong numbering for ext types
Allow packing maps even when {format,map} is not set
Fix Dialyzer invalid contract warning
Proper use of null for jiffy-style encoding/decoding
0.3.2
set back default style as jiffy
fix bugs around nil/null handling
0.3.0
supports map new in 17.0
jiffy-style maps will be deprecated in near future
set default style as map
0.2.8
0.2 series works with OTP 17.0, R16, R15, and with MessagePack's new
and old format. But does not support map type introduced in
OTP 17.0.
msgpack is brought to you by ⭐uptrace/uptrace.
Uptrace is an open source and blazingly fast
distributed tracing tool powered
by OpenTelemetry and ClickHouse. Give it a star as well!
Field names can be set in much the same way as the encoding/json package. For example:
typePersonstruct {
Namestring`msg:"name"`Addressstring`msg:"address"`Ageint`msg:"age"`Hiddenstring`msg:"-"`// this field is ignoredunexportedbool// this field is also ignored
}
By default, the code generator will satisfy msgp.Sizer, msgp.Encodable, msgp.Decodable,
msgp.Marshaler, and msgp.Unmarshaler. Carefully-designed applications can use these methods to do
marshalling/unmarshalling with zero heap allocations.
While msgp.Marshaler and msgp.Unmarshaler are quite similar to the standard library's
json.Marshaler and json.Unmarshaler, msgp.Encodable and msgp.Decodable are useful for
stream serialization. (*msgp.Writer and *msgp.Reader are essentially protocol-aware versions
of *bufio.Writer and *bufio.Reader, respectively.)
Features
Extremely fast generated code
Test and benchmark generation
JSON interoperability (see msgp.CopyToJSON() and msgp.UnmarshalAsJSON())
Support for complex type declarations
Native support for Go's time.Time, complex64, and complex128 types
Generation of both []byte-oriented and io.Reader/io.Writer-oriented methods
As long as the declarations of MyInt and Data are in the same file as Struct, the parser will determine that the type information for MyInt and Data can be passed into the definition of Struct before its methods are generated.
Extensions
MessagePack supports defining your own types through "extensions," which are just a tuple of
the data "type" (int8) and the raw binary. You can see a worked example in the wiki.
Status
Mostly stable, in that no breaking changes have been made to the /msgp library in more than a year. Newer versions
of the code may generate different code than older versions for performance reasons. I (@philhofer) am aware of a
number of stability-critical commercial applications that use this code with good results. But, caveat emptor.
You can read more about how msgp maps MessagePack types onto Go types in the wiki.
Here some of the known limitations/restrictions:
Identifiers from outside the processed source file are assumed (optimistically) to satisfy the generator's interfaces. If this isn't the case, your code will fail to compile.
Like most serializers, chan and func fields are ignored, as well as non-exported fields.
Encoding of interface{} is limited to built-ins or types that have explicit encoding methods.
Maps must have string keys. This is intentional (as it preserves JSON interop.) Although non-string map keys are not forbidden by the MessagePack standard, many serializers impose this restriction. (It also means any well-formed struct can be de-serialized into a map[string]interface{}.) The only exception to this rule is that the deserializers will allow you to read map keys encoded as bin types, due to the fact that some legacy encodings permitted this. (However, those values will still be cast to Go strings, and they will be converted to str types when re-encoded. It is the responsibility of the user to ensure that map keys are UTF-8 safe in this case.) The same rules hold true for JSON translation.
If the output compiles, then there's a pretty good chance things are fine. (Plus, we generate tests for you.) Please, please, please file an issue if you think the generator is writing broken code.
As one might expect, the generated methods that deal with []byte are faster for small objects, but the io.Reader/Writer methods are generally more memory-efficient (and, at some point, faster) for large (> 2KB) objects.
If your application serializes only primitive types, array, map and struct, code generation is also recommended.
You can get the fastest performance with msgpackgen.
Msgpack for HHVM, It is a msgpack binding for HHVM
API
msgpack_pack(mixed $input) : string;
pack a input to msgpack, object and resource are not supported, array and other types supported,
false on failure.
msgpack_unpack(string $pac) : mixed;
unpack a msgpack.
Installation
$ git clone https://github.com/reeze/msgpack-hhvm --depth=1
$ cd msgpack-hhvm
$ hphpize && cmake .&& make
$ cp msgpack.so /path/to/your/hhvm/ext/dir
If you don't have hphpize program, please intall package hhvm-dev
This Jackson extension library handles reading and writing of data encoded in MessagePack data format.
It extends standard Jackson streaming API (JsonFactory, JsonParser, JsonGenerator), and as such works seamlessly with all the higher level data abstractions (data binding, tree model, and pluggable extensions).
Maven dependency
To use this module on Maven-based projects, use following dependency:
MessagePack is a binary serialization format. If you need a fast and compact alternative of JSON, MessagePack is your friend. For example, a small integer can be encoded in a single byte, and short strings only need a single byte prefix + the original byte array. MessagePack implementation is already available in various languages (See also the list in http://msgpack.org) and works as a universal data format.
MessagePack v7 (or later) is a faster implementation of the previous version v06, and
supports all of the message pack types, including extension format.
Integration with Jackson ObjectMapper (jackson-databind)
msgpack-java supports serialization and deserialization of Java objects through jackson-databind.
For details, see msgpack-jackson/README.md. The template-based serialization mechanism used in v06 is deprecated.
Here is a list of sbt commands for daily development:
> ~compile # Compile source codes
> ~test:compile # Compile both source and test codes
> ~test # Run tests upon source code change
> ~testOnly *MessagePackTest # Run tests in the specified class
> ~testOnly *MessagePackTest -- (pattern) # Run tests matching the pattern
> project msgpack-core # Focus on a specific project
> package # Create a jar file in the target folder of each project
> findbugs # Produce findbugs report in target/findbugs
> jacoco:cover # Report the code coverage of tests to target/jacoco folder
> jcheckStyle # Run check style
> ;scalafmt;test:scalafmt;scalafmtSbt # Reformat Scala codes
Publishing
> publishLocal # Install to local .ivy2 repository
> publishM2 # Install to local .m2 Maven repository
> publish # Publishing a snapshot version to the Sonatype repository
Publish to Sonatype (Maven Central)
To publish a new version, you only need to add a new git tag and push it to GitHub. GitHub Action will deploy a new release version to Maven Central (Sonatype).
$ git tag v0.x.y
$ git push origin v0.x.y
To generate a release notes, you can use this command line:
If you need to publish to Maven central using a local machine, you need to configure sbt-sonatype plugin. First set Sonatype account information (user name and password) in the global sbt settings. To protect your password, never include this file in your project.
You may also need to configure GPG. See the instruction in sbt-pgp.
Then, run publishedSigned followed by sonatypeBundleRelease:
# [optional] When you need to perform the individual release steps manually, use the following commands:
> publishSigned # Publish GPG signed artifacts to the Sonatype repository
> sonatypeBundleRelease # Publish to the Maven Central (It will be synched within less than 4 hours)
If some sporadic error happens (e.g., Sonatype timeout), rerun sonatypeBundleRelease again.
Project Structure
msgpack-core # Contains packer/unpacker implementation that never uses third-party libraries
msgpack-jackson # Contains jackson-dataformat-java implementation
Pure JavaScript only (No node-gyp nor gcc required)
Faster than any other pure JavaScript libraries on node.js v4
Even faster than node-gyp C++ based msgpack library (90% faster on encoding)
Streaming encoding and decoding interface is also available. It's more faster.
Ready for Web browsers including Chrome, Firefox, Safari and even IE8
Tested on Node.js v0.10, v0.12, v4, v5 and v6 as well as Web browsers
Encoding and Decoding MessagePack
varmsgpack=require("msgpack-lite");// encode from JS Object to MessagePack (Buffer)varbuffer=msgpack.encode({"foo": "bar"});// decode from MessagePack (Buffer) to JS Objectvardata=msgpack.decode(buffer);// => {"foo": "bar"}// if encode/decode receives an invalid argument an error is thrown
Writing to MessagePack Stream
varfs=require("fs");varmsgpack=require("msgpack-lite");varwriteStream=fs.createWriteStream("test.msp");varencodeStream=msgpack.createEncodeStream();encodeStream.pipe(writeStream);// send multiple objects to streamencodeStream.write({foo: "bar"});encodeStream.write({baz: "qux"});// call this once you're done writing to the stream.encodeStream.end();
Reading from MessagePack Stream
varfs=require("fs");varmsgpack=require("msgpack-lite");varreadStream=fs.createReadStream("test.msp");vardecodeStream=msgpack.createDecodeStream();// show multiple objects decoded from streamreadStream.pipe(decodeStream).on("data",console.warn);
Decoding MessagePack Bytes Array
varmsgpack=require("msgpack-lite");// decode() accepts Buffer instance per defaultmsgpack.decode(Buffer([0x81,0xA3,0x66,0x6F,0x6F,0xA3,0x62,0x61,0x72]));// decode() also accepts Array instancemsgpack.decode([0x81,0xA3,0x66,0x6F,0x6F,0xA3,0x62,0x61,0x72]);// decode() accepts raw Uint8Array instance as wellmsgpack.decode(newUint8Array([0x81,0xA3,0x66,0x6F,0x6F,0xA3,0x62,0x61,0x72]));
Command Line Interface
A CLI tool bin/msgpack converts data stream from JSON to MessagePack and vice versa.
Proceed to the next steps if you prefer faster browserify compilation time.
Step #2: add browser property on package.json in your project. This refers the global msgpack object instead of including whole of msgpack-lite source code.
A benchmark tool lib/benchmark.js is available to compare encoding/decoding speed
(operation per second) with other MessagePack modules.
It counts operations of 1KB JSON document in 10 seconds.
Streaming benchmark tool lib/benchmark-stream.js is also available.
It counts milliseconds for 1,000,000 operations of 30 bytes fluentd msgpack fragment.
This shows streaming encoding and decoding are super faster.
$ npm run benchmark-stream 2
operation (1000000 x 2)
op
ms
op/s
stream.write(msgpack.encode(obj));
1000000
3027
330360
stream.write(notepack.encode(obj));
1000000
2012
497017
msgpack.Encoder().on("data",ondata).encode(obj);
1000000
2956
338294
msgpack.createEncodeStream().write(obj);
1000000
1888
529661
stream.write(msgpack.decode(buf));
1000000
2020
495049
stream.write(notepack.decode(buf));
1000000
1794
557413
msgpack.Decoder().on("data",ondata).decode(buf);
1000000
2744
364431
msgpack.createDecodeStream().write(buf);
1000000
1341
745712
Test environment: msgpack-lite 0.1.14, Node v4.2.3, Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz
MessagePack Mapping Table
The following table shows how JavaScript objects (value) will be mapped to
MessagePack formats
and vice versa.
Source Value
MessagePack Format
Value Decoded
null, undefined
nil format family
null
Boolean (true, false)
bool format family
Boolean (true, false)
Number (32bit int)
int format family
Number (int or double)
Number (64bit double)
float format family
Number (double)
String
str format family
String
Buffer
bin format family
Buffer
Array
array format family
Array
Map
map format family
Map (if usemap=true)
Object (plain object)
map format family
Object (or Map if usemap=true)
Object (see below)
ext format family
Object (see below)
Note that both null and undefined are mapped to nil 0xC1 type.
This means undefined value will be upgraded to null in other words.
Extension Types
The MessagePack specification allows 128 application-specific extension types.
The library uses the following types to make round-trip conversion possible
for JavaScript native objects.
Type
Object
Type
Object
0x00
0x10
0x01
EvalError
0x11
Int8Array
0x02
RangeError
0x12
Uint8Array
0x03
ReferenceError
0x13
Int16Array
0x04
SyntaxError
0x14
Uint16Array
0x05
TypeError
0x15
Int32Array
0x06
URIError
0x16
Uint32Array
0x07
0x17
Float32Array
0x08
0x18
Float64Array
0x09
0x19
Uint8ClampedArray
0x0A
RegExp
0x1A
ArrayBuffer
0x0B
Boolean
0x1B
Buffer
0x0C
String
0x1C
0x0D
Date
0x1D
DataView
0x0E
Error
0x1E
0x0F
Number
0x1F
Other extension types are mapped to built-in ExtBuffer object.
Custom Extension Types (Codecs)
Register a custom extension type number to serialize/deserialize your own class instances.
The first argument of addExtPacker and addExtUnpacker should be an integer within the range of 0 and 127 (0x0 and 0x7F). myClassPacker is a function that accepts an instance of MyClass, and should return a buffer representing that instance. myClassUnpacker is the opposite: it accepts a buffer and should return an instance of MyClass.
If you pass an array of functions to addExtPacker or addExtUnpacker, the value to be encoded/decoded will pass through each one in order. This allows you to do things like this:
You can also pass the codec option to msgpack.Decoder(options), msgpack.Encoder(options), msgpack.createEncodeStream(options), and msgpack.createDecodeStream(options).
If you wish to modify the default built-in codec, you can access it at msgpack.codec.preset.
Custom Codec Options
msgpack.createCodec() function accepts some options.
It does NOT have the preset extension types defined when no options given.
varcodec=msgpack.createCodec();
preset: It has the preset extension types described above.
varcodec=msgpack.createCodec({preset: true});
safe: It runs a validation of the value before writing it into buffer. This is the default behavior for some old browsers which do not support ArrayBuffer object.
varcodec=msgpack.createCodec({safe: true});
useraw: It uses raw formats instead of bin and str.
varcodec=msgpack.createCodec({useraw: true});
int64: It decodes msgpack's int64/uint64 formats with int64-buffer object.
varcodec=msgpack.createCodec({int64: true});
binarraybuffer: It ties msgpack's bin format with ArrayBuffer object, instead of Buffer object.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
This is a MessagePack serializer and deserializer written in JavaScript for web browsers (including IE 11) and Node.js.
It is compact but still fully-featured. This library supports the complete MessagePack specification released on 2017-08-09, including date/time values. No other extension types are implemented in this library, it’s only the standard types which is perfectly fine for interoperability with MessagePack codecs in other programming languages.
I’m using the MessagePack-CSharp library on the server side in my .NET applications.
MessagePack
MessagePack is an efficient binary serialisation format. It lets you exchange data among multiple languages like JSON. But it’s faster and smaller. Small integers are encoded into a single byte, and typical short strings require only one extra byte in addition to the strings themselves.
Size
This library is very lightweight. The source code has around 560 lines (incl. browser/Node detection), the minified file has 7.0 kB and can be GZip-compressed to 2.7 kB.
Performance
The file msgpack-tests.html contains some tests and a benchmark function that compares this library with msgpack-lite. Here are the results, in milliseconds (lower is better). All tests done on an Intel Core i7-3770 and Windows 10.
Function
Chrome 72
Firefox 65
Edge 16
IE 11
msgpack.js serialize
702 ms
+6%
1232 ms
−42%
2483 ms
+41%
2493 ms
−3%
msgpack-lite encode
663 ms
2124 ms
1762 ms
2578 ms
msgpack.js deserialize
652 ms
+13%
869 ms
+5%
821 ms
−48%
651 ms
−68%
msgpack-lite decode
577 ms
827 ms
1587 ms
2021 ms
The numbers show that this library is comparable with msgpack-lite. In Chrome it’s only 10% slower. But serializing in Firefox and deserializing in Microsoft browsers is twice as fast.
Usage
Browser
In browsers, a global msgpack object is created that contains the functions serialize and deserialize. The first can be called with any data and returns the serialized bytes. The second works in reverse, taking the serialized bytes and returning the runtime value.
Include the JavaScript file into your HTML document like this:
<scriptsrc="msgpack.min.js"></script>
You can use the library functions after loading the script.
If there should be a naming conflict with another library you want to load, you can change the global object name from msgpack to something else by setting msgpackJsName before loading the script file:
In Node.js, these functions are exported in the object you get from the require function.
varmsgpack=require("@ygoe/msgpack");
Example
Here’s a simple example:
// Define some datavarsourceData={number: 123,number2: -0.129,text: "Abc with Üñıçôðé and ユニコード",flag: true,list: [1,2,3],obj: {a: 1,b: "2",c: false,d: {a: 0,b: -1}},time: Date.now()};// Serialize to byte arrayvarbytes=msgpack.serialize(sourceData);// Deserialize againvardeserializedData=msgpack.deserialize(bytes);
Compatibility
You can also use the functions encode and decode which are aliases to serialize and deserialize. This makes it easier to replace other libraries that use these function names with msgpack.js.
New projects should use the preferred (and more precisely named) serialize and deserialize functions though.
The msgpackr package is an extremely fast MessagePack NodeJS/JavaScript implementation. Currently, it is significantly faster than any other known implementations, faster than Avro (for JS), and generally faster than native V8 JSON.stringify/parse, on NodeJS. It also includes an optional record extension (the r in msgpackr), for defining record structures that makes MessagePack even faster and more compact, often over twice as fast as even native JSON functions, several times faster than other JS implementations, and 15-50% more compact. See the performance section for more details. Structured cloning (with support for cyclical references) is also supported through optional extensions.
Basic Usage
Install with:
npm i msgpackr
And import or require it for basic standard serialization/encoding (pack) and deserialization/decoding (unpack) functions:
This pack function will generate standard MessagePack without any extensions that should be compatible with any standard MessagePack parser/decoder. It will serialize JavaScript objects as MessagePack maps by default. The unpack function will deserialize MessagePack maps as an Object with the properties from the map.
Node Usage
The msgpackr package runs on any modern JS platform, but is optimized for NodeJS usage (and will use a node addon for performance boost as an optional dependency).
Streams
We can use the including streaming functionality (which further improves performance). The PackrStream is a NodeJS transform stream that can be used to serialize objects to a binary stream (writing to network/socket, IPC, etc.), and the UnpackrStream can be used to deserialize objects from a binary sream (reading from network/socket, etc.):
Or for a full example of sending and receiving data on a stream:
import{PackrStream,UnpackrStream}from'msgpackr';letsendingStream=newPackrStream();letreceivingStream=newUnpackrStream();// we are just piping to our own stream, but normally you would send and// receive over some type of inter-process or network connection.sendingStream.pipe(receivingStream);sendingStream.write(myData);receivingStream.on('data',(data)=>{// received data});
The PackrStream and UnpackrStream instances will have also the record structure extension enabled by default (see below).
Deno Usage
Msgpackr modules are standard ESM modules and can be loaded directly from the deno.land registry for msgpackr for use in Deno. The standard pack/encode and unpack/decode functionality is available on Deno, like other platforms.
Browser Usage
Msgpackr works as standalone JavaScript as well, and runs on modern browsers. It includes a bundled script, at dist/index.js for ease of direct loading:
This is UMD based, and will register as a module if possible, or create a msgpackr global with all the exported functions.
For module-based development, it is recommended that you directly import the module of interest, to minimize dependencies that get pulled into your application:
import{unpack}from'msgpackr/unpack'// if you only need to unpack
The package also includes a minified bundle in index.min.js.
Additionally, the package includes a version that excludes dynamic code evaluation called index-no-eval.js, for situations where Content Security Policy (CSP) forbids eval/Function in code. The dynamic evaluation provides important performance optimizations (for records), so is not recommended unless required by CSP policy.
Structured Cloning
You can also use msgpackr for structured cloning. By enabling the structuredClone option, you can include references to other objects or cyclic references, and object identity will be preserved. Structured cloning also enables preserving certain typed objects like Error, Set, RegExp and TypedArray instances. For example:
This option is disabled by default because it uses extensions and reference checking degrades performance (by about 25-30%). (Note this implementation doesn't serialize every class/type specified in the HTML specification since not all of them make sense for storing across platforms.)
Alternate Terminology
If you prefer to use encoder/decode terminology, msgpackr exports aliases, so decode is equivalent to unpack, encode is pack, Encoder is Packr, Decoder is Unpackr, and EncoderStream and DecoderStream can be used as well.
Record / Object Structures
There is a critical difference between maps (or dictionaries) that hold an arbitrary set of keys and values (JavaScript Map is designed for these), and records or object structures that have a well-defined set of fields. Typical JS objects/records may have many instances re(use) the same structure. By using the record extension, this distinction is preserved in MessagePack and the encoding can reuse structures and not only provides better type preservation, but yield much more compact encodings and increase decoding performance by 2-3x. Msgpackr automatically generates record definitions that are reused and referenced by objects with the same structure. There are a number of ways to use this to our advantage. For large object structures with repeating nested objects with similar structures, simply serializing with the record extension can yield significant benefits. To use the record structures extension, we create a new Packr instance. By default a new Packr instance will have the record extension enabled:
Another way to further leverage the benefits of the msgpackr record structures is to use streams that naturally allow for data to reuse based on previous record structures. The stream classes have the record structure extension enabled by default and provide excellent out-of-the-box performance.
When creating a new Packr, Unpackr, PackrStream, or UnpackrStream instance, we can enable or disable the record structure extension with the useRecords property. When this is false, the record structure extension will be disabled (standard/compatibility mode), and all objects will revert to being serialized using MessageMap maps, and all maps will be deserialized to JS Objects as properties (like the standalone pack and unpack functions).
Streaming with record structures works by encoding a structure the first time it is seen in a stream and referencing the structure in later messages that are sent across that stream. When an encoder can expect a decoder to understand previous structure references, this can be configured using the sequential: true flag, which is auto-enabled by streams, but can also be used with Packr instances.
Shared Record Structures
Another useful way of using msgpackr, and the record extension, is for storing data in a databases, files, or other storage systems. If a number of objects with common data structures are being stored, a shared structure can be used to greatly improve data storage and deserialization efficiency. In the simplest form, provide a structures array, which is updated if any new object structure is encountered:
If you are working with persisted data, you will need to persist the structures data when it is updated. Msgpackr provides an API for loading and saving the structures on demand (which is robust and can be used in multiple-process situations where other processes may be updating this same structures array), we just need to provide a way to store the generated shared structure so it is available to deserialize stored data in the future:
import{Packr}from'msgpackr';letpackr=newPackr({getStructures(){// storing our data in file (but we could also store in a db or key-value store)returnunpack(readFileSync('my-shared-structures.mp'))||[];},saveStructures(structures){writeFileSync('my-shared-structures.mp',pack(structures));}});
Msgpackr will automatically add and saves structures as it encounters any new object structures (up to a limit of 32, by default). It will always add structures in an incremental/compatible way: Any object encoded with an earlier structure can be decoded with a later version (as long as it is persisted).
Shared Structures Options
By default there is a limit of 32 shared structures. This default is designed to record common shared structures, but also be resilient against sharing too many structures if there are many objects with dynamic properties that are likely to be repeated. This also allows for slightly more efficient one byte encoding. However, if your application has more structures that are commonly repeated, you can increase this limit by setting maxSharedStructures to a higher value. The maximum supported shared structures is 8160.
You can also provide a shouldShareStructure function in the options if you want to specifically indicate which structures should be shared. This is called during the encoding process with the array of keys for a structure that is being considered for addition to the shared structure. For example, you might want:
maxSharedStructures: 100,
shouldShareStructure(keys) {
return !(keys[0] > 1) // don't share structures that consist of numbers as keys
}
Reading Multiple Values
If you have a buffer with multiple values sequentially encoded, you can choose to parse and read multiple values. This can be done using the unpackMultiple function/method, which can return an array of all the values it can sequentially parse within the provided buffer. For example:
letdata=newUint8Array([1,2,3])// encodings of values 1, 2, and 3letvalues=unpackMultiple(data)// [1, 2, 3]
Alternately, you can provide a callback function that is called as the parsing occurs with each value, and can optionally terminate the parsing by returning false:
letdata=newUint8Array([1,2,3])// encodings of values 1, 2, and 3unpackMultiple(data,(value)=>{// called for each value// return false if you wish to end the parsing})
Options
The following options properties can be provided to the Packr or Unpackr constructor:
useRecords - Setting this to false disables the record extension and stores JavaScript objects as MessagePack maps, and unpacks maps as JavaScript Objects, which ensures compatibilty with other decoders.
structures - Provides the array of structures that is to be used for record extension, if you want the structures saved and used again. This array will be modified in place with new record structures that are serialized (if less than 32 structures are in the array).
moreTypes - Enable serialization of additional built-in types/classes including typed arrays, Sets, Maps, and Errors.
structuredClone - This enables the structured cloning extensions that will encode object/cyclic references. moreTypes is enabled by default when this is enabled.
mapsAsObjects - If true, this will decode MessagePack maps and JS Objects with the map entries decoded to object properties. If false, maps are decoded as JavaScript Maps. This is disabled by default if useRecords is enabled (which allows Maps to be preserved), and is enabled by default if useRecords is disabled.
useFloat32 - This will enable msgpackr to encode non-integer numbers as float32. See next section for possible values.
variableMapSize - This will use varying map size definition (fixmap, map16, map32) based on the number of keys when encoding objects, which yields slightly more compact encodings (for small objects), but is typically 5-10% slower during encoding. This is necessary if you need to use objects with more than 65535 keys. This is only relevant when record extension is disabled.
bundleStrings - If true this uses a custom extension that bundles strings together, so that they can be decoded more quickly on browsers and Deno that do not have access to the NodeJS addon. This a custom extension, so both encoder and decoder need to support this. This can yield significant decoding performance increases on browsers (30%-50%).
copyBuffers - When decoding a MessagePack with binary data (Buffers are encoded as binary data), copy the buffer rather than providing a slice/view of the buffer. If you want your input data to be collected or modified while the decoded embedded buffer continues to live on, you can use this option (there is extra overhead to copying).
useTimestamp32 - Encode JS Dates in 32-bit format when possible by dropping the milliseconds. This is a more efficient encoding of dates. You can also cause dates to use 32-bit format by manually setting the milliseconds to zero (date.setMilliseconds(0)).
sequential - Encode structures in serialized data, and reference previously encoded structures with expectation that decoder will read the encoded structures in the same order as encoded, with unpackMultiple.
largeBigIntToFloat - If a bigint needs to be encoded that is larger than will fit in 64-bit integers, it will be encoded as a float-64 (otherwise will throw a RangeError).
encodeUndefinedAsNil - Encodes a value of undefined as a MessagePack nil, the same as a null.
int64AsType - This will decode uint64 and int64 numbers as the specified type. The type can be bigint (default), number, or string.
onInvalidDate - This can be provided as function that will be called when an invalid date is provided. The function can throw an error, or return a value that will be encoded in place of the invalid date. If not provided, an invalid date will be encoded as an invalid timestamp (which decodes with msgpackr back to an invalid date).
32-bit Float Options
By default all non-integer numbers are serialized as 64-bit float (double). This is fast, and ensures maximum precision. However, often real-world data doesn't not need 64-bits of precision, and using 32-bit encoding can be much more space efficient. There are several options that provide more efficient encodings. Using the decimal rounding options for encoding and decoding provides lossless storage of common decimal representations like 7.99, in more efficient 32-bit format (rather than 64-bit). The useFloat32 property has several possible options, available from the module as constants:
ALWAYS (1) - Always will encode non-integers (absolute less than 2147483648) as 32-bit float.
DECIMAL_ROUND (3) - Always will encode non-integers as 32-bit float, and when decoding 32-bit float, round to the significant decimal digits (usually 7, but 6 or 8 digits for some ranges).
DECIMAL_FIT (4) - Only encode non-integers as 32-bit float if all significant digits (usually up to 7) can be unambiguously encoded as a 32-bit float, and decode/unpack with decimal rounding (same as above). This will ensure round-trip encoding/decoding without loss in precision and uses 32-bit when possible.
Note, that the performance is decreased with decimal rounding by about 20-25%, although if only 5% of your values are floating point, that will only have about a 1% impact overall.
In addition, msgpackr exports a roundFloat32(number) function that can be used to round floating point numbers to the maximum significant decimal digits that can be stored in 32-bit float, just as DECIMAL_ROUND does when decoding. This can be useful for determining how a number will be decoded prior to encoding it.
Performance
Native Acceleration
Msgpackr employs an optional native node-addon to accelerate the parsing of strings. This should be automatically installed and utilized on NodeJS. However, you can verify this by checking the isNativeAccelerationEnabled property that is exported from msgpackr. If this is false, the msgpackr-extract package may not have been properly installed, and you may want to verify that it is installed correctly:
import{isNativeAccelerationEnabled}from'msgpackr'if(!isNativeAccelerationEnabled)console.warn('Native acceleration not enabled, verify that install finished properly')
Benchmarks
Msgpackr is fast. Really fast. Here is comparison with the next fastest JS projects using the benchmark tool from msgpack-lite (and the sample data is from some clinical research data we use that has a good mix of different value types and structures). It also includes comparison to V8 native JSON functionality, and JavaScript Avro (avsc, a very optimized Avro implementation):
All benchmarks were performed on Node 15 / V8 8.6 (Windows i7-4770 3.4Ghz).
(avsc is schema-based and more comparable in style to msgpackr with shared structures).
Here is a benchmark of streaming data (again borrowed from msgpack-lite's benchmarking), where msgpackr is able to take advantage of the structured record extension and really demonstrate its performance capabilities:
operation (1000000 x 2)
op
ms
op/s
new PackrStream().write(obj);
1000000
372
2688172
new UnpackrStream().write(buf);
1000000
247
4048582
stream.write(msgpack.encode(obj));
1000000
2898
345065
stream.write(msgpack.decode(buf));
1000000
1969
507872
stream.write(notepack.encode(obj));
1000000
901
1109877
stream.write(notepack.decode(buf));
1000000
1012
988142
msgpack.Encoder().on("data",ondata).encode(obj);
1000000
1763
567214
msgpack.createDecodeStream().write(buf);
1000000
2222
450045
msgpack.createEncodeStream().write(obj);
1000000
1577
634115
msgpack.Decoder().on("data",ondata).decode(buf);
1000000
2246
445235
See the benchmark.md for more benchmarks and information about benchmarking.
Custom Extensions
You can add your own custom extensions, which can be used to encode specific types/classes in certain ways. This is done by using the addExtension function, and specifying the class, extension type code (should be a number from 1-100, reserving negatives for MessagePack, 101-127 for msgpackr), and your pack and unpack functions (or just the one you need).
import{addExtension,Packr}from'msgpackr';classMyCustomClass{...}letextPackr=newPackr();addExtension({Class: MyCustomClass,type: 11,// register your own extension code (a type code from 1-100)pack(instance){// define how your custom class should be encodedreturnBuffer.from([instance.myData]);// return a buffer}unpack(buffer){// define how your custom class should be decodedletinstance=newMyCustomClass();instance.myData=buffer[0];returninstance;// decoded value from buffer}});
If you want to use msgpackr to encode and decode the data within your extensions, you can use the read and write functions and read and write data/objects that will be encoded and decoded by msgpackr, which can be easier and faster than creating and receiving separate buffers:
import{addExtension,Packr}from'msgpackr';classMyCustomClass{...}letextPackr=newPackr();addExtension({Class: MyCustomClass,type: 11,// register your own extension code (a type code from 1-100)write(instance){// define how your custom class should be encodedreturninstance.myData;// return some data to be encoded}read(data){// define how your custom class should be decoded,// data will already be unpacked/decodedletinstance=newMyCustomClass();instance.myData=data;returninstance;// return decoded value}});
Note that you can just return the same object from write, and in this case msgpackr will encode it using the default object/array encoding:
You can also create an extension with Class and write methods, but no type (or read), if you just want to customize how a class is serialized without using MessagePack extension encoding.
Additional Performance Optimizations
Msgpackr is already fast, but here are some tips for making it faster:
Buffer Reuse
Msgpackr is designed to work well with reusable buffers. Allocating new buffers can be relatively expensive, so if you have Node addons, it can be much faster to reuse buffers and use memcpy to copy data into existing buffers. Then msgpackr unpack can be executed on the same buffer, with new data, and optionally take a second paramter indicating the effective size of the available data in the buffer.
Arena Allocation (useBuffer())
During the serialization process, data is written to buffers. Again, allocating new buffers is a relatively expensive process, and the useBuffer method can help allow reuse of buffers that will further improve performance. With useBuffer method, you can provide a buffer, serialize data into it, and when it is known that you are done using that buffer, you can call useBuffer again to reuse it. The use of useBuffer is never required, buffers will still be handled and cleaned up through GC if not used, it just provides a small performance boost.
Record Structure Extension Definition
The record struction extension uses extension id 0x72 ("r") to declare the use of this functionality. The extension "data" byte (or bytes) identifies the byte or bytes used to identify the start of a record in the subsequent MessagePack block or stream. The identifier byte (or the first byte in a sequence) must be from 0x40 - 0x7f (and therefore replaces one byte representations of positive integers 64 - 127, which can alternately be represented with int or uint types). The extension declaration must be immediately follow by an MessagePack array that defines the field names of the record structure.
Once a record identifier and record field names have been defined, the parser/decoder should proceed to read the next value. Any subsequent use of the record identifier as a value in the block or stream should parsed as a record instance, and the next n values, where is n is the number of fields (as defined in the array of field names), should be read as the values of the fields. For example, here we have defined a structure with fields "foo" and "bar", with the record identifier 0x40, and then read a record instance that defines the field values of 4 and 2, respectively:
Which should generate an object that would correspond to JSON:
{"foo": 4,"bar": 2}
Additional value types
msgpackr supports undefined (using fixext1 + type: 0 + data: 0 to match other JS implementations), NaN, Infinity, and -Infinity (using standard IEEE 754 representations with doubles/floats).
Dates
msgpackr saves all JavaScript Dates using the standard MessagePack date extension (type -1), using the smallest of 32-bit, 64-bit or 96-bit format needed to store the date without data loss (or using 32-bit if useTimestamp32 options is specified).
Structured Cloning
With structured cloning enabled, msgpackr will also use extensions to store Set, Map, Error, RegExp, ArrayBufferView objects and preserve their types.
Alternate Encoding/Package
The high-performance serialization and deserialization algorithms in the msgpackr package are also available in the cbor-x for the CBOR format, with the same API and design. A quick summary of the pros and cons of using MessagePack vs CBOR are:
MessagePack has wider adoption, and, at least with this implementation is slightly more efficient (by roughly 1%).
MessagePack can be a great choice for high-performance data delivery to browsers, as reasonable data size is possible without compression. And msgpackr works very well in modern browsers. However, it is worth noting that if you want highly compact data, brotli or gzip are most effective in compressing, and MessagePack's character frequency tends to defeat Huffman encoding used by these standard compression algorithms, resulting in less compact data than compressed JSON.
This library is a universal JavaScript, meaning it is compatible with all the major browsers and NodeJS. In addition, because it is implemented in TypeScript, type definition files (d.ts) are always up-to-date and bundled in the distribution.
Note that this is the second version of MessagePack for JavaScript. The first version, which was implemented in ES5 and was never released to npmjs.com, is tagged as classic.
It encodes data into a single MessagePack-encoded object, and returns a byte array as Uint8Array. It throws errors if data is, or includes, a non-serializable object such as a function or a symbol.
If you'd like to convert an uint8array to a NodeJS Buffer, use Buffer.from(arrayBuffer, offset, length) in order not to copy the underlying ArrayBuffer, while Buffer.from(uint8array) copies it:
import{encode}from"@msgpack/msgpack";constencoded: Uint8Array=encode({foo: "bar"});// `buffer` refers the same ArrayBuffer as `encoded`.constbuffer: Buffer=Buffer.from(encoded.buffer,encoded.byteOffset,encoded.byteLength);console.log(buffer);
It decodes buffer that includes a MessagePack-encoded object, and returns the decoded object typed unknown.
buffer must be an array of bytes, which is typically Uint8Array or ArrayBuffer. BufferSource is defined as ArrayBuffer | ArrayBufferView.
The buffer must include a single encoded object. If the buffer includes extra bytes after an object or the buffer is empty, it throws RangeError. To decode buffer that includes multiple encoded objects, use decodeMulti() or decodeMultiStream() (recommended) instead.
It decodes buffer that includes multiple MessagePack-encoded objects, and returns decoded objects as a generator. See also decodeMultiStream(), which is an asynchronous variant of this function.
This function is not recommended to decode a MessagePack binary via I/O stream including sockets because it's synchronous. Instead, decodeMultiStream() decodes a binary stream asynchronously, typically spending less CPU and memory.
It decodes stream, where ReadableStreamLike<T> is defined as ReadableStream<T> | AsyncIterable<T>, in an async iterable of byte arrays, and returns decoded object as unknown type, wrapped in Promise.
This function works asynchronously, and might CPU resources more efficiently compared with synchronous decode(), because it doesn't wait for the completion of downloading.
DecodeAsyncOptions is the same as DecodeOptions for decode().
This function is designed to work with whatwg fetch() like this:
import{decodeAsync}from"@msgpack/msgpack";constMSGPACK_TYPE="application/x-msgpack";constresponse=awaitfetch(url);constcontentType=response.headers.get("Content-Type");if(contentType&&contentType.startsWith(MSGPACK_TYPE)&&response.body!=null){constobject=awaitdecodeAsync(response.body);// do something with object}else{/* handle errors */}
It is alike to decodeAsync(), but only accepts a stream that includes an array of items, and emits a decoded item one by one.
for example:
import{decodeArrayStream}from"@msgpack/msgpack";conststream: AsyncIterator<Uint8Array>;// in an async function:forawait(constitemofdecodeArrayStream(stream)){console.log(item);}
It is alike to decodeAsync() and decodeArrayStream(), but the input stream must consist of multiple MessagePack-encoded items. This is an asynchronous variant for decodeMulti().
In other words, it could decode an unlimited stream and emits a decoded item one by one.
for example:
import{decodeMultiStream}from"@msgpack/msgpack";conststream: AsyncIterator<Uint8Array>;// in an async function:forawait(constitemofdecodeMultiStream(stream)){console.log(item);}
This function is available since v2.4.0; previously it was called as decodeStream().
Reusing Encoder and Decoder instances
Encoder and Decoder classes is provided to have better performance by reusing instances:
According to our benchmark, reusing Encoder instance is about 20% faster
than encode() function, and reusing Decoder instance is about 2% faster
than decode() function. Note that the result should vary in environments
and data structure.
This is an example to setup custom extension types that handles Map and Set classes in TypeScript:
import{encode,decode,ExtensionCodec}from"@msgpack/msgpack";constextensionCodec=newExtensionCodec();// Set<T>constSET_EXT_TYPE=0// Any in 0-127extensionCodec.register({type: SET_EXT_TYPE,encode: (object: unknown): Uint8Array|null=>{if(objectinstanceofSet){returnencode([...object]);}else{returnnull;}},decode: (data: Uint8Array)=>{constarray=decode(data)asArray<unknown>;returnnewSet(array);},});// Map<T>constMAP_EXT_TYPE=1;// Any in 0-127extensionCodec.register({type: MAP_EXT_TYPE,encode: (object: unknown): Uint8Array=>{if(objectinstanceofMap){returnencode([...object]);}else{returnnull;}},decode: (data: Uint8Array)=>{constarray=decode(data)asArray<[unknown,unknown]>;returnnewMap(array);},});constencoded=encode([newSet<any>(),newMap<any,any>()],{ extensionCodec });constdecoded=decode(encoded,{ extensionCodec });
Not that extension types for custom objects must be [0, 127], while [-1, -128] is reserved for MessagePack itself.
ExtensionCodec context
When you use an extension codec, it might be necessary to have encoding/decoding state to keep track of which objects got encoded/re-created. To do this, pass a context to the EncodeOptions and DecodeOptions:
import{encode,decode,ExtensionCodec}from"@msgpack/msgpack";classMyContext{track(object: any){/*...*/}}classMyType{/* ... */}constextensionCodec=newExtensionCodec<MyContext>();// MyTypeconstMYTYPE_EXT_TYPE=0// Any in 0-127extensionCodec.register({type: MYTYPE_EXT_TYPE,encode: (object,context)=>{if(objectinstanceofMyType){context.track(object);// <-- like thisreturnencode(object.toJSON(),{ extensionCodec, context });}else{returnnull;}},decode: (data,extType,context)=>{constdecoded=decode(data,{ extensionCodec, context });constmy=newMyType(decoded);context.track(my);// <-- and like thisreturnmy;},});// and laterimport{encode,decode}from"@msgpack/msgpack";constcontext=newMyContext();constencoded==encode({myType: newMyType<any>()},{ extensionCodec, context });constdecoded=decode(encoded,{ extensionCodec, context });
Handling BigInt with ExtensionCodec
This library does not handle BigInt by default, but you can handle it with ExtensionCodec like this:
import{deepStrictEqual}from"assert";import{encode,decode,ExtensionCodec}from"@msgpack/msgpack";constBIGINT_EXT_TYPE=0;// Any in 0-127constextensionCodec=newExtensionCodec();extensionCodec.register({type: BIGINT_EXT_TYPE,encode: (input: unknown)=>{if(typeofinput==="bigint"){if(input<=Number.MAX_SAFE_INTEGER&&input>=Number.MIN_SAFE_INTEGER){returnencode(parseInt(input.toString(),10));}else{returnencode(input.toString());}}else{returnnull;}},decode: (data: Uint8Array)=>{returnBigInt(decode(data));},});constvalue=BigInt(Number.MAX_SAFE_INTEGER)+BigInt(1);constencoded: =encode(value,{ extensionCodec });deepStrictEqual(decode(encoded,{ extensionCodec }),value);
The temporal module as timestamp extensions
There is a proposal for a new date/time representations in JavaScript:
This library maps Date to the MessagePack timestamp extension by default, but you can re-map the temporal module (or Temporal Polyfill) to the timestamp extension like this:
This will become default in this library with major-version increment, if the temporal module is standardized.
Decoding a Blob
Blob is a binary data container provided by browsers. To read its contents, you can use Blob#arrayBuffer() or Blob#stream(). Blob#stream()
is recommended if your target platform support it. This is because streaming
decode should be faster for large objects. In both ways, you need to use
asynchronous API.
asyncfunctiondecodeFromBlob(blob: Blob): unknown{if(blob.stream){// Blob#stream(): ReadableStream<Uint8Array> (recommended)returnawaitdecodeAsync(blob.stream());}else{// Blob#arrayBuffer(): Promise<ArrayBuffer> (if stream() is not available)returndecode(awaitblob.arrayBuffer());}}
MessagePack Specification
This library is compatible with the "August 2017" revision of MessagePack specification at the point where timestamp ext was added:
Note that as of June 2019 there're no official "version" on the MessagePack specification. See msgpack/msgpack#195 for the discussions.
MessagePack Mapping Table
The following table shows how JavaScript values are mapped to MessagePack formats and vice versa.
Source Value
MessagePack Format
Value Decoded
null, undefined
nil
null (*1)
boolean (true, false)
bool family
boolean (true, false)
number (53-bit int)
int family
number (53-bit int)
number (64-bit float)
float family
number (64-bit float)
string
str family
string
ArrayBufferView
bin family
Uint8Array (*2)
Array
array family
Array
Object
map family
Object (*3)
Date
timestamp ext family
Date (*4)
*1 Both null and undefined are mapped to nil (0xC0) type, and are decoded into null
*2 Any ArrayBufferViews including NodeJS's Buffer are mapped to bin family, and are decoded into Uint8Array
*3 In handling Object, it is regarded as Record<string, unknown> in terms of TypeScript
*4 MessagePack timestamps may have nanoseconds, which will lost when it is decoded into JavaScript Date. This behavior can be overridden by registering -1 for the extension codec.
Prerequisites
This is a universal JavaScript library that supports major browsers and NodeJS.
ECMA-262
ES5 language features
ES2018 standard library, including:
Typed arrays (ES2015)
Async iterations (ES2018)
Features added in ES2015-ES2018
ES2018 standard library used in this library can be polyfilled with core-js.
If you support IE11, import core-js in your application entrypoints, as this library does in testing for browsers.
NodeJS before v10 will work by importing @msgpack/msgpack/dist.es5+umd/msgpack.
TypeScript Compiler / Type Definitions
This module requires type definitions of AsyncIterator, SourceBuffer, whatwg streams, and so on. They are provided by "lib": ["ES2021", "DOM"] in tsconfig.json.
Regarding the TypeScript compiler version, only the latest TypeScript is tested in development.
Benchmark
Run-time performance is not the only reason to use MessagePack, but it's important to choose MessagePack libraries, so a benchmark suite is provided to monitor the performance of this library.
V8's built-in JSON has been improved for years, esp. JSON.parse() is significantly improved in V8/7.6, it is the fastest deserializer as of 2019, as the benchmark result bellow suggests.
However, MessagePack can handles binary data effectively, actual performance depends on situations. You'd better take benchmark on your own use-case if performance matters.
Benchmark on NodeJS/v18.1.0 (V8/10.1)
operation
op
ms
op/s
buf = Buffer.from(JSON.stringify(obj));
902100
5000
180420
obj = JSON.parse(buf.toString("utf-8"));
898700
5000
179740
buf = require("msgpack-lite").encode(obj);
411000
5000
82200
obj = require("msgpack-lite").decode(buf);
246200
5001
49230
buf = require("@msgpack/msgpack").encode(obj);
843300
5000
168660
obj = require("@msgpack/msgpack").decode(buf);
489300
5000
97860
buf = /* @msgpack/msgpack */ encoder.encode(obj);
1154200
5000
230840
obj = /* @msgpack/msgpack */ decoder.decode(buf);
448900
5000
89780
Note that JSON cases use Buffer to emulate I/O where a JavaScript string must be converted into a byte array encoded in UTF-8, whereas MessagePack modules deal with byte arrays.
Distribution
NPM / npmjs.com
The NPM package distributed in npmjs.com includes both ES2015+ and ES5 files:
dist/ is compiled into ES2019 with CommomJS, provided for NodeJS v10
dist.es5+umd/ is compiled into ES5 with UMD
dist.es5+umd/msgpack.min.js - the minified file
dist.es5+umd/msgpack.js - the non-minified file
dist.es5+esm/ is compiled into ES5 with ES modules, provided for webpack-like bundlers and NodeJS's ESM-mode
If you use NodeJS and/or webpack, their module resolvers use the suitable one automatically.
# run tests on NodeJS, Chrome, and Firefoxmake test-all
# edit the changelogcode CHANGELOG.md
# bump versionnpm version patch|minor|major
# run the publishing taskmake publish
this feature isn't in MessagePack spec but added as a convenience feature in 1.1.3
dictionaries allow us to decrease our buffer output size by recognizing strings used as object keys and replacing them with shorter-byte integer values during the encoding process
these shorter-byte placeholder values are then restored to their respective strings during the decoding process
the trade-off in using dictionaries is an insignificantly slower encoding and decoding time in exchange of a significantly smaller buffer output, which results into a lower network bandwidth and storage consumption in the long run
the best part: the byte placeholders starts from -32 then increments upwards, values -32 to 127 are encoded in single byte, which means your first (32 + 128) = 160 keys will be encoded as a single byte instead of encoding the whole string