FlatBuffers

FlatBuffers
Original author(s) Wouter van Oortmerssen
Initial release June 17, 2014 (2014-06-17)[1]
Stable release
1.9.0 / April 6, 2018 (2018-04-06)[2]
Repository
Edit this at Wikidata
Written in C++
Operating system Android, Microsoft Windows, Mac OS X, Linux
Type serialization format and library, IDL compiler
License Apache 2
Website google.github.io/flatbuffers/

In computer programming, FlatBuffers is a free software library implementing a serialization format similar to Protocol Buffers, Thrift, Apache Avro, SBE, and Cap'n Proto, primarily written by Wouter van Oortmerssen and open-sourced by Google. Like Cap'n Proto and SBE, it supports “zero-copy” deserialization, so that accessing the serialized data does not require first copying it into a separate part of memory, which makes accessing data in these formats much faster than data in formats requiring more extensive processing, such as JSON, CSV, and Protocol Buffers. It is the data serialization format used internally in Facebook's Android client, replacing JSON; in that role, they credit it for dramatic performance improvements in the client.[3] Cocos2d-x, the most popular free-software 2-D game programming library, also uses FlatBuffers to serialize all of its game data.[4] FlatBuffers is a very popular project on GitHub, with 10,228 stars, 260 contributors, 1,605 forks, and 600 watchers on GitHub as of 2018-09-19.[5] The FlatBuffers discussion mailing list has 440 members as of 2017-06-15.[6]

FlatBuffers can be used in software written in C++, C#, C, Golang, Java, JavaScript, PHP, Python and Rust. The schema compiler runs on Android, Microsoft Windows, Mac OS X, and Linux,[5] but games and other programs using FlatBuffers for serialization work on many other operating systems as well, including iOS, Amazon's Fire OS, and Windows Phone.[7]

Oortmerssen originally developed FlatBuffers for game development and similar applications.[8][1]

Although FlatBuffers has its own interface definition language to define the data to be serialized with it, it also supports schemas defined in the Protocol Buffers .proto format. [9]

Schema evolution

Serialization formats in general face the problem of schema evolution — the fact that, except under the most unusual circumstances, at times one version of the software will attempt to deserialize data that was serialized by an older version of the software or even by an independent implementation of the same format, while new versions of the software may desire to add new features. In a few cases, it is feasible to translate all the existing data into a new format, but this has several drawbacks:

  1. Sometimes it isn't possible to locate all the data in the old format, for example, because some of it only exists on backup tapes or saved on users' memory cards.
  2. Even when it is possible to locate all the old data, translating all of it can be costly.
  3. Even when translating all the old data is feasible and affordable, often it isn't feasible to stop new data from being generated in the old format, because of old versions of the software or independent implementations of the format.
  4. Even when that is not the case, translating all the old data runs the risk of undetected corruption due to bugs in the translation code, corruption which is irreversible once the old version has been discarded.

So, in nearly all cases, it is desirable to preserve interoperability with older versions of the data format. As an extreme example, RFC-822 email messages produced today (2017) use the same basic header names and syntax as ARPANet mail messages from the late 1970s, with the result that messages written using software that is 40 years old can be read using modern software, and vice versa.

However, RFC-822 messages are inherently inefficient, as are pure name-value formats like JSON “objects”. The header names occupy substantial storage space, finding the boundaries of the header value requires slow byte-by-byte sequential scanning, and finding a particular header such as “Subject” requires sequentially scanning over all the headers at least once, if only to store them into a hash table.

FlatBuffers, like its forebear Protocol Buffers, is designed to permit extensibility at lower storage and execution time cost. Both formats achieve this by permitting the addition of new fields at the end of existing record types (called “tables” in FlatBuffers); while Protocol Buffers requires explicit IDs on every field in its IDL, FlatBuffers merely assigns IDs sequentially by default, so new fields must be added at the end of the table, and a central authority is needed to define the order of fields for a given schema — decentralized extensions of a single table type would result in field ID clashes.

See also

References

  1. 1 2 Wouter van Oortmerssen (2014-06-17). "FlatBuffers: A Memory-Efficient Serialization Library". Retrieved 2017-06-15.
  2. ↑ "FlatBuffers Releases". GitHub. Retrieved 2018-01-23.
  3. ↑ George Xie (2015-07-31). "Improving Facebook's performance on Android with FlatBuffers". Retrieved 2017-06-15. Story load time from disk cache is reduced from 35 ms to 4 ms per story. Transient memory allocations are reduced by 75 percent. Cold start time is improved by 10-15 percent. We have reduced storage size by 15 percent.
  4. ↑ http://www.cocos2d-x.org/reference/native-cpp/V3.5/d7/d2d/namespaceflatbuffers.html
  5. 1 2 "GitHub — google/flatbuffers: Memory Efficient Serialization Library". GitHub. Retrieved 2018-09-19.
  6. ↑ https://groups.google.com/forum/#!aboutgroup/flatbuffers
  7. ↑ "FlatBuffers for Unity". eXiin. 2015-09-21. Retrieved 2017-06-15. We tested flatbuffers [sic] on all major mobile platforms ( iOS, Android, Amazon Os [sic], Windows Phone ) we’re building on[,] and it works pretty well.
  8. ↑ "FlatBuffers Documentation". Retrieved 2017-06-21. FlatBuffers is an efficient cross platform serialization library for C++, C#, C, Go, Java, JavaScript, PHP, and Python. It was originally created at Google for game development and other performance-critical applications.
  9. ↑ Kenton Varda (2014-06-17). "Cap'n Proto, FlatBuffers, and SBE". Retrieved 2017-06-15.
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.