Protocol Buffers — A binary protocol beyond RPC

The news sounds so innocent and yet it has stirred up so much controversy. I mean the announcement of the release of Protocol Buffers, essentially a binary format developed and used by Google to exchange data efficiently. Stop if any of these questions come to mind.

• What do you mean, a binary format?
• What do you mean, exchanging data?
• What the hell do we have XML for?
• Is this an XML killer?

Right, those are exactly the issues the announcement set people talking about. As they say, things are never as bad as they first seem. So let’s go through it in order. Protocol Buffers (PB from here on) grew out of the need for as efficient a format as possible for exchanging data. Google needs a data format that can be serialized and deserialized quickly and has as little overhead as possible. That requirement ruled XML out straightaway.

Protocol buffers have many advantages over XML for serializing structured data. Protocol buffers:

• are simpler
• are 3 to 10 times smaller
• are 20 to 100 times faster
• are less ambiguous

generate data access classes that are easier to use programmatically

Of course, even Google’s engineers don’t work with the binary protocol directly. PB therefore includes a generator for the object representation. Here’s how you work with PB:

• Use a dedicated language (the protocol buffer language) to describe the structure of your data, creating a message schema.
• Run the compiler for your preferred language (Java, C++, Python, Perl, Haskell) to generate an object representation for your data. From a Java perspective, it’s a conventional JavaBean with setters and getters, plus a method for serializing to and deserializing from a byte array.
• Work with the generated representation in your code.

Here’s a simple example I pulled from the PB Java tutorial.

Describing the data structure

package tutorial;

option java_package = "com.example.tutorial";
option java_outer_classname = "AddressBookProtos";

message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;

enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

message PhoneNumber {
    required string number = 1;
    optional PhoneType type = 2 [default = HOME];
  }

repeated PhoneNumber phone = 4;
}

Working in Java

Person john =
    Person.newBuilder()
    .setId(1234)
    .setName("John Doe")
    .setEmail("jdoe@example.com")
    .addPhone(
      Person.PhoneNumber.newBuilder()
        .setNumber("555-4321")
        .setType(Person.PhoneType.HOME))
    .build();

//serializace    
FileOutputStream output = new FileOutputStream("john.dat");
john.writeTo(output);

//deserilizace
john = Person.parseFrom(new FileInputStream("john.dat"));

Simple, right? Google disclosed that it uses PB in RPC systems and for persistent storage of data in a variety of storage systems, and that Protocol buffers are now Google's lingua franca for data – at time of writing, there are 48,162 different message types defined in the Google code tree across 12,183 .proto files (.proto files describe the structure).

The very mention of RPC became a major test for the true believers in XML. Quite a few people gave Google a hard time over it. But as so often happens, the point is somewhere else. The fact that Google uses PB for RPC doesn’t mean you have to. Ted Neward put it well in his article So You Say You Want to Kill XML....

In the end, if you want an endpoint that is loosely coupled and offers the maximum flexibility, stick with XML, either wrapped in a SOAP envelope or in a RESTful envelope as dictated by the underlying transport (which means HTTP, since REST over anything else has never really been defined clearly by the Restafarians). If you need a binary format, then Protocol Buffers are certainly one answer… but so is ICE, or even CORBA (though this is fast losing its appeal thanks to the slow decline of the players in this space). Don’t lose sight of the technical advantages or disadvantages of each of those solutions just because something has the Google name on it.

By the way, PB isn’t the only solution of its kind. There’s also Thrift (already in the Apache incubator). For anyone interested, here’s a comparison of their features.

Sources: This article draws extensively on information obtained directly or indirectly from Google Introduces Binary Encoding Format: Protocol Buffers.