purescript-protobuf

Purescript library and code generator for Google Protocol Buffers version 3.

This library operates on ArrayBuffer, so it will run both in Node.js and in browser environments.

We aim to support binary-encoded (not JSON-encoded) syntax = "proto3";.

Many syntax = "proto2"; descriptor files will also work, as long as they don't use proto2 features, like groups.

The generated optional record fields will use Nothing instead of the default values.

We do not support extensions.

We do not support services.

At the time of this writing, we pass all 651 of the Google conformance tests for binary-wire-format proto3 v3.14.0. (https://github.com/protocolbuffers/protobuf/blob/f763a2a86084371fd0da95f3eeb879c2ff26b06d/CHANGES.txt#L223)

See the conformance/README.md in this repository for details.

We also have our own unit tests, see test/README.md in this repository.

The shell.nix environment provides

• The Purescript toolchain: purs, spago, and npm.
• The protoc compiler
• The protoc-gen-purescript executable plugin for protoc on the PATH so that protoc can find it.

$ nix-shell

Purescript Protobuf development environment.
To build purescript-protobuf, run:

    npm install
    spago build

To generate Purescript .purs files from .proto files, run:

    protoc --purescript_out=path_to_output file.proto

[nix-shell]$

If you don't want to use Nix, then install the Purescript toolchain and protoc, and add the executable script bin/protoc-gen-purescript to your PATH.

The code generator will use the package import statement in the .proto file and the base .proto file name as the Purescript module name for that file.

A message in a shapes.proto descriptor file declared as

package interproc;

message Rectangle {
  double width = 1;
  double height = 2;
}

will export these four names from module Interproc.Shapes in a generated shapes.Interproc.purs file.

1. A message data type

   newtype Rectangle = Rectangle { width :: Maybe Number, height :: Maybe Number }

   The message data type will also include an __unknown_fields array field for holding received fields which were not in the descriptor .proto file. We can completely ignore __unknown_fields if we want to.

2. A message maker which constructs a message from a Record with some message fields

   mkRectangle :: forall r. Record r -> Rectangle

   All message fields are optional, and can be elided when making a message. There are some extra type constraints, not shown here, which will cause a compiler error if we try to add a field which is not in the message data type.

   If we want the compiler to check that we've explicitly supplied all the fields, then we can use the ordinary message data type constructor Rectangle.

3. A message encoder which works with purescript-arraybuffer-builder

   putRectangle :: forall m. MonadEffect m => Rectangle -> PutM m Unit

4. A message decoder which works with purescript-parsing-dataview

   parseRectangle :: forall m. MonadEffect m => Int -> ParserT DataView m Rectangle

   The message decoder needs an argument which tells it the length of the message which it’s about to decode, because “the Protocol Buffer wire format is not self-delimiting.”

In our program, our imports will look something like this. The only module from this package which we will import into our program will be the Protobuf.Library module. We'll import the message modules from the generated .purs files. We'll also import modules for reading and writing ArrayBuffers.

import Protobuf.Library (Bytes(..), parseMaybe)
import Interproc.Shapes (Rectangle, mkRectangle, putRectangle, parseRectangle)
import Text.Parsing.Parser (runParserT, ParseError)
import Data.ArrayBuffer.Builder (execPutM)
import Data.ArrayBuffer.DataView (whole)
import Data.ArrayBuffer.ArrayBuffer (byteLength)
import Data.Tuple (Tuple)
import Data.Newtype (unwrap)

This is how we serialize a Rectangle to an ArrayBuffer. We must be in a MonadEffect.

do
    arraybuffer <- execPutM $ putRectangle $ mkRectangle
        { width: Just 3.0
        , height: Just 4.0
        }

Next we'll deserialize Rectangle from the ArrayBuffer that we just made.

    result :: Either ParseError (Tuple Number Number)
      <- runParserT (whole arraybuffer) $ do

        rectangle :: Rectangle <- parseRectangle (byteLength arraybuffer)

At this point we've consumed all of the parser input and constructed our Rectangle message, but we're not finished parsing. We want to “validate” the Rectangle message to make sure it has all of the fields that we require, because in proto3, all fields are optional.

Fortunately we are already in the ParserT monad, so we can do better than to “validate”: Parse, don't validate.

We will construct a Tuple Number Number with the width and height of the Rectangle. If the width or height are missing from the Rectangle message, then we will fail in the ParserT monad.

For this validation step, pattern matching on the Rectangle message type works well, so we could validate this way:

        case rectangle of
            Rectangle { width: Just width, height: Just height } ->
                pure $ Tuple width height
            _ -> fail "Missing required width or height"

Or we might want to use parseMaybe, one of the convenience parsing functions exported by Protobuf.Library, for more fine-grained validation:

        width <- parseMaybe "Missing required width" (unwrap rectangle).width
        height <- parseMaybe "Missing required height" (unwrap rectangle).height
        pure $ Tuple width height

And now the result is either a parsing error or a fully validated rectangle.

The generated code modules will import modules from this package.

The generated code depends on packages

  , "protobuf"
  , "arraybuffer"
  , "arraybuffer-types"
  , "arraybuffer-builder"
  , "parsing"
  , "parsing-dataview"
  , "uint"
  , "longs"
  , "text-encoding"

which are all in package-sets.

It also depends on the Javascript package long.

All of the generated message types have instances of Eq, Show, Generic, NewType.

The purescript-protobuf repository contains three executable Node.js programs which use code generated by purescript-protobuf. Refer to these for further examples of how to use the generated code.

1. The protoc compiler plugin. The code generator imports generated code. Trippy, right? This program literally writes itself.
2. The unit test suite
3. The Google conformance test program

When the decode parser encounters an invalid encoding in the protobuf input stream then it will fail to parse.

When Text.Parsing.Parser.ParserT fails it will return a ParseError String (Position {line::Int,column::Int}).

The byte offset at which the parse failure occured is given by the formula column - 1.

The path to the protobuf definition which failed to parse will be included in the ParseError String and delimited by '/', something like "Message1 / string_field_1 / Invalid UTF8 encoding.".

The Protobuf import statement allows Protobuf messages to have fields consisting of Protobuf messages imported from another file, and qualified by the package name in that file. In order to generate the correct Purescript module name qualifier on the types of imported message fields, the code generator must be able to lookup the package name statement in the imported file.

For that reason, we can only use top-level (not nested) message and enum types from a Protobuf import.

The generated Purescript code will usually have module imports which cause the purs compiler to emit warnings. We beg your pardon.

If we want to run the .proto → .purs generation step as part of a pure Nix derivation, then import the top-level default.nix from this repository as a nativeBuildInput.

Then protoc --purescript_out=path_to_output file.proto will be runnable in our derivation phases.

See the nix/demo.nix file for an example.

Pull requests welcome.