The Buffrs Book

Buffrs manages versioned Protocol Buffer and gRPC API packages. It helps teams publish shared API definitions, consume them with full SemVer support, and keep migrations manageable with lockfiles and multi-version installs when needed.

Quick Start

# Install Buffrs from GitHub Releases or build it from source first

# Check for an update later
buffrs self-update --check-only

# Initialize a new API package
buffrs init --api

# Add a dependency
buffrs add --registry https://your.registry.com datatypes/user@^1.2.0

# Publish your package
buffrs publish

Get started →

Why Buffrs?

• 📦 Full SemVer support for gRPC APIs: declare compatible version ranges instead of pinning everything by hand
• 🔀 Multi-version support for staged migrations: install two API versions side by side when different consumers need different releases
• 🏷️ Package aliases for side-by-side versions: name each dependency entry clearly when one manifest references multiple versions of the same API
• 🛠️ Build tool integration: works with Cargo, CMake, Python, and other generated-code workflows

Documentation

• Getting Started: Install Buffrs and create your first package.
• Buffrs Guide: Learn core concepts and best practices.
• Commands Reference: Browse the full CLI reference.
• FAQ: Find common questions and troubleshooting guidance.

Need help? Check the FAQ or visit our GitHub repository.

Getting Started

Getting Started

To get started with buffrs, install it and set up your first package.

• Installation
• First steps with Buffrs

Installation

Install from GitHub Releases

Download the appropriate archive for your platform from GitHub Releases, extract it, and place the buffrs binary on your PATH.

Use this option if you want the published fork binary exactly as released internally.

Build from Source

Alternatively, clone the Buffrs Repository and build locally:

git clone https://github.com/globusmedical/ext-buffrs
cd ext-buffrs
cargo install --path .

Update an Existing Installation

If buffrs is already installed, use the built-in updater to check for and install the latest published release:

buffrs self-update --check-only
buffrs self-update

Verify Installation

buffrs --version

Authenticate with a Registry

To publish packages or access private dependencies, authenticate with your registry:

buffrs login --registry https://your-registry.example.com/artifactory

You’ll be prompted for an authentication token. Contact your registry administrator for credentials.

Tip

You can authenticate with multiple registries. Each registry uses separate credentials stored securely in your ~/.config/buffrs/credentials.toml.

Next: First Steps with Buffrs

First Steps with Buffrs

First Steps with Buffrs

Let’s create your first Protocol Buffer package with Buffrs.

1. Initialize a New Package

Create a new API package:

mkdir web-server && cd web-server
buffrs init --api

This creates:

.
├── Proto.toml          # Package manifest
└── proto/              # Your .proto files
    └── vendor/         # Dependencies install here

Note

Use --lib for library packages, or omit both flags for consumer-only projects (e.g., server implementations without protobuf definitions to publish).

2. Review the Manifest

The Proto.toml file defines your package:

[package]
name = "web-server"
version = "0.1.0"
type = "api"

[dependencies]

3. Add a Dependency

Add a package from your registry:

buffrs add --registry https://your.registry.com datatypes/user@=0.1.0

This updates Proto.toml:

[dependencies.user]
version = "=0.1.0"
repository = "datatypes"
registry = "https://your.registry.com/"

4. Install Dependencies

Download and set up dependencies:

buffrs install

Dependencies are placed in proto/vendor/ and ready to import.

What’s Next?

• Buffrs Guide: Learn core concepts and workflows
• Commands Reference: Explore all CLI commands
• Package Types: Understand APIs vs libraries

Buffrs Guide

Buffrs Guide

This guide will give you all that you need to know about how to use Buffrs to develop and consume protocol buffer packages.

• Why Buffrs Exists
• The Framework
• Package Types
• Dependencies
• Creating a Package
• Consuming Packages
• Local Dependencies
• Import System
• Project Layout
• Manifest vs Lockfile
• Continuous Integration
• Buffrs Home

Why Buffrs Exists

Modern gRPC based software platforms make extensive use of protocol buffers to define and implement inter-service APIs. While the open-source protocol buffers ecosystem makes it very easy to get started with a few services and a handful of messages, recurring difficulties occur with scaling gRPC APIs beyond immediate repository boundaries for two main reasons:

1. Limited tooling for packaging, publishing, and distributing protocol buffer definitions.
2. The lack of widely-adopted best practices for API decomposition and reuse.

To overcome these problems we developed Buffrs - a package manager for protocol buffers. Using Buffrs, engineers package protocol buffer definitions, publish them to a shared registry, and integrate them seamlessly into their projects using versioned API dependencies and batteries-included build tool integration.

For the practical workflow, continue with the Getting Started guide and the rest of this book.

The Framework

The Framework

To understand the distribution and decomposition framework that Buffrs provides it is useful to understand which properties of API management systems are desirable and why. Key aspects include:

Versioning

A versioning scheme for APIs — similar to versioned library dependencies — explicitly encodes compatibility properties. This allows developers to make either backwards-compatible or breaking changes and encode the compatibility guarantees in the API version: a minor version upgrade “just works”, while a new major version API may require manual migration/adaption in the consuming server/client implementations.

Source Compatibility

Given versioned protocol buffer APIs with explicit compatibility guarantees, it is desirable to have a system in which wire-format compatible APIs are also source-code compatible. This means that engineers can update minor patches automatically and that APIs that build upon the same protocol buffer types can be used with the same generated code types. This is especially important in strict languages like Rust (due to, e.g., the orphan rule).

Composition

Enabling engineers to reuse and combine code in order to build new systems from existing building blocks is a key feature. A common composition scheme with protocol buffers is to use a set of base messages and data types across many APIs.

Discoverability

Before engineers can reuse and compose protocol buffers, they need to be able to discover and understand what APIs exist and how to use them. Discoverability is a significant accelerator for engineering productivity and helps developers stay abreast of the evolution of APIs and architecture.

Protocol Buffers as a First Class Citizen

Buffrs decided to take a unique approach compared to other protocol buffer management systems (or systems capable of distributing protocol buffers) like buf and bazel. Buffrs is essentially a package manager for protocol buffers. Protocol buffers are treated as a first class citizen within Buffrs – which means that they are seen as distributable units called packages.

Complex software projects frequently turn out to depend on different versions of the same APIs over time and individual components in those systems may have diverging compatibility guarantees. Internal services may break their API backwards compatibility, way more frequent than external gateways that serve millions of users.

The three fundamental ideas of Buffrs to enable a stable management approach for scenarios like the above are:

Buffrs Packages

A closed and meaningful unit of protocol buffers which enables either productivity through shared types (e.g., Google’s google.protobuf.Timestamp) or describes a domain / API (e.g., service GeoLocator).

Buffrs Registry

A central registry for managing and hosting packages, documentation, enabling engineers to search and find and to implement access control.

Local Code Generation

The last block is local code generation. This enables projects to freely diverge in their actual implementations by choosing a code generation implementation which fits their specific needs while still maintaining complete wire compatibility. This prevents language lock-ins and makes any version of any package usable for any language that exists today or might exist in the future (given it has a protobuf code generator).

Package Types

Package Types

Buffrs makes distinctions between two different packages:

[package]
type = "lib" | "api"

This is used in order to fuel composition and type reuse across APIs and thus enable shared types + wire compatibility.

lib – Libraries

Libraries contain atomic and composite type definitions that describe a domain. (e.g. physics, auth, time etc.). This pattern is really useful for scaling systems and maintaining dozens of APIs as they can share types and thus get the aforementioned benefits of source code and wire compatibility for free.

An example of a proto library named time that depends on google:

[package]
name = "time"
type = "lib"
version = "1.0.0"

[dependencies]
google = { version = "=1.0.0", ... }

syntax = "proto3";

package time;

import "google/timestamp.proto";

/// A timestamp wrapper for various formats
message Time {
  oneof format {
    string rfc3339 = 1;
    uint64 unix = 2;
    google.protobuf.Timestamp google = 3;
    ..
  }
}

api – APIs

APIs are the next logical building block for real world systems – they define services and RPCs that your server can implement. You can use the aforementioned libraries to fuel your development / API definition experience.

A good example of an API could be an imaginary logging service that makes use of the just declared time.Time:

[package]
name = "logging"
type = "api"
version = "1.0.0"

[dependencies]
time = { version = "=1.0.0", ... }

syntax = "proto3";

package logging;

import "time/time.proto";

service Logging {
  rpc critical(LogInput) returns (LogOutput);
  rpc telemetry(LogInput) returns (LogOutput);
  rpc healthiness(HealthInput) returns (HealthOutput);
}

message LogInput { string context = 1; time.Time timestamp = 2; }
message LogOutput { }

message HealthInput { bool db = 1; }
message HealthOutput { }

Creating a Package

Creating a Package

Creating a new, consumable, Buffrs package can be done in four steps:

1. Initialize a project using buffrs init
2. Set metadata, package type, version correctly and declare the dependencies that you want to use.
3. Declare message, enum and service types you want to publish in the newly created proto folder using .proto files.
4. Publish to a registry using buffrs publish.

Example: Publishing physics

Initialize Your Project

Start by initializing a new Buffrs project using the buffrs init command. This will set up the basic structure for your package and allow you to manage your package’s metadata and dependencies.

mkdir physics
cd physics
buffrs init --lib

Define Package Metadata and Dependencies

In your project folder, locate the Proto.toml file. This is where you’ll specify your package’s metadata and dependencies. Open this file in a text editor.

Here’s an example Proto.toml file:

# Package metadata
[package]
package = "physics"
version = "1.0.0"
type = "lib"
description = "A library containing physic related types"

# Declare dependencies (none in this case)
[dependencies]

Define Message, Enum, and Service Types

Inside your project directory, buffrs init created a proto folder. This is where you will store your .proto files that define your library or api.

An example file structure:

physics
├── Proto.toml
└── proto
    ├── temperature.proto
    ├── mass.proto
    └── vendor

Write your Protocol Buffer definitions in these .proto files. Here’s a simple example of the temperature.proto file that could be in a physics library:

syntax = "proto3";

package physics.temperature;

// Define temperature units
enum TemperatureUnit {
  CELSIUS = 0;
  FAHRENHEIT = 1;
  KELVIN = 2;
}

// Define a message for temperature
message Temperature {
  double value = 1;            // Temperature value
  TemperatureUnit unit = 2;    // Temperature unit (Celsius, Fahrenheit, Kelvin)
}

Publish Your Package

Once you’ve set up your Buffrs package and defined your Protocol Buffer definitions, you can publish it to a registry using the buffrs publish command. Make sure you’re logged in to the registry if required.

buffrs publish --registry https://your.registry.com --repository tutorial

Your package will be uploaded to the registry, and others can now consume it using Buffrs.

Congratulations! You’ve successfully published your Buffrs package. Other developers can now use your Protocol Buffer definitions by adding your package as a dependency in their Buffrs projects.

That’s it! You’ve created a Buffrs package that others can easily consume. Remember to keep your package up-to-date and well-documented to make it even more valuable to the community.

Consuming Packages

Consuming Packages

As described in the package types section you can declare dependencies through your project’s Proto.toml. This is true for both libraries and APIs. But what if you want to implement your server and you need to consume buffrs packages that contain your type and service definitions?

So there are three scenarios in which you would want to depend on other packages:

a) You are defining a library and you want to make use of an external type coming from another library (e.g. google for basic types such as google.None). b) You are defining an API and you want to use libraries to reuse common types for that domain (e.g. time or physics) c) You are implementing your server and you want to get access to API definitions to generate bindings.

The good news are: They are all achieved in a similar fashion. You make use of the [dependencies] key in your manifest to declare the packages that your projects needs – either to publish another package or to compile to protos to bindings.

Examples

Libraries & APIs

This section is identical for libraries and APIs.

An example of the time library reusing the google library:

[package]
name = "time"
type = "lib"
version = "1.0.0"

[dependencies]
google = { version = "=1.0.0", registry = "<your-registry>", repository = "<your-repository>" }

Running buffrs install yields you with the following filesystem:

time
├── Proto.toml
└── proto
    ├── time.proto
    └── vendor
        ├── time
        ├   └── ..
        └── google
            ├── any.proto
            ├── ..
            ├── struct.proto
            └── timestamp.proto

You can now develop your library and publish it using buffrs publish.

Servers

If you want to implement your server and thus use e.g. a logging API the only major difference is the lack of the [package] section in your manifest.

[dependencies]
logging = { version = "=1.0.0", registry = "<your-registry>", repository = "<your-repository>" }

Multi-Version Dependencies

In larger projects, you may need different parts of your codebase to use different versions of the same package. Starting with buffrs 1.0.0, you can enable this with resolver = "multiversion":

[dependencies]
logging-v1 = { package = "logging", version = "=1.0.0", resolver = "multiversion", registry = "<your-registry>", repository = "<your-repository>" }
logging-v2 = { package = "logging", version = "=2.0.0", resolver = "multiversion", registry = "<your-registry>", repository = "<your-repository>" }

See Multi-Version Dependencies for complete documentation.

Running a buffrs install yields you the very same as above, except for the omitted local package and the logging dependency instead of time.

.
├── Proto.toml
└── proto
    └── vendor
        └── logging
            └── logger.proto

Import System

Import System

To reuse already defined messages, protobuf files can be imported from both dependencies and the managed package itself. Unique identification of the files is made available through the name of the package as declared in Proto.toml which is used as root of the imports.

For a dependency units:

units
├── Proto.toml
├── weight.proto
└── length.proto

and a package named physic:

physic
├── Proto.toml
└── proto
    ├── root.proto
    ├── length.proto
    └── calculations
        ├── distance.proto
        └── graph.proto

messages can be imported from both packages relative to their root:

// root.proto
syntax = "proto3";

package physic;

import "physic/length.proto";
import "physic/calculations/distance.proto";
import "units/length.proto";

message Lengths {
    units.Meter meter = 1;
    physic.Parsec parsec = 2;
    physic.calculations.Haversine = 3;
}

Project Layout

Project Layout

To get an understanding of the project layout that buffers uses it is helpful to start in a clean manner and introspect the outcome.

Lets create a new clean directory initialize for our physic library.

mkdir physic
cd physic
buffrs init --lib

This will initialize the following project structure:

physic
├── Proto.toml
└── proto
    └── vendor

This will create the Proto.toml file which is the manifest file that buffrs uses. The proto directory, which is the source directory for all your protocol buffer definitions and the proto/vendor directory, which contains external protocol buffers.

Important: The vendor directory is managed by Buffrs, all manual changes will be overridden / can cause unreproducible behavior.

Manifest vs Lockfile

Manifest (Proto.toml) vs Lockfile (Proto.lock)

Manifest – Proto.toml

Purpose: The Manifest (Proto.toml) serves as a specification file for the current package. It includes metadata, dependencies, and other package-related information.

Contents:

• Package Name: A unique name for the package.
• Version: Specifies the package version.
• Type: Specifies the type of this package (see Package Types).
• Description: Provides a brief description of the package.
• Metadata: Additional metadata like package category, tags, and any other relevant package details.
• Dependencies: Contains package names, version constraints, registry locations etc of the dependencies of the current package.

Usage: The Proto.toml is used to define the package’s characteristics, metadata, and its dependencies, making it a comprehensive specification format. This file is included in compressed package artefacts and distributed alongside. For a usage guide see Creating A New Package.

Lockfile – Proto.lock

Purpose: The Lockfile (Proto.lock) is a separate, autogenerated and automanaged, file that records the exact versions of packages, including their transitive dependencies, which have been successfully resolved and installed.

Contents: It contains a detailed record of the package versions used, the registry or package source used during the installation and cryptographic hashes to ensure package integrity.

Usage: The lockfile is crucial for ensuring the reproducibility of installations. It guarantees that the same package versions are installed, regardless of changes in the upstream package registry, ensuring consistency across different installations.

Buffrs Home

Buffrs Home

$HOME/.buffrs

The buffrs home directory is a place for global configuration and data belonging to buffrs such as credentials and a cache. You should not need to interact with this folder manually.

The home directory that is used by buffrs can be configured via the BUFFRS_HOME environment variable. This enables you do override the default location in case you want to keep your home directory clean.

Multi-Version Dependencies

Buffrs can allow multiple versions of the same package to coexist in one dependency graph. This is most useful during gRPC API migrations, when one part of a system still needs an older API while another part has already moved to a newer release.

The Problem

By default, buffrs enforces single-version resolution: each package name can only resolve to one version. This prevents diamond dependency conflicts but can be limiting in large codebases where:

• Different teams work on different features requiring different API versions
• Gradual migration between API versions is in progress
• Independent subgraphs legitimately need different versions

Critically, consumers don’t control upstream proto definitions. When you depend on lib-algo-base@0.1.2 and lib-algo-base@0.1.3, you cannot mandate that the upstream maintainer change their package gm.algo.base; declaration. Buffrs solves this by automatically rewriting proto namespaces when multi-version is enabled.

Enabling Multi-Version

To allow multiple versions of a specific dependency, add resolver = "multiversion" to that dependency in your Proto.toml:

[dependencies]
# Old version we still need
lib-algo-base = { version = "=0.1.2", resolver = "multiversion" }

# New version we're migrating to
[dependencies.lib-algo-base-new]
package = "lib-algo-base"
version = "=0.1.3"
resolver = "multiversion"

The lib-algo-base-new key is a package alias. Aliases are how one Proto.toml can refer to two versions of the same package at the same time, so they are part of the multi-version workflow rather than a separate dependency model.

This grants permission for buffrs to resolve multiple versions of lib-algo-base if the dependency constraints require it. It does not force duplicates—if a single version satisfies all constraints, only one version is resolved.

Which dependencies need the flag? All dependencies of the same package that may coexist need resolver = "multiversion". The flag means “I accept coexisting with other versions of this package.”

Automatic Namespace Rewriting

When multi-version resolution results in multiple versions of the same package, buffrs automatically rewrites protobuf package declarations to include version information. This ensures generated code has unique symbols without requiring upstream changes.

How It Works

When you run buffrs install with multi-version enabled, buffrs:

1. Detects packages that have multiple versions resolved
2. Rewrites package declarations to include a version suffix
3. Writes the modified files to the vendor directory

Example transformation:

// Original (in published package):
package gm.algo.base;

// After buffrs install (version 0.1.2):
package gm.algo.base._v0_1_2;

// After buffrs install (version 0.1.3-SPINE-4384):
package gm.algo.base._v0_1_3_SPINE_4384;

Version Suffix Format

The version suffix follows the format _v<major>_<minor>_<patch> with special characters sanitized:

Impact on C++ Code

With multi-version enabled, C++ code MUST use versioned namespaces.

The rewritten proto packages result in unique C++ namespaces:

Consumer Code Example

#include <gm/algo/base/_v0_1_2/types.pb.h>
#include <gm/algo/base/_v0_1_3/types.pb.h>

// Option 1: Use full versioned namespace
void process_old(const gm::algo::base::_v0_1_2::SomeMessage& msg);
void process_new(const gm::algo::base::_v0_1_3::SomeMessage& msg);

// Option 2: Use namespace aliases (recommended)
namespace algo_old = gm::algo::base::_v0_1_2;
namespace algo_new = gm::algo::base::_v0_1_3;

void adapt(const algo_old::Request& old_req) {
    algo_new::Request new_req;
    // ... convert between versions
}

Rust Consumer Code Example

#![allow(unused)]
fn main() {
// The generated Rust modules also use versioned names
mod gm {
    pub mod algo {
        pub mod base {
            pub mod _v0_1_2 {
                include!(concat!(env!("OUT_DIR"), "/gm.algo.base._v0_1_2.rs"));
            }
            pub mod _v0_1_3 {
                include!(concat!(env!("OUT_DIR"), "/gm.algo.base._v0_1_3.rs"));
            }
        }
    }
}

use gm::algo::base::_v0_1_2 as algo_old;
use gm::algo::base::_v0_1_3 as algo_new;
}

Impact on Python Code

Python projects using single-version dependencies are NOT affected. Multi-version namespace rewriting ONLY occurs when multiple versions of the same package are actually resolved. If your Python project uses only one version of each dependency (which is the common case for isolated Python scripts), no changes are needed.

With multi-version enabled, Python imports MUST use versioned module paths.

The rewritten proto packages result in unique Python module paths:

Single-Version Python (No Changes Required)

If your Python project only depends on one version of each API, nothing changes:

# Proto.toml (single version - no multi-version flag needed)
# [dependencies]
# lib-algo-base = { version = "0.1.2", ... }

# Your Python code continues to work unchanged:
from gm.algo import base_pb2
msg = base_pb2.SomeMessage()

Multi-Version Python Consumer Example

When using multi-version, Python imports must reference the versioned module:

# Proto.toml has:
# lib-algo-base = { version = "=0.1.2", ..., resolver = "multiversion" }
# lib-algo-base-new = { package = "lib-algo-base", version = "=0.1.3", ..., resolver = "multiversion" }

# Import from versioned module paths
from gm.algo.base._v0_1_2 import base_pb2 as base_v012
from gm.algo.base._v0_1_3 import base_pb2 as base_v013

# Use with version-specific aliases
msg_old = base_v012.SomeMessage(value="old")
msg_new = base_v013.SomeMessage(value="new", extra="field")

# Convert between versions if needed
def upgrade_message(old_msg: base_v012.SomeMessage) -> base_v013.SomeMessage:
    return base_v013.SomeMessage(
        value=old_msg.value,
        extra="default"
    )

Python gRPC Services with Multi-Version

# gRPC stubs also follow the versioned module pattern
from gm.algo.base._v0_1_2 import service_pb2_grpc as service_v012
from gm.algo.base._v0_1_3 import service_pb2_grpc as service_v013

# Create clients for different API versions
channel = grpc.insecure_channel('localhost:50051')
client_old = service_v012.BaseServiceStub(channel)
client_new = service_v013.BaseServiceStub(channel)

Running buffrs install for Python

Python projects typically run buffrs install as part of their setup. After installation, run the protobuf compiler to generate Python code:

# Install buffrs dependencies
buffrs install

# Generate Python code from proto files (example using grpc_tools)
python -m grpc_tools.protoc \
    -I proto \
    -I proto/vendor \
    --python_out=. \
    --grpc_python_out=. \
    proto/vendor/lib-algo-base@0.1.2/*.proto \
    proto/vendor/lib-algo-base@0.1.3/*.proto

The generated _pb2.py and _pb2_grpc.py files will be placed according to the rewritten package names, creating the versioned module structure automatically.

Vendor Layout

When multi-version resolution results in multiple versions of the same package, buffrs uses version-qualified directory names:

proto/vendor/
├── lib-algo-base@0.1.2/
│   └── ...
├── lib-algo-base@0.1.3/
│   └── ...
└── other-package/
    └── ...

Packages without version conflicts continue to use simple directory names (e.g., other-package/).

Namespace Overlap Policy

When multiple versions of a package exist, buffrs provides a namespace_overlap policy to control how conflicts are handled.

rewrite (default)

Automatically rewrite package declarations to include version suffixes. This is the default and recommended policy:

[dependencies.my-package]
version = "=1.0.0"
resolver = "multiversion"
# namespace_overlap = "rewrite" is implied

identical_only

Skip rewriting and allow overlap only if the proto file content hashes match (i.e., the files are identical):

[dependencies.my-package]
version = "=1.0.0"
resolver = "multiversion"
namespace_overlap = "identical_only"

This is useful when you know different versions share common base types with identical definitions.

forbidden

Fail if multi-version would result in namespace overlap. Use this when you cannot adapt consumer code to use versioned namespaces:

[dependencies.my-package]
version = "=1.0.0"
resolver = "multiversion"
namespace_overlap = "forbidden"

Best Practices

1. Use sparingly: Multi-version should be the exception, not the rule. Prefer updating all consumers to a single version when possible.

2. Plan for versioned namespaces: When enabling multi-version, anticipate that your C++, Rust, and Python code will need to use versioned namespace/module prefixes like _v0_1_2.

3. Create namespace/module aliases: Make your code cleaner with aliases:

   // C++
   namespace algo_v1 = gm::algo::base::_v0_1_2;
   namespace algo_v2 = gm::algo::base::_v0_1_3;
   

   # Python
   from gm.algo.base._v0_1_2 import base_pb2 as algo_v1
   from gm.algo.base._v0_1_3 import base_pb2 as algo_v2
   

4. Audit your dependency graph: Before enabling multi-version, understand why different versions are needed. Sometimes the root cause is an outdated transitive dependency that should be updated.

5. Test thoroughly: Multiple versions can introduce subtle runtime issues. Ensure your test coverage includes scenarios with multi-version dependencies.

6. Isolated Python projects are safe: If your Python project only uses one version of each dependency, multi-version changes won’t affect you. The namespace rewriting only activates when multiple versions are actually resolved.

Monorepo Considerations

In monorepos with multiple Proto.toml files (e.g., one per CMake target), different manifests can independently resolve different versions. This is safe as long as the final linked binary doesn’t include conflicting protobuf namespaces.

See the CMake Integration section for information on link-time safety checks.

CMake Integration

Buffrs emits metadata in _buffrs_meta/ that build systems can use for link-unit validation:

• graph.json - Full dependency graph with versions, registries, relationships
• namespaces.json - Mapping from protobuf namespaces to packages
• buffrs.cmake - CMake variables for integration (data only)

Link-Unit Validation

Build systems should implement their own buffrs_validate_link_unit() function to check whether a final binary links multiple targets with conflicting namespace sources. The validation logic:

1. Collect BUFFRS_NAMESPACE_SOURCES properties from all linked targets
2. For each namespace, verify all sources resolve to the same package@version
3. Fail at configure time if conflicts are detected

Example CMake implementation pattern:

function(buffrs_validate_link_unit TARGET)
    # Collect namespace sources from target and all its dependencies
    # Check for conflicts (same namespace from different package@version)
    # Report error if conflicts found
endfunction()

This catches multi-version namespace conflicts at configure time rather than experiencing mysterious runtime failures.

Limitations

• Multi-version resolution is per-Proto.toml, not global across a monorepo
• Build systems must implement their own link-unit validation using the emitted metadata

Continuous Integration

To utilize continuous integration for your Buffrs package (e.g. to automate code review and publishing of your packages) you can utilize the following templates for GitHub Actions and GitLab CI:

GitHub Actions

name: Buffrs

on:
  push:
    branches:
      - "*"
  tags:
    - "*"

env:
  BUFFRS_VERSION: 1.2.6
  REGISTRY: https://<org>.jfrog.io/artifactoy
  REPOSITORY: your-artifactory-repo

jobs:
  verify:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Install Rust
        uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          toolchain: nightly

      - name: Set up Rust environment
        run: |
          rustup target add aarch64-unknown-linux-gnu
        shell: bash

      - name: Install Buffrs
        run: |
          curl -L -o buffrs.tar.gz https://github.com/globusmedical/ext-buffrs/releases/download/gm%2Fv${BUFFRS_VERSION}/v${BUFFRS_VERSION}-x86_64-unknown-linux-gnu.tar.gz
          tar -xzf buffrs.tar.gz
          install -m 755 buffrs "$HOME/.local/bin/buffrs"
          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
        shell: bash

      - name: Verify
        run: |
          echo $TOKEN | buffrs login --registry $REGISTRY
          buffrs lint
        env:
          TOKEN: ${{ secrets.BUFFRS_TOKEN }}
        shell: bash

  publish:
    runs-on: ubuntu-latest
    needs: build
    if: startsWith(github.ref, 'refs/tags/')

    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Install Buffrs
        run: |
          curl -L -o buffrs.tar.gz https://github.com/globusmedical/ext-buffrs/releases/download/gm%2Fv${BUFFRS_VERSION}/v${BUFFRS_VERSION}-x86_64-unknown-linux-gnu.tar.gz
          tar -xzf buffrs.tar.gz
          install -m 755 buffrs "$HOME/.local/bin/buffrs"
          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
        shell: bash

      - name: Publish on tag
        run: |
          echo $TOKEN | buffrs login --registry $REGISTRY
          buffrs publish --registry $REGISTRY --repository $REPOSITORY
        env:
          TOKEN: ${{ secrets.BUFFRS_TOKEN }}
        shell: bash

GitLab CI

stages:
  - verify
  - publish

variables:
  BUFFRS_VERSION: 1.2.6
  TOKEN: $BUFFRS_TOKEN # Your secret artifactory token
  REGISTRY: https://<org>.jfrog.io/artifactory
  REPOSITORY: your-artifactory-repo

verify:
  stage: verify
  script:
    - curl -L -o buffrs.tar.gz https://github.com/globusmedical/ext-buffrs/releases/download/gm%2Fv${BUFFRS_VERSION}/v${BUFFRS_VERSION}-x86_64-unknown-linux-gnu.tar.gz
    - tar -xzf buffrs.tar.gz
    - install -m 755 buffrs "$HOME/.local/bin/buffrs"
    - export PATH="$HOME/.local/bin:$PATH"
    - echo $TOKEN | buffrs login --registry $REGISTRY
    - buffrs lint
  only:
    - branches

publish:
  stage: publish
  script:
    - curl -L -o buffrs.tar.gz https://github.com/globusmedical/ext-buffrs/releases/download/gm%2Fv${BUFFRS_VERSION}/v${BUFFRS_VERSION}-x86_64-unknown-linux-gnu.tar.gz
    - tar -xzf buffrs.tar.gz
    - install -m 755 buffrs "$HOME/.local/bin/buffrs"
    - export PATH="$HOME/.local/bin:$PATH"
    - echo $TOKEN | buffrs login --registry $REGISTRY
    - buffrs publish --registry $REGISTRY --repository $REPOSITORY
  only:
    - tags

Buffrs Reference

This section covers detailed documentation on the Buffrs Ecosystem / Framework.

Index

• Editions - Version compatibility and edition system
• The Manifest Format - Complete Proto.toml documentation
• Specifying Dependencies - Dependency declaration syntax
• Dependency Resolution - How buffrs resolves dependencies
• Protocol Buffer Rules - Validation rules for .proto files
• Configuration - Authentication, TLS, and proxy settings

Editions

Editions

Editions of buffrs mark a specific evolutionary state of the Proto.toml manifest format. The edition system exists to allow for format evolution while maintaining backward compatibility with existing packages.

Editions can be either explicitly stated in the Proto.toml or are automatically inlined once a package is created using buffrs. This ensures that you dont need to care about them as a user but get the benefits.

Note: The proto.toml edition is independent of the buffrs crate version. The edition only changes when the Proto.toml format itself changes. If you release a package with an edition that is incompatible with another one (e.g. if 0.7 is incompatible with 0.8) you will need to re-release the package for the new edition (by bumping the version, or overriding the existing package) to regain compatibility.

You may see errors like this if you try to consume (or work on) a package of another edition.

Error:   × could not deserialize Proto.toml
  ╰─▶ TOML parse error at line 1, column 1
        |
      1 | edition = "0.7"
        | ^^^^^^^^^^^^^^^
      unsupported manifest edition, supported editions of 1.0.0 are: 0.50

Edition Compatibility

Buffrs 1.0.0 supports the following editions:

Current Edition

edition = "0.50"

The proto.toml edition is independent of the buffrs program version. Edition 0.50 introduces multi-version dependencies in buffrs 1.0.0.

Buffrs 1.0.0 Features

Buffrs 1.0.0 introduces:

• Per-dependency multi-version support: Use resolver = "multiversion" to allow multiple versions of the same package
• Namespace overlap policies: Control how proto namespace collisions are handled with namespace_overlap
• Version-qualified vendor directories: Multi-version packages use name@version/ directory format

See Multi-Version Dependencies for details.

Protocol Buffer Rules

Protocol Buffer Rules

This specification defines rules enforced by Buffrs to prevent package colisions, and provide uniformity and transparency for package consumers.

Rules with the 00XX code define the package and filesystem layout, where as rules with a 01XX code enforce certain protocol buffer definition rules.

0000 – Package Name Prefix / Symmetry

Enforces that the Buffrs Package ID is used as the prefix for all protocol buffer package declarations.

So given a Buffrs Package with the ID physics this enforces that the package only contains protocol buffer package declarations matching physics|physics.*;

A violation would cause type colisions and ambiguity when trying to resolve a type.

0010 – Sub-Package Declaration

Enforces that subpackages are declared through a sensible folder structure. Given a Buffrs Package with the ID physics the protocol buffer file that declares package physics.units; has to be called proto/units.proto.

Nested subpackages are represented / grouped through folders. So if one wants to declare package physics.units.temperature; the respective file must be located at proto/units/temperature.proto.

0020 – Root Package Declaration

Enforces that only one file at a time declares the root package.

Namely: If a Buffrs Package with the ID physics is defined, the proto/physics.proto must declare the the same package in the protocol buffer syntax through package physics;.

Buffrs Commands

This section covers detailed documentation on operating the Buffrs CLI. Some of the content may also apply to the library, particularly the command module.

The goal is to provide sufficient information to use the CLI effectively to manage a Buffrs project, also also capture context and behaviour that may otherwise be only expressed in code.

The help command should also provide useful information and is expected to generally be more up-to-date.

Index

• General Commands
  • buffrs
  • buffrs help
• Build Commands
  • buffrs list
• Manifest Commands
  • buffrs add
  • buffrs remove
• Package Commands
  • buffrs init
  • buffrs lint
  • buffrs package
  • buffrs install
  • buffrs uninstall
• Publishing Commands
  • buffrs login
  • buffrs logout
  • buffrs publish

General Commands

General commands interface with the CLI itself, in order to obtain access to built-in help or version information.

Index

• buffrs
• buffrs help

buffrs

buffrs

The official Buffrs command-line interface.

Synopsis

buffrs

Description

When invoked without any arguments, the Buffrs binary defaults to printing out help information to the standard output.

This is equivalent to buffrs help, or invoking with the -h or --help flags.

Providing the -V or --version flags is also equivalent to buffrs version.

Output

Modern protobuf package management

Usage: buffrs <COMMAND>

Commands:
  init       Initializes a buffrs setup
  lint       Check rule violations for this package
  add        Adds dependencies to a manifest file
  remove     Removes dependencies from a manifest file
  package    Exports the current package into a distributable tgz archive
  publish    Packages and uploads this api to the registry
  install    Installs dependencies
  uninstall  Uninstalls dependencies
  list       Lists all protobuf files managed by Buffrs to stdout
  generate   Generate code from installed buffrs packages
  login      Logs you in for a registry
  logout     Logs you out from a registry
  help       Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version

buffrs help

buffrs help

Prints out information about how to use the CLI.

Synopsis

buffrs help [command]

Description

When called by itself, this command lists all the supported commands along with a brief description.

When called with a command argument, it will provide specific help for that command.

Passing the -h or --help flags is equivalent to invoking this command.

Examples

> buffrs help
Modern protobuf package management

Usage: buffrs <COMMAND>

Commands:
  init       Initializes a buffrs setup
  lint       Check rule violations for this package
  add        Adds dependencies to a manifest file
  remove     Removes dependencies from a manifest file
  package    Exports the current package into a distributable tgz archive
  publish    Packages and uploads this api to the registry
  install    Installs dependencies
  uninstall  Uninstalls dependencies
  generate   Generate code from installed buffrs packages
  login      Logs you in for a registry
  logout     Logs you out from a registry
  help       Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version

> buffrs help init
Initializes a buffrs setup

Usage: buffrs init [OPTIONS] [PACKAGE]

Arguments:
  [PACKAGE]  The package name used for initialization

Options:
      --lib      Sets up the package as lib
      --api      Sets up the package as api
  -h, --help     Print help
  -V, --version  Print version

Build Commands

Build commands assist implementation projects with generating code and documentation from Buffrs-managed protocol buffer files.

Index

• buffrs list

buffrs generate

buffrs list

buffrs list

Lists all protobuf files (.proto) managed by Buffrs to standard out.

Synopsis

buffrs list|ls

Description

This command lists all protobuf files managed by Buffrs. This way the output can be fed dynamically into external code generation tools like protoc.

Example

Given a project that depends on a physics package (that provides two .proto files: temperature.proto and mass.proto). Once it’s dependencies are installed, the structure of the filesystem would look similar to this:

.
├── Proto.toml
└── proto
    ├── some.proto
    └── vendor
        └── physics
            ├── Proto.toml
            ├── temperature.proto
            └── mass.proto

Using buffrs ls you can feed the installed protocol buffer files of all package dynamically into another command line tool like protoc to generate code, or run lints:

protoc --cpp_out ./cpp --include proto $(buffrs ls)

The raw output of buffrs ls would return the following three paths:

proto/some.proto proto/vendor/physics/temperature.proto proto/vendor/physics/mass.proto

Manifest Commands

Manifest commands manage the package’s manifest file, which declares important metadata like the package name, version and dependencies.

These commands can be used to include, remove and update dependencies while keeping the lockfile in sync.

Index

• buffrs add
• buffrs remove

buffrs add

Adds a new dependency to the package manifest.

Synopsis

buffrs add --registry <REGISTRY> <DEPENDENCY>

Description

The add command is the recommended way to include a new dependency in the current package. It modifies the local manifest file and will overwrite a pre-existing entry for the same dependency package if it exists.

Dependency locator format

The dependency should be specified with the repository, name and version according to the following format:

<repository>/<package>[@<version>]

Note: the version can be omitted (or set to @latest), in which case it will default to the latest version of this artifact in the registry.

The repository name should adhere to lower-kebab case (e.g. my-buffrs-repo). The package name has its own set of constraints as detailed in Package Name Specification. When specified, the version must adhere to the Semantic Version convention (e.g. 1.2.3) – see SemVer compatibility for more information.

Buffrs supports full SemVer version requirements here, including ^1.0.0, ~2.0.0, exact pins such as =1.2.3, and comparison ranges like >=1.2.0, <2.0.0.

Options

--resolver <RESOLVER>

Specify the dependency resolver strategy:

• default - Standard single-version resolution (default)
• multiversion - Allow multiple versions of this package to coexist

Use multiversion when you need different parts of your codebase to depend on different versions of the same package. See Multi-Version Dependencies for details.

buffrs add --registry <REGISTRY> --resolver multiversion <DEPENDENCY>

If one manifest needs two versions of the same package, declare one of them under an alias key in Proto.toml. That alias is how you refer to each version clearly after multi-version resolution.

Lockfile interaction

Currently adding a new dependency won’t automatically update the lockfile (Proto.lock). This is planned to change, but for now follow up with buffrs install after adding a new dependency to make sure your lockfile is kept in sync.

buffrs remove

buffrs remove

Removes an existing dependency from the package manifest.

Synopsis

buffrs remove <PACKAGE>

Description

The remove command is the recommended way to remove a dependency (identified by package name) from the manifest. It modifies the manifest and will produce an error if the specified package cannot be found. It implements the opposite operation from the add command.

Lockfile Interaction

Currently removing a dependency won’t automatically update the lockfile (Proto.lock). This is planned to change, but for now make sure to follow up with buffrs install after adding a new dependency to make sure your lockfile is kept in sync.

Package Commands

Package commands manage a local package, and are responsible for initializing the project structure, installing and uninstalling dependencies and building a publishable artifact from the current package state.

Index

• buffrs init
• buffrs lint
• buffrs package
• buffrs install
• buffrs uninstall

buffrs init

buffrs init

Initializes the current directory as a Buffrs project.

Synopsis

buffrs init [name]

buffrs init --lib [name]

buffrs init --api [name]

Description

This command prepares the current directory as a Buffrs project, by creating a manifest file (Proto.toml) as well as proto and proto/vendor directories.

By default, if no name is given, the current directory name is used as the package name. Note that there are special constraints on valid package names (see Package Name Specification for more details).

By default, if no package type is provided, impl (implementation) will be used. The meaning of this is described in Package Types.

buffrs new

buffrs init

Initializes a Buffrs project in a new folder created in the current directory.

Synopsis

buffrs new <NAME>

buffrs new --lib <NAME>

buffrs new --api <NAME>

Description

This command creates a new Buffrs project with the provided name by creating a manifest file (Proto.toml) as well as proto and proto/vendor directories in a new directory created at the current location.

By default, if no package type is provided, impl (implementation) will be used. The meaning of this is described in Package Types.

buffrs lint

Lints your protocol buffers for the (Buffrs Protocol Buffer Rules)

Synopsis

buffrs lint

Description

This command lints your local package (defined in proto/*.proto) for a set of rules defined in the (Buffrs Protocol Buffer Rules). They contain a set of rules ranging from style to package layout (like filenaming, package declaration etc.). This enables a common flavor to Buffrs packages which affect users.

One good example why this is required is the enforcement of euqality between the package declaration in the protocol buffers files (*.proto) and the Buffrs Package ID. This enables to expect that a Buffrs Package a declares the protocol buffer package a.* and prevents type colisions / ambiguity.

Example

Given a Buffrs Package abc that contains a protocol buffer file with the following file (proto/xyz.proto):

syntax = "proto3";

package xyz;

Executing buffrs lint would return a rule violation:

PackageName (https://globusmedical.github.io/ext-buffrs/reference/protocol-buffer-rules.html)

  × Make sure that the protobuf package name matches the buffer package name.
  ╰─▶   × package name is xyz but should have abc prefix

   ╭─[xyz.proto:1:1]
   ╰────
  help: Make sure the file name matches the package. For example, a package with the name `package.subpackage` should be stored in `proto/package/subpackage.proto`.

buffrs package

buffrs package

Generates a release tarball for the package in the current directory.

Synopsis

buffrs package

Options

• --dry-run: prevents buffrs from actually writing the tarball to the filesystem
• --output-directory: allows you to specify a directory to output the package
• --set-version: allows you to override the version set in the manifest

Description

Like the publish command, the package command bundles the package’s protocol buffer files and manifest into a gzip-compressed tarball. However, unlike the publish command it does not actually interact with the registry, instead it only writes the release tarball into the current directory. This is useful for manual distribution and for safely validating the package setup.

buffrs install

buffrs install

Downloads and installs dependencies specified in the manifest.

Synopsis

buffrs install

Description

This command manages Buffrs local set of installed dependency packages. It is meant to be run from the root of the Buffrs project, where the Proto.toml manifest file can be found. Currently, only API and implementation packages can have dependencies, so this command is only useful for those package types.

The installation process will respect the requirements stated in the manifest file – specifically, the version, registry and repository provided for each dependency. Each dependency may specify its own dependencies, via its manifest file, which will also be downloaded and its contents unpacked flatly to the local filesystem, under the shared proto/vendor path prefix (see Project Layout for more information).

By default, only one version of each package can be installed. If there is a conflicting requirement, installation will fail. However, starting with buffrs 1.0.0, you can enable multi-version dependencies by adding resolver = "multiversion" to allow multiple versions of a package to coexist.

Once installation has completed, the resolved packages versions will be frozen and captured in a Proto.lock file, which ensures that future installations (local or performed in another machine) will install the exact same dependency versions. This file is managed automatically and should be kept under version control, so that others can reproduce your local installation.

Lockfile

The install command manages the Buffrs lockfile (Proto.lock) automatically. If one doesn’t exist when the command is invoked, one is created after the installation has completed.

If dependencies have been added or removed since the last invocation, the lockfile will be modified accordingly. If the manifest requirements conflict with the lockfile (i.e. the manifest requests a different version than the one that was locked), installation will fail.

Versions are locked upon first installation, and will persist until the lockfile is regenerated with buffrs lock, dependencies are explicitly upgraded via buffrs update (or a manual edit of the manifest) or they have been removed. Once removed, if dependencies are added back again, a different version may be automatically selected and locked.

Transitive dependencies

Transitive dependencies are also managed by the current project’s lockfile. Even if dependencies provide their own lockfile, those won’t be used.

buffrs uninstall

buffrs uninstall

Deletes all installed dependencies from the local filesystem.

Synopsis

buffrs uninstall

Description

This command does the reverse operation from the install command, and will clear out the proto/vendors directory, thus removing all installed dependencies from the local filesystem. This is generally safe to do as the vendors directory is managed by Buffrs and shouldn’t contain any custom proto files. Subsequently invoking the install command should restore the exact same files, assuming the lockfile hasn’t been regenerated.

Publishing Commands

Publish commands interface with a remote registry and are primarily responsible for managing release publications.

Also in this category are commands to manage locally saved registry credentials.

Index

• buffrs login
• buffrs logout
• buffrs publish

buffrs login

buffrs login

Saves an authentication token in the credentials store.

Synopsis

buffrs login --registry <url>

Description

This command prompts for an API or Identity token that can be used to authenticate with Artifactory for downloading and publishing packages.

The token is currently stored in $HOME/.buffrs/credentials.toml in the following format:

[[credentials]]
uri = "https://example.com/artifactory"
token = "<secret>"

buffrs logout

buffrs logout

Removes an authentication token from the credentials store.

Synopsis

buffrs logout --registry <url>

Description

This command removes a previously saved token from the credentials store by its associated registry URL. Future invocations of publish and install that involve the given registry will then default to unauthenticated mode.

The credentials are currently stored in $HOME/.buffrs/credentials.toml.

buffrs publish

buffrs publish

Generates a release and publishes it to the specified registry.

Synopsis

buffrs publish [OPTIONS] --registry <REGISTRY> --repository <REPOSITORY>

Options

• --allow-dirty: allows publishing the package even if the repository has uncommitted changes.
• --dry-run: causes a release bundle to be generated but skips uploading to the registry.
• --set-version: allows you to override the version set in the manifest

Description

The publish command bundles the package’s protocol buffer files and manifest into a gzip-compressed tarball, which is then uploaded to the specified registry and repository for publication. Once published the artifact will be available for other packages to be installed as dependencies.

In order for this command to be successful, the registry must be reachable via the network, and if authorization is required, credentials must have been previously saved via a buffrs login invocation.

By default, Buffrs does not allow publishing packages from git repositories in a dirty state (note: this requires the git feature to be enabled). This behaviour can be overridden by passing the --allow-dirty flag.

Supported project types

Only Buffrs libraries and API packages can be packaged and published. More details in Package Types.

Library packages cannot have dependencies, so releasing this kind of package may fail if any are provided in the manifest. API dependencies on library packages is also forbidden and will cause publication to fail.

FAQ

Why doesn’t buffrs add, buffrs publish, or buffrs login work anymore?

Important

Recent versions of Buffrs support multiple registries. Commands now require the --registry flag.

We expanded Buffrs to handle connections to multiple registries simultaneously. Add --registry https://your-registry.com/artifactory to these commands:

buffrs add --registry https://your-registry.com/artifactory <package>
buffrs publish --registry https://your-registry.com/artifactory
buffrs login --registry https://your-registry.com/artifactory

Note

The buffrs login flag was renamed from --url to --registry for consistency.

Why is my credentials.toml file broken?

Important

The credentials file format changed to support multiple registries.

Old format (single registry):

[artifactory]
url = "https://org.jfrog.io/artifactory"
password = "some-token"

New format (multiple registries):

[[credentials]]
uri = "https://org1.jfrog.io/artifactory"
token = "some-token"

[[credentials]]
uri = "https://org2.jfrog.io/artifactory"
token = "some-other-token"

Tip

Run buffrs login --registry <url> to automatically update your credentials file.

Why can’t I log in with a username?

Important

Username-based authentication is no longer supported. Use tokens instead.

The --username flag has been removed in favor of token-based authentication. This enables support for:

• Identity tokens
• JWT tokens
• Encoded basic auth tokens

All authentication now uses the Authorization header for better security and flexibility.

Tip

Generate an access token from your registry provider and use buffrs login --registry <url> to store it.