In the past few parts, we talked about low-level technical features that OpenBao core maintainers and plugin authors can take advantage of to make secrets management safer and more scalable.

This part focuses on something that applies to operators of OpenBao: better operator experience for initial configuration. We focus on one question:

The answer lies in declarative self-initialization and declarative audit device creation.

These features available since OpenBao v2.4.0 allow operators to define the state of OpenBao prior to deploying it.

Audit Devices​

Sensitive programs like Vault and OpenBao should always have audit logs. Ideally it would be hard to not them set up without audit logs, or at least, make it very hard to do so.

However, the API model of Vault made this difficult originally:

1. Operators would call sys/init to create seal information and return a root token.
2. Operators would then have to call sys/audit to create an audit device.

In the meantime, they'd want to be configuring OpenBao: creating auth mounts, policies, secret engines, and the like. Perhaps this was fully automated, like in OpenTofu, and it would be hard to guarantee a strict ordering, due to inherent parallelism.

Motivating us to solve this was one of the few remote code execution vulnerabilities (RCEs) in Vault: audit device configuration frequently interacted with the broader environment, including potentially in ways that lead to RCEs. While HashiCorp opted just to patch one small hole, we opted to reimagine how audit device configuration could be controlled by configuration owners.

Thus declarative audit devices were born.

Here, system operators--those with privileged access to the underlying configuration and broader execution environment--can define audit devices through configuration files, reloading them on SIGHUP. These look like the following:

audit "file" "my-device" {
    description = "This audit device writes to stdout which never fails."
    options {
        file_path = "stdout"
    }
}

While writing to stdout doesn't matter as much, consider the implications when writing to network devices: operators can disable API-driven creation by setting unsafe_allow_api_audit_creation = false (the default) and ensure that a leaked admin token doesn't result in secrets being exfiltrated to an attacker over the network.

As an added bonus, every single request is now audited after initialization, including any declarative self-initialization requests that we'll see below.

Self-Initialization​

Going hand-in-hand with configuration-driven audit devices is declarative self-initialization.

Here, operators define requests that they want executed when OpenBao first starts up:

initialize "authentication" {
    request "mount-userpass" {
        path      = "sys/auth/userpass"
        operation = "create"
        data = {
            type        = "userpass"
            description = "Administrative access to OpenBao."
        }
    }

    request "create-user" {
        path      = "auth/userpass/users/admin"
        operation = "create"
        data = {
            password = {
                eval_source = "env"
                eval_type = "string"

                // Read the initial administrator password from an
                // environment variable.
                env_var = "INITIAL_ADMIN_PASSWORD"
                require_present = true
            }

            token_policies = ["admin"]
        }
    }

    request "create-policy" {
        operation = "create"
        path = "sys/policies/acl/admin"
        data = {
            policy = <<EOP

path "*" {
  capabilities = ["create", "update", "patch", "read", "delete", "list", "scan", "sudo"]
}

EOP

        }
    }
}

This would:

1. Create a new auth/userpass mount,
2. Create an administrative user, and
3. Create a highly privileged policy for that user.

While this may not look that different from an operator running commands manually, consider the implications for OpenBao as a building block of some larger system. This system would need:

• A source of identity, to tie OpenBao into;
• A KMS or similar auto-unseal device (including static) to automate unseal;
• A service orchestration layer (whether systemd unit files or Kubernetes),
• A system to manage day one OpenBao operations.

Given those exist, operators can provision the initial state fully declaratively.

OpenBao gives operators several tools for interacting in the form of dynamic data source. These can be environment variables:

password = {
    eval_source = "env"
    eval_type = "string"

    // Read the initial administrator password from an
    // environment variable.
    env_var = "INITIAL_ADMIN_PASSWORD"
    require_present = true
}

or files:

policy = {
    eval_source = "file"
    eval_type = "string"

    path = "/path/to/admin-policy.hcl"
}

OpenBao also supports chaining from the value of past requests or responses:

entity_id = {
    eval_source = "response"
    eval_type = "string"

    initialize_name = "identity"
    response_name = "entity"
    field_selector = ["data", "id"]
}

and can do CEL or text/template based templating:

path = {
    eval_source = "template"
    eval_type = "string"

    template = "auth/userpass/users/{{ .input.username }}"
}

This allows construction of fairly advanced configurations and interactions with nearly every subsystem regardless of complexity.

Think of self-initialization like the building block of a fully reproducible environment: operators define the minimal configuration necessary to stand up OpenBao and then systems like OpenTofu can take over from there.

Today we focus on transactional storage. While the earlier blog posts focused on the what and how of transactions in Raft, this post will focus on the measurable impact of transactions in OpenBao and their lack in Vault. We will demo some possible ways of creating snapshots which cannot restore and are not consistent on Vault and show how we used transactions to achieve consistency on OpenBao.

Historical context​

Recall from last week's post that our starting storage interface was rather limited:

// Storage is the way that logical backends are able read/write data.
type Storage interface {
    List(context.Context, prefix string) (entries []string, err error)
    Get(context.Context, path string) (entry *StorageEntry, err error)
    Put(context.Context, entry *StorageEntry) error
    Delete(context.Context, path string) error
}

(from fork-point:sdk/logical/storage.go).

Also included in the storage model of the fork, though implemented in only a few backends (Raft, CockroachDB, Consul, FoundationDB, and Spanner) was a basic batch application mechanism:

// TxnEntry is an operation that takes atomically as part of
// a transactional update. Only supported by Transactional backends.
type TxnEntry struct {
	Operation Operation
	Entry     *Entry
}

...

// Transactional is an optional interface for backends that
// support doing transactional updates of multiple keys. This is
// required for some features such as replication.
type Transactional interface {
	// The function to run a transaction
	Transaction(context.Context, []*TxnEntry) error
}

(from fork-point:sdk/physical/transactions.go).

Notably, from the implementation of physical.GenericTransactionHandler, we see that this was not an implementation of check-and-set semantics: LIST operations are entirely ignored, any GET operations are dispatched ahead of any writes, and while a rollback log is created and entries read before issuing any writes, they are not compared against any sent writes. This makes batch application rather unsafe if used incorrectly: if a (distributed) lock mechanism or other exclusive ownership semantic does not exist, multiple in-flight transactions can write to the same storage entries. This may produce unexpected results.

Luckily, this mechanism was not exposed at the logical level, hiding its use within Core and all auth and secret plugins, preventing its misuse.

From commit messages we can guess this mechanism was an internal implementation detail of the proprietary Vault Enterprise Performance Replication mode and thus not relevant to improving snapshot consistency.

OpenBao instead moved to a much more powerful interactive transaction model:

// Transactional is an optional interface for backends that support
// interactive (mixed code & statement) transactions in a similar
// style as Go's Database paradigm. This is equivalent to
// physical.Transactional, not the earlier, one-shot version of the
// interface.
type Transactional interface {
	// This function allows the creation of a new interactive transaction
	// handle, only supporting read operations. Attempts to perform write
	// operations (Put(...) or Delete(...)) will err.
	BeginReadOnlyTx(ctx context.Context) (txn Transaction, err error)

	// This function allows the creation of a new interactive transaction
	// handle, supporting read/write transactions. In some cases, the
	// underlying physical storage backend cannot handle parallel read/write
	// transactions.
	BeginTx(ctx context.Context) (txn Transaction, err error)
}

// Transaction is an interactive transactional interface: backend storage
// operations can be performed, and when finished, Commit or Rollback can
// be called. When a read-only transaction is created, write calls (Put(...)
// and Delete(...)) will err out.
type Transaction interface {
	Storage

	// Commit a transaction; this is equivalent to Rollback on a read-only
	// transaction. Either Commit or Rollback must be called to release
	// resources.
	Commit(ctx context.Context) error

	// Rollback a transaction, preventing any changes from being persisted.
	// Either Commit or Rollback must be called to release resources.
	Rollback(ctx context.Context) error
}

(from main:sdk/logical/storage_transactions.go).

Callers of this API can perform arbitrary storage operations interleaved with non-storage calls and have consistency amongst all of them. This is implemented in both supported storage backends, Raft and PostgreSQL.

Reproducers​

Looking at plugin code in our fork-point tag, any series of multi-write flow could potentially be affected by a snapshot consistency issue. However, to be affected by snapshot consistency issues, the server needs to assume that either both writes succeeded or neither did.

PKI​

One such is in the PKI engine.

When creating a new issuer via <mount>/root/generate/internal, the following storage operations are performed:

1. config/key/<id>, to store the new root CA's key
2. config/keys, to store the new default issuer config
3. config/issuer/<id>, to import the new root CA's certificate
4. config/issuers, to store the new default issuer config
5. crls/<id>, to store the initial empty CRL

Notably consistency between storage operations 3 and 4 are the crucial: failure to store the initial issuer's identifier as default will mean that API compatibility with older Vault versions (and many third-party applications) which are not aware of multi-issuer features is broken. This will cause the API to return an error like:

err=Error making API request.

URL: GET http://localhost:8200/v1/058a0c0f-2dc3-a4a3-2e22-b00811700bac/issuer/default
Code: 500. Errors:

* 1 error occurred:
        * no default issuer currently configured

resp=<nil>

which will persist on all operations until an operator manually creates an association between the generated issuer and the default. Similar issues could occur whenever an issuer is rotated; the key could be persisted but the signing certificate could be dropped from the backup.

Notably, PKI's usage here would be a poor fit for check-and-set semantics: composability of importKey and importIssuer into writeCaBundle would be difficult to achieve while retaining transactional properties. This is why the stronger interactive transaction model is better for OpenBao's use case, even though it limits the theoretical storage backends one could implement as not every potential storage engine (like S3) implements interactive transactions.

Along with broader discussions of how OpenBao and gittuf might integrate, we talked about Shamir's unsealing and its fundamental problem: it is a side-effecting process with high-entropy results. You can wrap it around a common dictionary, hex or base64 encoding, or other means to make the key shares more consumable by humans, but the results will still be complex and hard to input and store.

The problems I'm looking for a scheme to solve are two fold:

1. Secure storage of Shamir shares are hard. It effectively relies on another, externally secure system (whether physical or digital) to safely store. In emergency scenarios, this is fine, but if Shamir's is in regular use, this can be cumbersome depending on whether systems are air gapped or similar.
2. We can't easily use declarative self-initialization as we required a side-effect-less initialization.

On Shamir's​

While I will spare people too much math, some initial context on why this is non-trivial with Shamir's is important. For a conceptual model, think of Shamir's secret sharing as using polynomial equations:

s + a_1 x_1 + a_2 x_2^2 + a_3 x_3 ^ 3 + ...

By convention, the secret is embedded as the constant term of this polynomial, with constants (a_1 ... a_n) being randomly chosen values. To allow for a threshold of n+1 shares required to recover, a n degree polynomial must be chosen. For two share recovery, a linear equation is used; for three shares to recover, a quadratic, &c. This is because while n+1 (x, y) points uniquely describe a n-degree polynomial, n shares leaks no information: nearly any possible constant term could fall out.

This gives Shamir's secret sharing a useful property: information theoretic security.

Requirements​

Cappos proposed a model for Shamir's that worked by masking password hashes with the Shamir shares (via an xor); but in our context, we would mask Shamir shares with password hashes. Supposing such a scheme could exist, we'd care about the following properties:

1. It isn't much more computationally expensive. Shamir's is easy to verify: combining the pieces and performing a decryption of the root key using the recovered AES-GCM key is relatively cheap.
2. It doesn't have side effects: we can't require the owner to know which Shamir share a given password maps to.
3. It retains the core cryptographic properties of Shamir's when used in our barrier encryption scheme: thresholds, arbitrary share ordering, &c must all be retained.
4. Storage of shares must be safe in plaintext. Because this is the root of our cryptography, we don't have any other secure storage mechanism available yet.
5. Hinted at above, but passwords and shares must be independent. While the security of the scheme as a whole will be dictated by the combined entropy of the (threshold-determined) weakest passwords, we can put system limits on their construction. Passwords shouldn't derive into shares and thus have unexpected determinism: we'd want strictly independent root keys to allow for rotation under the same passwords.

Results​

Assuming such a scheme existed, this would give us a great way to extend our side-effect free declarative self-initialization to Shamir's seal mechanisms:

1. Operators would establish configuration like they do for auto-unseal devices and namespace sealing, specifying the number of generated shares and thresholds. This could perhaps be denoted via a new seal type, shamir-pass or similar.

2. Operators could optionally specify minimum password complexity rules using our existing password policies, which support validating against existing passwords.

3. We have two options for how initialization could go:

   • We could either immediately initialize, requiring operators to authenticate with any provisioned authentication provider and then subsequently only allow initialization requests (blocking broader requests to the system) to provide the required passwords until concluded.

   • Or, we could delay self-initialization until sufficient passwords have been provided, following the existing (unauthenticated) initialization or rotation patterns.

   Notably, neither of these two approaches would yield keys to the API callers which must be stored.

Within namespace sealing, we would already have an authenticated rotation mechanism and be able to reuse that for initialization.

From an unseal perspective, this also makes it easier for air-gapped operators wanting to use Shamir's to type and unseal OpenBao.

Nearly every single networked interface returning a list of results supports subsets. SQL supports the LIMIT and OFFSET keywords, along with a rich language for filtering returned results. Google Cloud KMS APIs supports pageSize, yielding a nextPageToken, for iterating over multiple pages of results.

Many resources in Vault and OpenBao return lists: KVv2 secrets, PKI's certificate lists, SSH's roles, and more.

Paginated lists were shipped in OpenBao v2.0.0 as our very first feature in our very first release!

So, why doesn't Vault support paginated lists?

The answer lies in its historical great-common-denominator storage interface:

// Storage is the way that logical backends are able read/write data.
type Storage interface {
    List(context.Context, prefix string) (entries []string, err error)
    Get(context.Context, path string) (entry *StorageEntry, err error)
    Put(context.Context, entry *StorageEntry) error
    Delete(context.Context, path string) error
}

(from fork-point:sdk/logical/storage.go).

In supporting over twenty different storage backends, Vault had to support a common core of simple APIs over all potential storage backends. This meant keeping Storage.List(...) to a simple get-all-results list operation. If LIST APIs were to limit results, this would mean they'd still have to fetch them all from storage and only limit them in memory before sending to the client.

Early in OpenBao's history, the decision was made to remove all but the Raft storage backend. This made it possible to iterate on our storage model. Concretely, paginated lists and transactional storage (to be discussed another time!) came out of that. As we re-introduced storage backends like PostgreSQL, we made sure to include the improvements we've made to the storage API from the start.

OpenBao's storage interface now looks like:

// Storage is the way that logical backends are able read/write data.
type Storage interface {
	List(context.Context, prefix string) (entries []string, err error)
	ListPage(context.Context, prefix string, after string, limit int) (entries []string, err error)
	Get(context.Context, path string) (entry *StorageEntry, err error)
	Put(context.Context, entry *StorageEntry) error
	Delete(context.Context, path string) error
}

(from v2.5.5:sdk/logical/storage.go)

meaning that API handlers can now fetch only a subset of results.

This follows a SQL-like interface:

func (b *RaftBackend) ListPage(ctx context.Context, prefix string, after string, limit int) ([]string, error) {
	// ... implementation elided ...
}

(from v2.5.5:physical/raft/raft.go)

taking the new parameters after and limit. These are:

• after: an optional entry to begin listing after for pagination; not required to exist in the list results.
• limit: an optional number of entries to return; defaults to all entries when set to a non-positive number.

We suggest plugin authors pass these through on all API endpoints to the underlying storage calls.

For plugin authors who build against OpenBao's SDK but wish to have compatibility with HashiCorp Vault, we've stubbed the implementation, meaning you can safely expose and use a paginated list API and support it on both server implementations.

In addition, use of paginated lists has lead to improvements like #678 by Fatima Patel, to use pagination during PKI's tidy operations. In the past, tidy operations could consume a lot of memory if a large number of PKI mounts contained a large number of stored leaf certificates. We exposed a new page_size option (defaulting to 1000) to limit the number of certificate serial numbers in memory during a single PKI tidy operation.

By enforcing this with pagination_limit in an ACL policy, operators can now ensure that clients use pagination and set maximum result set sizes going forward.

In short, OpenBao aligns with long-standing industry expectations for expensive list calls and operators have more control with OpenBao than with Vault for managing the performance impact of these types of API requests.

Tune in next time for a discussion on transactional storage!

For a video, see the Linux Foundation's official YouTube channel.

Welcome everyone to my talk on OpenBao and how we added horizontal scalability to the project. I'm Alex Scheel, Head of OpenBao Development at ControlPlane, a long time member of the OpenBao TSC, and chair of the OpenBao Development Working Group.

I've been fortunate to have a hand in the development of OpenBao since nearly the beginning of the project, and before that, at HashiCorp's Vault CryptoSec team.

If, like me, you were wishing you could get out for a post-lunch walk, thank you for staying, but we'll have to settle for some photos of Minneapolis I've sprinkled through the presentation. And thank you all for visiting Minnesota, whether from near or far!

First, what is OpenBao?

OpenBao is an open-source secrets manager, featuring everything from static and dynamic secrets to PKI and key management services. We have integrations with everything from External Secrets Operator and Cert Manager to OpenTofu, SOPS, and cosign.

We also have a growing ecosystem; check out our ecosystem page afterwards and if you're an adopter, integrator, or supporter, consider adding your logo!

OpenBao is an OpenSSF Sandbox project and is licensed under the MPLv2.

If this sounds familiar, it should be!

OpenBao is the open-source continuation of HashiCorp Vault, started in late 2023 after the relicensing to the non-OSI BUSL license.

We aim to keep API compatibility for client applications, but to improve the operator and developer experience. Towards that goal, we've landed several Vault Enterprise features--such as horizontal scalability, which this talk focuses on--but also many original improvements like declarative self-initialization, storage-level improvements for better performance and snapshot consistency, and landing in our next release, things like an externally pluggable KMS interface, operator-defined workflows built on the existing profile engine, and per-namespace (per-tenant) barrier encryption keys with optional namespace sealing support.

(This photo is taken from the Mill Ruins park, facing the Mill City Museum which is hosting tonight's drone show.)

OpenBao's top-level governance body, its technical steering committee (TSC), is currently made up of the following companies: my employer, ControlPlane, a seat jointly occupied by SAP and Liquid Reply who SAP has contracted with to contribute to our community, Adfinis, Wallix, IOTech, and GitLab.

Our governance is openly documented in the project’s main repository. We have tracks for leadership (in the form of the TSC), for developers (in the form of the Dev Working Group and its many sub-working groups), and maintainership for those interested in contributing to reviews on the project. We’ve also started a marketing working group if talking about OpenBao or its usage is more your style.

In my early 2025 talk at FOSDEM (which you can see on the OpenBao blog archive), I mentioned that I had hoped to grow a community around OpenBao of companies that make money supporting it, just like Kubernetes. Not long after that talk, both Adfinis and SAP stepped up and started contributing to the community. This March, I joined ControlPlane to further that mission as well, leading our efforts to commercialize support and maintenance of the project. You can read more about that on ControlPlane's blog.

ControlPlane employs maintainers for FluxCD, is a contributor to several CNCF and OpenSSF efforts, and focuses on a highly regulated customer base like banks and government entities.

I'm happy to report for anyone following along, the OpenBao community is healthy, growing, and my hopes have been answered! Though of course, we always welcome more contributions of any sort!

This now brings us to what we're here to discuss. Horizontal Scalability, and the journey to support it.

I'll focus on three parts mostly:

1. What building blocks did the fork have? What were we starting with?

2. How did we support Raft in OpenBao v2.5.0? What challenges did we face along the way?

3. How will we support PostgreSQL and more going forward? How will this improve the operator experience?

We started by trying to understand what we inherited from HashiCorp Vault.

Most importantly, we started with a highly available (though, not horizontally scalable!) base. In OpenBao as it stood before this feature, we had data replication already taken care of. We supported two HA storage backends (Integrated Storage aka Raft and PostgreSQL),

OpenBao and HashiCorp Vault have a single active node which can perform write operations; this is a limitation of the Raft protocol and used as a design decision elsewhere.

In Vault Community Edition which we inherited, probably for open-core feature differentiation with Vault Enterprise, HashiCorp only supported cold standbys: these were "unsealed" and ready to take over if the active node went down, but could only forward requests to it and did not do any processing locally.

In OpenBao v2.5.0, we wanted to land horizontal scalability, which meant we needed a way for these cold standby nodes to serve read requests. They still wouldn't handle write requests, but they could help alleviate some of the load from the single active node, depending on the user's workloads.

And, of course, we had a constraint that we wanted to behave similarly to Vault Enterprise's Performance Standby node types.

How does Vault Enterprise behave? And what is a read request?

Vault Enterprise roughly defines a read request as something that doesn't cause any storage writes.

Confusingly, this has a very loose association with HTTP verbs. For instance, the Certificate Revocation List (CRL) rotation endpoint in the PKI engine incurs a storage write (the CRL itself), but uses the GET verb. More obviously, login, e.g., via the userpass authentication method, causes several storage writes.

Both of these, if they come into a read-enabled standby, would be forwarded.

For the types of requests that will be handled by a standby node, a KVv1 secret read operation would be handled locally; it is guaranteed to not have any write operations. Most KVv2 operations will not have any writes either. Additionally, some POST operations will be strictly handled locally: if a PKI engine's role is configured not to store leaf certificates, the issuance endpoint could be handled on the read-enabled standby node.

From an operational standpoint it is now clear what we want to achieve, though we have a few quirks we need to work out with our storage backends. How do we handle invalidation of caching that occurs naturally?

For integrated storage, each OpenBao node becomes its own storage backend. It uses the Raft consensus protocol to handle replication of data between all nodes. This protocol is based on a leadership election process, which requires an odd number of voting nodes to be consistent. Each write sent by the leader is confirmed by another vote.

This write confirmation forms the basis of the write-ahead-log (WAL) mechanism. This WAL mechanism contains enough information about every storage write, which gives us an easy way to invalidate cached storage entries.

On PostgreSQL however, any number of nodes can be used as they all race to acquire a lock stored in the database. While PostgreSQL handles the data replication for us--giving us a M:N OpenBao service to database node decoupling--it doesn't by default provide enough information for invalidation on standby nodes. But more on that later.

Looking at the existing code base, there's a few hints we have on how invalidation is supposed to work.

This is a snippet of code from the PKI secrets engine. There are two parts I bolded:

1. When we invalidate, we're given a context and the path of the storage entry that changed. We don't get the contents of the entry or if it was updated or deleted.
2. Similarly, we see that we shouldn't block too long: this code path spins off a goroutine to handle the update in the background. Other code paths in this function just flip the value of an atomic boolean for the next API request to adjust.

Putting this together, we learned that cache invalidation must be definite, operating on specific keys which were invalidated; it isn't TTL based. There must be some storage-path-based routing engine that dispatches invalidations from the storage layer, through core, to the plugin.

And, because invalidation must be in some hot path (in the case of Raft, we know it must be part of the WAL application process), we've learned that handling the invalidation synchronously isn't good for performance, so we should execute it asynchronously.

(This photo is features the Third Avenue bridge, looking back on downtown Minneapolis.)

Moving right along, it was time to implement this.

The design idea roughly looked something like this:

1. We started by assuming we could just load all core subsystems and plugins on the standby node.
2. We allowed requests to come in, rather than automatically forwarding them. After this, we built a mechanism to conditionally forward requests when a storage write occurred.
3. And we built a cache invalidation layer based on the earlier analysis of our existing code, and a mechanism for routing storage paths to the owning subsystem.

Simple, right?

Starting from the existing HA code, it took a lot of work to refactor core to (correctly!) bring up the standby node as a reader. While conceptually simple, we found that we had some partial state (from being non-read-enabled) that we need to make sure we reset. And we found we needed a new post-unseal strategy which did not incur any writes and didn't load any subsystems which only performed writes, such as automatic barrier key rotation or starting the request forwarding server (which standbys connect to).

When processing read requests, we changed the request routing to now only conditionally forward the request (if the node was still a cold standby), and default to processing requests. This necessitated a second forward pass if the request could not be handled locally. Notably, a failed write request would result in no state, as the standby nodes cannot write to their underlying storage.

In creating this, we needed to refactor how requests were processed at the http layer: previously we only read the body once, either to forward it or to process it on the active node. However, as we now consumed the body twice, we built a way of spooling the request body and resetting back to the initial state on forward.

The contract between OpenBao's core and its storage engines is conceptually simple: the storage engine just needs to tell us when a write occurs, so we can define a callback approach for this in the base sdk/physical interface.

This was implemented in the Integrated Storage (Raft) backend.

However, the actual dispatch of invalidations is more complex.

We take pending callbacks from the storage engine and queue them into an asynchronous fairshare job queue, sharded by expected namespace. This prevents one particularly noisy tenant from starving all other tenants, at the cost of not necessarily guaranteeing relative order of writes across namespaces.

This job then handles the storage-level routing of written path to owning backend. Some backends are privileged, like the core of OpenBao itself, and can take longer to process invalidations. For instance, the mount table explicitly reloads the new entry, potentially spinning off a new external plugin process so that it is ready before the next inbound request. Requests routed to an sandboxed plugin, e.g., PKI, get processed with a short, 2 second context to enforce strict asynchronous behaviors and use of their own background worker contexts.

The result of all this is that standby nodes now process read requests. In the chart on the left, before adding read request handling, only a single service process was consuming CPU. Now, on the right, all three service processes share this developer's machine and consume CPU processing requests.

When it comes to running in a real, multi-node deployment, the impact becomes more clear. Where there was more contention for certain locks, the OpenBao v2.4.4 cluster's single active node was able to handle far fewer PKI certificate issuance operations. The other two standbys were completely idle.

In OpenBao v2.5.1, all three nodes were able to service requests, decreasing lock contention, giving us higher throughput and lower latency.

You can read more about this in Philipp's entry on the OpenBao blog.

Of course, things don't always go as planned. For some example of bugs we've fixed:

1. When loading current state, we didn't pass through standby vs active node status to all places that needed it. This meant that, when upgrading a legacy non-transactional storage type to a read-enabled standby (say, in a direct migration from OpenBao v2.0.0 to v2.5.0), we'd fail to start up.

2. Certain types of specially handled requests (like root token generation) were not correctly forwarded to the active node as they were outside the regular request routing mechanism. This will be fixed in 2.5.4 coming this week.

3. But most of our bugs have been in the largest section of new code: invalidation routing. While plugin's invalidation logic was already present in Vault Community Edition, most of the core invalidation logic was presumably hidden in Vault Enterprise and wasn't part of our code base. This is the most difficult to test as it requires understanding and validating each individual subsystem.

Mostly these come from community bug reports -- and some patches have even come from drive-by contributors, which is always appreciated.

The release series containing horizontal read scalability, OpenBao 2.5.x, was shipped at the start of the year, and so may already be running on your systems!

Before we move on, I'd like to thank again the contributors to this feature: Fatima was the 2025 OpenBao Mentee sponsored by Adfinis, Philipp, an Adfinis employee, was her mentor and also contributed to the design and implementation, and I helped with implementation, reviews, and debugging.

(This photo was taken by my wife, Katherine Mayo, one winter, looking from the iconic Stone Arch bridge back on the city.)

You might be thinking... wait! ... you mentioned that the invalidation hook was implemented in the Raft backend, but what happened to the PostgreSQL backend?

Well, it turns out it is slightly more complicated to implement.

We started by thinking that maybe the built-in LISTEN+NOTIFY support in PostgreSQL would allow a sort of storage-level inter-process communication (IPC) which would save us from implementing it at the node-level.

Among other challenges, it turns out that these events may be silently dropped if OpenBao disconnects and reconnects through a proxy such as pgbouncer. This makes it unreliable for our use in invalidations.

PostgreSQL has a WAL as well; why can't we use it? For starters, database operators would need to configure PostgreSQL to have a higher WAL verbosity level (wal_level=logical). Additionally, OpenBao cannot subscribe to the WAL on read-replica PostgreSQL nodes; only the PostgreSQL primary sends WAL events. This means having two database connections rather than just one on standby nodes.

We thought about adding either a last modification time to the existing OpenBao schema, which loses the ability to easily detect deletes, or adding a new WAL subscription table. The latter puts garbage collection pressure on the database and has some of the same peer longevity problems as the PostgreSQL WAL implementation. We'd need to manually remove processed invalidations and keep track of which invalidations were seen by which live standby nodes.

What about moving to a more standard TTL-based expiration? The biggest issues here are that OpenBao relies on invalidating lists--that is, knowing when a new entry was created--and that invalidating certain entries, such as the mount table, is rather expensive as it holds a few global locks that block incoming requests. This means more refactoring would be needed before we could adopt this approach.

Ultimately, we settled on implementing a GRPC stream-based notification mechanism, using our existing GRPC connection for forwarding requests. The standby will subscribe to the leader for invalidations, wait for the initial index to ensure it is caught up, and then begin processing both read requests and invalidations from the active. The active will send the expected index at which the data was written, which the secondary can then wait for.

Right now, OpenBao will have to manually wait for this event, but PostgreSQL 19 will introduce a WAIT FOR LSN operation.

The benefits of this is that OpenBao will strictly follow PostgreSQL’s lead: operators will promote or demote PostgreSQL nodes to primary, allowing writes to occur, and whichever OpenBao nodes are connected to it will then self-select to become active. No OpenBao specific failover is necessary.

The good news is that this approach will hopefully scale to any indexed replicated storage backend, and open up new opportunities in the future.

This is under active development and should be in OpenBao v2.7.0.

(This photo showcases Target Field, where the Minnesota Twins play -- I hear the Linux Foundation has been organizing some visits to watch them play this week.)

Briefly before we wrap up, a few more things to mention:

1. We're introducing stronger consistency semantics for clients through index headers, aligning with and extending Vault Enterprise's implementation here. This should hopefully help clients to see fewer stale reads.
2. We also have a roadmap towards write scalability through per-namespace storage backends.

In Vault Enterprise, this looks like the performance secondary cluster architecture, allowing a very limited form of write scalability: mounts (regardless of namespace) can be mounted as cluster-local, essentially creating a forked/hybrid cluster. Plugin authors can also opt-in to write certain paths to cluster-local storage, such as the PKI plugin’s leaf certificates. This has caused problems for plugins whose clients expect global knowledge: PKI’s CRL functionality needed special cross-cluster synchronization support.

OpenBao is thinking about this in a different way: by using namespaces as the isolating boundary, we know very few storage operations will cross the namespace boundary. This will form the basis for our envisioned multi-writer support: first we’ll add lightweight storage separation (different tables at the BBolt and PostgreSQL levels) using the same storage backend, allowing things like namespace-level storage quotas. Then we’ll introduce out-of-hierarchy namespaces, defined in the configuration file. These won’t inherit policies and access from the root namespace, behaving more like a lightweight, fully-isolated core. Lastly, we’ll move to separate storage backends altogether, adding in the ability for namespaces to designate themselves as active (versus a single global active node). This will allow write scalability across namespaces.

Ultimately, we hope this will lead to a different set of operator tradeoffs: the simplicity of a flat cluster (versus a treed hierarchy) should be easier for operators to understand and we’ll have better write scalability across an entire cluster rather than on a more adhoc basis.

Lastly, if you're interested in getting involved with the project, check out our community calendar on Proton! You can contribute to a development direction item, join a working group, like our posts on social media, add your logo to the OpenBao Ecosystem page, or just start using OpenBao for your secrets management needs.

Thank you very much!

If you have any questions later or just want OpenBao stickers, feel free to come and find me at the conference. I may be at the OpenSSF booth, but if not, they--or you--should be able to contact me on their Slack instance. John from ControlPlane and a newly announced OpenSSF Ambassador is also wandering around if I'm busy!

Does anyone have questions now?

In this blog post, I will give you an overview of the new Horizontal Scalability feature of OpenBao, its (current) limitations and planned future developments. In the second part, I will show some benchmarks to see in which cases the new feature helps (spoiler: it works best in read-heavy workloads, but doesn't improve write-heavy workloads).

Recap on Scalability Terminology​

Let's quickly recap what "horizontal" means in the context of scalability: To scale a service up (i.e., allow it to handle more load) there are two options: Give each instance more resources (CPU, memory, network speed, disk capacity, ...) or increase the number of instances. Increasing instance size is referred to as "vertical" and increasing the number of instances is referred to as "horizontal" scaling.

Whether this is successful, depends on the problem at hand and the implementation. There are some problems that scale very well - both horizontally and vertically - like serving static files. Given that the server is implemented reasonably well, either doubling the resources per instance or doubling the number of instances should both double the number of static files you can serve. Once you start to introduce mutability scalability starts to suffer, especially if you want to provide strong consistency guarantees.

Recently released Horizontal Scalability Features​

With OpenBao v2.5.0, we introduced what we call "read scalability". To ensure High Availability, OpenBao has supported "standby nodes" since we forked from Vault. One of these nodes would automatically take over active duty, if the current leader nodes goes down (planned or unplanned). We extended this feature to allow the "standby nodes" to become "read-only nodes", meaning they can now serve read requests on their own, while write requests are still forwarded to the leader.

This has some limits:

• The obvious being: If you have mostly write requests with only a few read-only requests, then read scalability won't help.

  info

  Keep in mind that a read request on the API level (e.g., HTTP GET) might still require a write to the storage, e.g. reading a dynamic secret from the database plugin will require a write to storage, because after its TTL has expired, it needs to be deleted and for this OpenBao has to do "bookkeeping".

• Currently, only the Raft storage backend is supported. Support for PostgreSQL is in the works.

There are also some caveats:

• It might require changes to your load-balancer setup: If no traffic is routed to the "read-only nodes", then they won't remove any load from the primary.

• Data read from a standby node might be stale, if it has recently been updated on the primary. If this is unacceptable to you in general, you can set disable_standby_reads = true in your configuration to disable reads from standby. Or, if this is acceptable most of the time, you can ensure to read from the primary in cases where you need the latest data.

Horizontal Scalability Outlook​

Consistent writing in a distributed fashion is an inherently hard problem. If we were set out to solve this in a general purpose way, we'd be doomed to fail. We need to look out for properties of our specific case, that we can exploit: Currently, there is one leader node per cluster, which will handle all write request. We are planning to change this and have leader nodes per namespace, which will handle all write requests for their namespace. As write requests between sibling namespaces are independent, there is no risk of violating data integrity (a property we can exploit 🎉).

This again has an obvious limitation: if you do not use namespaces or if most of your traffic is within a single namespace (e.g. you have a test and prod namespace and prod accounts for 99% of the load) this won't help. But let's take one step at a time and maybe it is even good enough, because a use cases is either small enough to avoid scalability problems or big enough to require namespaces anyway.

Now it is time to look at some benchmarks.

Benchmarks​

Setup​

I have set up two 3-node OpenBao clusters on Azure, one running OpenBao 2.4.4 and one running OpenBao 2.5.1. Each cluster has a dedicated load balancer, but the load balancer for 2.4.4 is configured to direct traffic only to the primary node and the load balancer for the 2.5.1 cluster will distribute the traffic among all nodes.

I ran two benchmarks, one for the KV engine and one for the PKI engine. I ran both of them for 5 minutes per cluster using the benchmark-openbao tool.

KV Engine Benchmark​

The benchmark aims for 500 requests per second, with 90% reads and 10% writes.

duration = "5m"
cleanup  = true
workers = 2000
rps = 500

disable_keep_alive = true

test "kvv1_write" "writes" {
  weight = 10
  config {
    numkvs = 10
    kvsize = 100
  }
}

test "kvv1_read" "reads" {
  weight = 90
  config {
    numkvs = 10
    kvsize = 100
  }
}

If we take a look at the per-node CPU usage, we can clearly see the effects. The first plot shows the CPU usage of the 2.4.4 cluster over a 15 minute timespan (average per minute) with the 5 minute benchmark roughly centered. Before and after the benchmark, we can see the active node hovering around 4% CPU usage, while the standbys use around 2%. During the benchmark it raises to around 60% for the primary and 8% for the standbys.

The next plot shows the same data for the 2.5.1 cluster. In the first plot we could clearly see which line is the primary, but here all lines are very similar. Before and after the benchmark the CPU usage hovers around 4% jumping up to 30 to 35% for all nodes during the benchmark.

This results in better latencies:

PKI Engine Benchmark​

The benchmarks aims for only 5 requests per second, generating RSA 2048 certificates with no_store = true, which makes this a read-only operation. Generating a (RSA) private key is quite heavy on the CPU, therefore the low number of requests.

duration = "5m"
cleanup  = true
workers = 15
rps = 5

disable_keep_alive = true

test "pki_issue" "pki_issue" {
    weight = 100
    config {
        setup_delay="2s"
        root_ca {
            common_name = "benchmark.test Root Authority"
            key_type = "rsa"
            key_bits = "2048"
        }
        intermediate_csr {
            common_name = "benchmark.test Intermediate Authority"
            key_type = "rsa"
            key_bits = "2048"
        }
        role {
            ttl = "10s"
            no_store = true
            generate_lease = false
            key_type = "rsa"
            key_bits = "2048"
        }
    }
}

Here again we have two graphs, first from the 2.4.4 cluster followed by the 2.5.1 cluster. During idle the results are the same as before.

On 2.4.4 cluster the primary spikes to 100% CPU usage, while the standbys don't see any change (because in contrast to the KV benchmark there are no writes happening, so they have no raft updates to apply).

On the 2.5.1 cluster the load is spread pretty well at roughly 40%.

This time we also see better latencies, but additionally the 2.4.4 cluster failed to even fulfill the 5 requests per second target and some requests (0.73%) even failed.

Missing Benchmarks​

I decided against showing a write heavy benchmark, like PKI with no_store = false or KV with 100% write. Not because I want to hide the fact that our read scalability feature does not help here, but because it would be boring to look at. The results for both versions would be similar, the only change you could see is that for 2.5.1 the standby nodes use a little more CPU while the cluster is idle (as we have seen before).

1. Many technical initiatives have landed, from namespaces to transactional storage, to CEL support, and many smaller things in between.

2. Many continuing working groups and large technical initiates have been started, from the horizontal scalability WG, focused on read scalability; the UI WG, focused on a rewrite of our EmberJS Web UI into React; to the PKCS#11/KMS WG, focused on including external keys into OpenBao. Thank you to everyone who participates in these!

3. Many, many contributions from many, many contributors! We welcomed:

   • 741 commits to main,
   • 287 contributors in the past year,
   • 10 new committers, and
   • 4 new moderators!

   And like I say nearly every meeting, a special thanks to all net-new contributors! A fresh set of eyes brings wonders to a storied project, and often revisiting earlier design choices let us improve the experience.

And yes, not everything on the 2024-2025 roadmap was completed. Don't worry, you can still contribute items from it if you want! But for a community just starting out, with its first formal direction proposal, I think the response we got exceeded my wildest expectations.

Much like last year, this roadmap opens with three major categories:

1. "Operator Experience": to enable easier or safer operation of OpenBao, through changes like profiles, break-glass and backup/restore procedures, and improved monitoring capabilities;
2. "Scalability": to improve optimization and utilization of OpenBao in large, complex environments; and
3. "Sustainability": to ensure the long-term viability of the code base, react to changing secrets management directives, and stabilize our ability to maintain the project indefinitely.

Together, these spell OSS, reaffirming our commitment to an open-source project lead under open governance principals.

Like last time, our roadmap features several items I'd especially like to highlight:

1. Expanding CEL Support for non-ACL policies, in the profile system, and other plugins. This allows greater operator control over authentication and authorization. Operator Experience.
2. Lazy loading of mounts, namespaces, removing them from memory when no requests have accessed the path in a while. This would allow OpenBao to be substantially over-committed if workload allows. Scalability.
3. KMIP Server for Transit and PKCS#11 client for Transit, to allow consumption and safe usage of Transit-stored keys from other servers via KMIP and PKCS#11 protocols. Sustainability.
4. Usage guides and tutorials for various components. Sustainability.
5. Versioned documentation so that pre-release features do not show in the documentation by default and documentation for particular releases are maintained accordingly. Sustainability.

Spoiler alert: it worked. Spectacularly. But not because we had all the answers on day one. It worked because we treated the mentorship itself like an open-source project: we iterated, adapted, and optimized for success in real-time.

This is the story of how we structured—and restructured—a 12-week program that ended up giving OpenBao a scalability boost and fantastic contributions.

The Setup: An Experiment with High Stakes​

The goal was ambitious: transform OpenBao’s passive standby nodes into active cluster participants that can handle read requests, a key step towards horizontal scalability. Adfinis provided the crucial fuel by funding the program, and the initiative was kicked off by Alexander Scheel, who created the project on the LFX Mentorship Platform.

My role was to be the primary mentor. This was a shared investment: Adfinis invested funds and my time, while Fatima invested her talent and immense effort.

We started with a hypothesis: intensive, daily collaboration would be the fastest way to onboard her into the complex codebase.

The Rhythm of Mentorship: Finding Our Flow​

For the first few weeks, we met every single day for an hour. This intensive pace was essential for building context and momentum. But we quickly learned that the magic wasn't just in the meetings—it was in the deep, uninterrupted work happening in between.

So, we adapted. Around the halfway mark, we shifted our rhythm to three meetings per week. This wasn't a step back; it was a strategic pivot. The focus moved away from daily "what do we do next?" check-ins to structured sessions where we could define clear tasks for the upcoming week. This gave Fatima the autonomy for deep, focused work and essential self-teaching, while ensuring she always had support to overcome obstacles.

And the support team grew! In the second part of the program, Fatima had the fantastic opportunity to work closely with Philipp Stehle from Adfinis, who was tackling cache invalidation challenges between nodes. This turned the mentorship into a true team collaboration. Instead of one mentor, she effectively had two, which was an incredible advantage and made the experience much more dynamic.

The Mentee’s Lens: Lessons Beyond the Code​

From my side of the screen, this mentorship was more than a 12-week coding sprint — it taught me the importance of confidence, perseverance and how collaboration and adapting strategies as new problems arise are essential parts of working effectively in a team.

When we initially discussed horizontal scalability as the topic for the mentorship, I remember feeling underprepared, having never worked in that space before. My mentor, Andrii’s uplifting encouragement and guidance helped me dive in, understand the problem space, and start experimenting with solutions. Once Philipp joined the project, the pace picked up and the discussions became even more engaging.

The community also held bi-monthly meetings led by Alex to review progress on high availability and discuss next steps. These sessions ensured the work stayed aligned with the community’s goals and that our progress moved in the right direction.

Looking back, the experience shaped not just how I write code, but how I approach challenges — with curiosity, collaboration, and confidence!

The 12-Week Sprint: A Choice We'd Make Again​

The LFX platform offers 12-week full-time or 24-week part-time programs. We’re incredibly glad Fatima chose the 12-week option. It forced a healthy intensity—a focused sprint that kept everyone laser-focused on the goal: delivering a tangible, impactful feature for the community.

And what a feature it was. Under this adapted structure, Fatima’s contributions were profound:

• She enabled standby nodes to handle read requests, a fundamental shift from idle replicas to active participants.
• She implemented the post-unsealing logic, ensuring standbys are fully initialized and ready to work.

The result? A foundational improvement that makes OpenBao clusters more efficient and performant. Standby nodes are no longer resource-consuming fallback insurance; they are now part of the active workforce.

The Real Win: A Model for Community-Led Growth​

This program was a win on every level.

• For OpenBao: We gained a critical feature from our roadmap and a valuable contributor.
• For Fatima: She tackled a complex project with direct, high-impact outcomes and the support of a collaborative team.
• For Adfinis: The investment in funding and our time directly contributed to the health of an open-source project we believe in.

The real success wasn't just the code. It was proving that with the right combination of funding, flexibility, and a team spirit, you can structure a mentorship to achieve incredible things. It’s about creating a focused environment where a talented developer, like Fatima Patel, can deeply integrate with a maintainer's workflow to accelerate complex, high-impact projects that benefit the entire community.

This is how we build a sustainable open-source future: not by waiting for perfect plans, but by creating opportunities, adapting as we go, and investing in people.

Want to learn more about the technical outcomes? Check out the pull requests and join the conversation on our OpenBao GitHub repository.

Inspired to get involved? Explore the LFX Mentorship Platform and see how you can contribute.

One theme was clear: Companies and individuals demand tools to manage secrets across their systems, and solutions like OpenBao are foundational for security and compliance.

Together with the Team from RISC-V and DeepComputing we were able to successfully test our riscv64 binaries on a DeepComputing DC-ROMA RISC-V laptop. With our added RISC-V support in PR #168 this was super easy and just worked out of the box.

Throughout the conference, we shared how OpenBao is able to help organizations to run production workloads securely. We showcased existing features such as strong encryption, fine-grained access controls and namespace isolation. We also gave a glimpse into what’s next: declarative initialization, improvements around namespaces and extended key management capabilities.

Furthermore, the conference was also a wonderful opportunity for OpenBao maintainers to meet in person, exchange ideas, and align on next steps. A big thank you goes to Adfinis, Paymenttools and Reply, who supported the project by sending maintainers to represent OpenBao on-site.

And finally, a heartfelt thanks to everyone who stopped by, joined discussions, or offered feedback. Your energy and curiosity keep the project moving forward. To everyone who was not able to join the conference, feel free to learn more at openbao.org or join the conversation on GitHub.

Open source builders, security engineers, and platform teams are converging on Amsterdam and OpenBao will be in the middle of it, thanks to the generosity of the Linux Foundation.

We’re excited to announce our first official booth at Open Source Summit Europe 2025. If you care about secure automation, multi‑tenant architectures, or meeting the stewards of your critical infrastructure tooling, come talk with us!

Official event site: Open Source Summit Europe

Find Us
Event: Open Source Summit Europe 2025
Dates: 25–27 August 2025
Location: RAI Amsterdam, Netherlands
Booth: G/S7
Hashtag: #OSSummit #OpenBao

This isn’t just a swag stop (expect some stickers); it’s a place to trade real-world stories, get practical guidance, and influence the OpenBao roadmap, now ready for its second iteration!

What You’ll Get at the Booth​

Guided Walkthroughs – We’ll walk you through the latest capabilities: Namespaces, advanced policy handling, paginated lists, and more... Ask deep architectural questions (scaling, HSM integrations, and migration strategies) and get answers from the people writing the code.

Deployment & Design Q&A – Are you architecting resilient deployments, tightening policy boundaries, or integrating with existing PKI or signing workflows? Bring real problems, leave with practical next steps.

Roadmap Preview – Near‑term energy: safer ops (policy and audit hygiene), resilience (parallel and namespace unseal), practical scalability (standby node read request handling). Bigger ecosystem/registry ideas come later. Tell us what you need most!

Community Meetups – Connect with maintainers, contributors, downstream integrators, and fellow operators. Hallway conversations around pain points often become features.

Contribution Guidance – Want to help? We’ll highlight good first issues, active design discussions, and how to start, whether you're an experienced developer new to Open Source, someone looking to get into technical writing, or anything in between!

Why It Matters​

OpenBao represents a community‑first, transparently governed evolution in secrets management. No lock‑in. No unclear licensing pivots. Just open, auditable, extensible security primitives stewarded in the open. Our presence at OSS Europe marks a milestone in maturity—and a recommitment to collaborative innovation.

Who Should Stop By?​

• OSPS leadership looking to understand viable alternatives to their company's vendor lock-in problems
• Platform & DevSecOps teams standardizing secret workflows
• Security engineers evaluating trust, provenance, and lifecycle controls
• Architects designing regulated or multi‑tenant environments
• Contributors (or future ones) curious where to jump in
• Anyone comparing OpenBao with legacy or closed models

Key Themes We’ll Be Talking About​

• Secure multi‑tenancy with Namespaces
• Operational trust: auditing, rotation, lifecycle management
• Plugin & ecosystem growth
• Secrets management at scale
• Community governance & sustainability

Engage Before / During / After​

• Plan Your Visit: Add our booth to your event schedule (check the published schedule once live).
• Register: Early reg & travel info via the event site’s registration page.
• Follow Along: openbao.org & our LinkedIn page (OpenBao on LinkedIn) for micro‑updates and drop‑in session times.
• Ask Early: Have a thorny problem? Post it in advance in community channels so we can prep a live walkthrough.
• Remote? We’ll share recaps, photos, and key takeaways using #OSSummit and #OpenBao.

Call to Action​

Show up. Challenge us. Tell us what you need from the next generation of open‑source secrets management, and help build and shape it. OpenBao moves fastest when the community is loud.

See you in Amsterdam!
— The OpenBao Team

We’re excited to welcome OpenBao to the Open Source Security Foundation (OpenSSF) as a newly accepted sandbox project!

OpenBao is an open source identity-based secrets and encryption management system that helps organizations securely store, manage, and audit access to sensitive data like API keys, passwords, and certificates. Originally developed under LF Edge and forked from HashiCorp Vault, OpenBao has now found its new home in the OpenSSF, where it aligns more directly with the needs of security professionals and open source maintainers.

“Joining OpenSSF has been a dream come true,” said Alex Scheel, Chair of the OpenBao Technical Steering Committee. “This wouldn’t be possible without the support of OpenSSF’s Technical Advisory Council—many thanks for their help and consideration!”

Why OpenBao?​

Modern systems rely on a growing number of secrets to operate securely—from database credentials to cloud service API tokens. Managing and auditing access to these secrets is complex, and building custom solutions can introduce more risk than reward. OpenBao solves this problem by providing robust encryption services that are tightly gated by authentication and authorization controls. It integrates with the tools and ecosystems open source developers and security teams already use, including Sigstore and SLSA.

Through its CLI, UI, and HTTP API, OpenBao makes it easier to:

• Securely manage and distribute secrets across teams and environments
• Apply fine-grained access controls
• Rotate keys and credentials automatically
• Maintain detailed audit trails for compliance and incident response

Moving to the OpenSSF​

The move from LF Edge to OpenSSF reflects a strategic decision by OpenBao’s Technical Steering Committee to better align with its contributor base and project mission. While the project is deeply grateful for the support from LF Edge during its early development, the OpenSSF offers a broader and more targeted platform to grow its community and expand adoption.

“We’re looking forward to collaborating with various working groups and SIGs within OpenSSF—especially to help strengthen secrets management best practices in open source policy and guidance,” said Scheel. “We also remain committed to maintaining partnerships with LF Edge and other Linux Foundation projects as our ecosystems evolve.”

What’s Next​

Now in the sandbox stage of OpenSSF’s project lifecycle, OpenBao is focused on increasing transparency, improving governance, and deepening integration with other OpenSSF initiatives. Its ongoing work will support supply chain security, secrets hygiene, and broader security-by-design efforts across open source projects.

We’re thrilled to support the OpenBao community and look forward to seeing its impact grow across the open source security landscape.

🔐 Get Involved: Visit the OpenBao GitHub repository and join the conversation on the OpenSSF Slack and Zulip.

This post was originally authored by the OpenSSF marketing team and posted on the OpenSSF blog.

What Are Namespaces?​

Namespaces in OpenBao are logical partitions within a single OpenBao instance, functioning as isolated environments where teams, organizations, or applications can operate independently.

Each namespace acts like a mini-OpenBao, with its own policies, authentication methods, secret engines, tokens, and identity groups. This architecture enables organizations to implement a true OpenBao-as-a-Service model, empowering internal customers to self-manage their environments securely and efficiently.

Why Namespaces?​

Strong isolation between teams, business units, or tenants becomes critical as organizations scale, especially when handling sensitive data.

Namespaces enable secure multi-tenancy. Each tenant (e.g., team, organisation, or application) operates within its namespace, isolated from others. Permissions are strictly scoped, preventing users from accessing or interfering with resources outside their assigned namespace.

Furthermore, namespaces enable the delegation of administration and promote self-service. Namespace admins can manage their own policies, secret engines, auth modes, or even quotas, without impacting other tenants, reducing the burden on cluster-level operators.

Finally, namespaces are one of a few planned stepping stones towards OpenBao's horizontal scalability journey. OpenBao aims to allow support for large deployments with many infrequently accessed mounts, without overloading cluster nodes, while keeping a simpler cluster topology.

How to use Namespaces​

Namespaces will enable plenty of use cases and multi-tenancy scenarios. Let's consider a possible namespace setup for a SaaS Company. The platform team retains a top-level namespace for shared infrastructure. On the other hand, each tenant receives a fully isolated namespace to, e.g., model a staging system.

To use this new feature, no extra setup or configuration is required. The OpenBao CLI now features a special namespace sub-command, which can be used to administrate Namespaces.

To recreate the structure of the given example, we would do the following workflow:

1. Create Namespaces:

# creates a namespace "tenant" on root-level
$ bao namespace create tenants
# creates a child-namespace within tenants
$ bao namespace create -ns tenants tenant-c
# creates a child-namespace within tenants/tenants-c
$ bao namespace create -ns tenants/tenant-c prod

Note how we use the -ns (short form for -namespace) flag to create a child-namespace tenant-c within the namespace tenant. To view status and metadata, we can utilize the lookup command:

$ bao namespace lookup -ns tenants/tenant-c prod
Key                Value
---                -----
custom_metadata    map[]
id                 8QujqL
path               tenants/tenant-c/prod/
tainted            false
uuid               f06d10c7-1300-6c4d-8616-c203b5fdf705

Furthermore, we can easily enumerate our hierarchy of namespaces by using the scan subcommand.

$ bao namespace scan
Keys
----
tenants/
tenants/tenant-c/
tenants/tenant-c/prod/

2. Create Resources within Namespaces

Since the introduction of Namespaces, many OpenBao commands have become namespace-aware. Similar to our example before, we can add -namespace or in short -ns to the command.

For demonstration purposes, let's create a KV secrets engine and populate it with a secret.

$ bao secrets enable -ns=tenants/tenant-c/prod -description="Production Secrets" kv
Success! Enabled the kv secrets engine at: kv/

Now that we have created a new kv secrets engine, it will be listed among our default engines:

$ bao secrets list -ns tenants/tenant-c/prod
Path          Type            Accessor              Description
----          ----            --------              -----------
cubbyhole/    ns_cubbyhole    cubbyhole_9a4e44d3    per-token private secret storage
identity/     ns_identity     identity_61c42408     identity store
kv/           kv              kv_b993bbec           Production Secrets
sys/          ns_system       system_ba514ba2       system endpoints used for control, policy and debugging

Let's read and write a secret...

# Create the secret within namespace
$ bao kv put -namespace=tenants/tenant-c/prod -mount=kv auth-token foo=bar
Success! Data written to: kv/auth-token

# Read it
$ bao kv get -namespace=tenants/tenant-c/prod -mount=kv auth-token
=== Data ===
Key    Value
---    -----
foo    bar

3. Lifecycle Operations

Over time, tenants will inevitably require various changes and customizations to their namespaces. Moving the administration away from the OpenBao operators means that the tenant operator can perform those changes on demand and individually. Consequently, we implemented the associated (and probably expected) lifecycle features such as:

• Namespace-aware policies and quotas
• Moving or renaming secret or auth engines across namespaces
• Locking and unlocking of namespaces

While we have many plans on extending namespace capabilities in the future to make them even more helpful, we still maintain API compatibility with Vault Enterprise to enable a smooth migration path.

Looking Ahead: Horizontal Scalability​

Namespaces will give organisations the possibility to better structure and isolate secret information. However, the introduction of namespaces is just the first step. Our vision includes supporting lazy loading of namespaces and mounts, allowing OpenBao clusters to efficiently serve workloads with many infrequently accessed resources. This will enable even greater scalability and resilience, as nodes will no longer need to load the entire system state at once.

Here is another article that you can read more about OpenBaos' scalability efforts. Sounds interesting? Reach out to the project via Github, Zulip, or Mailinglist if you want to support our work in the areas of namespaces or scalability.

Get Started​

We just carved a beta release of OpenBao 2.3. We encourage you to explore the new features and share your feedback as we continue to improve OpenBao as a secure, scalable, and self-service secrets management solution.

Looking ahead, the namespaces working group is actively exploring improvements such as namespace sealing, non-hierarchical namespaces, per-namespace storage backends, and plugin isolation. These efforts aim to make namespace usage more flexible, efficient, and secure in future releases.

We welcome contributions from individuals and organizations, whether helping to improve documentation, clarifying existing content, or contributing to upcoming features. Your input plays a vital role in shaping the future of OpenBao.

What is EdgeX Foundry?​

For those unfamiliar, EdgeX Foundry is an open-source, IoT/edge computing framework hosted by the Linux Foundation. It’s designed to enable seamless communication between devices, applications and services using a flexible, microservices-based architecture. Whether you’re working in automation, energy, or building management, EdgeX helps bring everything together in a standardized way.

The Need for Secret Storage​

Security is a top priority in EdgeX, and managing secrets—like API keys, passwords, and certificates—is crucial. Instead of reinventing the wheel, EdgeX has always integrated third-party open-source solutions for secret management. Until now, it relied on HashiCorp Vault for securely storing sensitive information.

However, with Vault moving to a Business Source License (BSL), the EdgeX community wanted to consider an alternative going forward. That’s where OpenBao comes in.

The Role of OpenBao​

OpenBao is a community-driven, open-source project under the Linux Foundation. It provides an identity-based secrets and encryption management system, ensuring sensitive data stays protected. Since it shares a strong foundation with Vault, OpenBao makes for a natural transition.

Why OpenBao?​

Adopting OpenBao brings several advantages to EdgeX Foundry, including:

✅ Seamless Migration – OpenBao is designed to be API compatible with its upstream, making the switch smooth and hassle-free.

✅ Open & Vendor-Neutral Licensing – open-source freedom and long-term community collaboration.

✅ Security-First Approach – Strong encryption and identity-based access controls keep secrets safe.

✅ Active Community Support – A dedicated team ensures ongoing improvements, security updates and feature enhancements.

What this means for EdgeX Users​

If you’re already using EdgeX, this change should be practically seamless. The Core Services have been updated to work with OpenBao while keeping the same APIs as before, meaning minimal disruption. You’ll still manage secrets securely, with the added benefit of an actively maintained and community-driven solution.

Looking Ahead​

The move to OpenBao reflects EdgeX Foundry’s ongoing dedication to security, transparency, and open-source collaboration. With EdgeX 4.0 (codenamed Odesa) now released, it is the perfect time to explore OpenBao, share feedback, and get involved in shaping its future.

Stay tuned for more updates on OpenBao and keep in touch with EdgeX and related commercial products, as they enjoy the benefits of security and openness with OpenBao going forward!

Scaling Read Operations​

Most obviously, the first step for the Horizontal Scalability working group will be to allow standby nodes to service requests as described in our project direction & roadmap:

4. Allow HA standby nodes to service read-only (from a storage modification PoV) requests. (scalability)
   • Currently HA mode standby nodes forward all requests up to the active node, preventing horizontal scalability of OpenBao. Due to limitations in Raft (only the active node can perform storage writes), we can't immediately scale writes. Thus, start by bringing these nodes "online" (loading the mount table, plugins, &c) and allowing them to service read-only requests, returning ErrReadOnly on storage write operations to trigger automatic request forwarding.

The shortcoming of this is that writes are not scalable: there is still only a single active writer node. Note that a read operation in this context refers to any API request (of any operation type) that results only in get or list ("read") storage operations. Any operations involving a storage write (put or delete) would be forwarded to the active node for processing.

This behaves similarly to Performance Secondary Standby nodes.

Scaling Write Operations​

In our RFC for namespaces, there is the following future work item:

Per-Namespace Storage Segments​

At the physical storage level, a namespace could be implemented as a new database schema in PostgreSQL (with each plugin being a new database table) or a new disk directory in Raft/BoltDB. This can likely build on top of storage views, using the existing JSON namespace data to translate source path to destination. This would help with scaling OpenBao: each tenant would have its own data storage location and so could impact other tenants less. Theoretically this could even lead to per-segment unique storage backend depending on workload characteristics assuming no cross-segment consistency is required.

However, this work is strictly disjoint from implementing namespaces as a feature and so will be done later.

In conjunction with additional changes to horizontal scalability, this change could allow each namespace to have a different active node dictated by its storage backend, distributing writes across the cluster. Notably, very few operations are truly cross-namespace; this is mostly limited to the creation and deletion of namespaces and mount move operations, which affects the namespace store in the parent context in addition to the actual target namespace.

The benefit of this approach is that it doesn't affect plugins, token stores, ACL stores, or any other namespace-specific functionality. This is because our namespace design opted to place all namespace-specific data within the namespace's path (/namespaces/<uuid>) rather than mixing it within the root of storage (/core or /sys) across all namespaces.

Additionally, this gives a natural place for segmenting storage, allowing smaller databases when used with PostgreSQL or Raft. This allows greater scalability as many have scalability problems when lots of data is written to a single backend instance. By supporting different storage types (e.g., mixing and matching PostgreSQL and Raft), namespaces with different SLAs or workload write/read ratios can be supported.

One shortcoming with this approach is it doesn't allow scaling all types of workloads. For instance, a single PKI mount which stores certificates will not be horizontally scalable as writes will not be able to be distributed. However, the majority of large, diverse workloads will be able to be distributed because of this change.

Another shortcoming is the complexity this entails for an operator. Some of this may be mitigated by allowing namespace-native configuration, without requiring full configuration file changes.

Disaster Recovery​

OpenBao recently added support for Raft non-voter nodes. This can likely form the basis of disaster recovery, especially if enablement for other storage backends (like PostgreSQL) is added.

Notably in Raft, non-voter status means that these nodes do not contribute to quorum requirements, allowing writes to commit faster but still allowing the use of Raft to distribute updates. This means that adding non-voter nodes results in additional traffic from the leader but doesn't otherwise impact write speeds. By putting these non-voters in secondary data centers, with the ability to promote non-voter nodes, we can subsequently initiate a failover operation in the event the primary cluster goes down, with minimal data loss.

PostgreSQL can similarly be extended to support a non-voter setup, wherein certain nodes will not attempt to become leaders unless updated later. This will behave similarly to Raft, allowing more read scalability via the use of read-only PostgreSQL replicas in other secondary data centers.

By broadening the use of transactions, we can further guarantee that these other nodes are consistent, independent of what storage backend is used.

Additionally, we could allow standby or non-voter nodes to serve read requests without an active leader. This could potentially be extended to support generating offline authentication tokens using JWTs or similar signature-based schemes for use with non-lease paths. This would allow for greater high availability in the event of a leadership outage.

Offline Recovery Mode​

Sometimes a hybrid replication mode would be preferred; Vault Enterprise supports this via a path filtering operation.

When we combine the earlier work for per-namespace storage backends with an additional Namespaces future work item:

Per-Namespace Seal Mechanisms​

The existing Vault Enterprise Namespace feature supports locking and unlocking namespaces by an operator with access to the parent namespace. While potentially useful to limit requests to a namespace without impacting other users, we could add a similar mechanism using the Barrier + Keyring functionality to also give us per-tenant encryption. This also lets the tenant control their own encryption keys. With lazy loading of namespaces (preventing inbound requests unless the namespace is unsealed), this behaves similarly to locking and unlocking the namespace though doesn’t conflict with it.

However, this work is strictly disjoint from implementing namespaces as a feature and so will be done later.

If we implement non-hierarchical namespaces, which do not chaining to root as a parent for tokens, ACL policies, or layered sealing, this allows us to have cluster-independent namespaces. We would additionally need the Shamir fallback from the parallel unseal RFCs.

This gives two nice properties:

1. We can have a stronger path filtering, allowing replication of namespaces in disjoint cluster topologies. For instance, a secondary site may have only a subset of namespaces, which it may be non-voter on until promoted to leader during an outage.
2. A namespace could potentially be shared by two different organizations as a form of secret sharing with some peer establishment protocol.

Notably, the parallel unseal allows either Shamir's based initial establishment or the use of public keys explicitly provisioned for local-only seal material usable by the remote cluster. This allows local disaster nodes to survive the outage of the primary seal mechanism.

Lazy Loading​

By making Namespaces have their own seal mechanism, we'll be forcing OpenBao to handle the scenario when a namespace cannot be loaded. We have two options:

1. To hard fail, requiring operators to rectify the problem immediately. With things like parallel unseal, this should be doable unless a storage engine itself is down.
2. To soft fail, refusing to load the namespace (and any children) but continuing regular operation on all other namespaces.

This sits on different sides of a security/availability problem: we could be available, but not processing key revocations because a single namespace is offline, or we could be in a similar place, not processing all revocations because a single offline namespace resulted in all namespaces being unavailable.

I'd like to see namespaces, and eventually mounts, be lazily loadable. This will likely require new interfaces for backends to indicate when they'd next like their Rollback function to be issued, if that occurs too infrequently. And mounts will need to be audited for lease expiry, with some window for mount reuse without unmounting refreshed on last request.

But, this will allow us to scale OpenBao to workloads with lots of infrequently-accessed mounts without requiring all nodes in a cluster have the capability of loading the entire system at once.

For a video, see our official YouTube channel or on the FOSDEM video mirror.

Hello everyone! I'm Alex Scheel, a Staff Backend Engineer at GitLab and Chair of the OpenBao Technical Steering Committee. I'm here to talk about OpenBao and its usage at GitLab.

To get started, the most important question is, "What is OpenBao"?

OpenBao is a fork of HashiCorp Vault, remaining under the original Mozilla Public License, with open governance under the Linux Foundation's LF Edge subproject.

It is an advanced secrets manager supporting four categories of features:

1. Static secrets, provisioned by users directly and securely stored.
2. Dynamic secrets, automatically generated on-demand to integrate with things like databases or cloud provider identities.
3. Encryption services, supporting PKI or SSH certificate creation and KMS-like encryption-as-a-service functionality.
4. And lastly, Sync, Visibility, and Management of these secrets: integrations with Kubernetes External Secret Operator, Cert-Manager, and a custom templating agent to handle last-mile delivery of secrets. All of this comes with detailed audit logs tracking access to the system.

A secrets manager is, to organizations and developers writing applications, what a password manager is for humans.

OpenBao is API-driven and highly flexible. It supports many different authentication engines, from OIDC and Kubernetes to LDAP and Kerberos, and creating or storing many types of identities, letting it function as an identity broker.

The goal is to trade-off a little bit of operational risk -- one more, admittedly complex, service to run -- to greatly lessen the security risk of compromise: you can enforce a healthy secrets posture to ensure that rotation and revocation of sensitive credentials are possible, limiting your organization's exposure window in the event of compromise.

While a complete timeline of OpenBao would take a while, there's a few important events I want to highlight:

• HashiCorp first released Vault in April of 2015, so there has been nearly a decade of work on it.
• In August of 2023, HashiCorp announced that they'd be relicensing their products under the non-OSI Business Source License (violating freedom zero); this triggered the early fork of Terraform into OpenTofu, our sister project also under the Linux Foundation.
• The fork of Vault by IBM Software came much later, in November of that year, and kept with the naming convention. Unlike OpenTofu, which was made up of many member companies which sponsored full-time development, OpenBao has been more of a grass-roots, community effort.
• We've done 8 releases -- two major and six bug fixes, shipped several core improvements, and put together our Technical Steering Committee and governing documents.
• My employer, GitLab, joined the project officially in July and achieved voting status in October of last year. To the best of my knowledge, I'm incredibly fortunate to be one of the few people employed to work on OpenBao full time.

One of the things you might be asking is, OK, it is another fork of a HashiCorp product. What's different about this one?

While OpenBao remains in the spirit of the original project, we've made a few changes to make it easier to maintain and contribute:

1. We started with a reduced base, removing many components that an open-source group with no formal funding could not support. That meant making hard choices around storage backends -- Raft and now PostgreSQL are the only supported two -- but also the removal of proprietary cloud integrations for authentication and dynamic secrets for the time being. We hope to revive especially the cloud auth and secret plugins as we get more funding and interest.
2. Using that space, we've made core technological improvements to the storage subsystem--paginated lists and transactions--which in turn allowed us to improve scalability. These improvements were directly made possible by removing every single storage backend but Raft. We hope these types of improvements will continue after we add APIs for additional external, community-provided storage plugins, so that we can make long-term improvements to upcoming features like namespaces and horizontal scalability.
3. Lastly, we've set up an open organization with clear contribution and project leadership processes. Our RFC process is open to anyone to draft designs and propose major or minor changes to the architecture or project. We've started a community mentorship program, helping two individuals with different familiarity with OpenBao to contribute. But throughout this, we continued under the original MPL license.

In short, my personal vision for where I hope OpenBao can go is to build an ecosystem similar to Kubernetes: companies large and small, and individuals experienced or just starting out, can contribute to the project and find market segments to provide derivative offerings while contributing to a common code base.

Next, you might be asking "why GitLab"?

For a while, GitLab has been focused on building a more complete DevSecOps platform.

A common problem our customers face is managing secrets in their CI/CD Pipelines. Today, they have one of two options:

1. They can use masked variables. These have very few controls over the scope of access and customers can't easily differentiate in the dashboard between secrets and inconsequential data, though all are stored encrypted at rest. A few customers have many thousands of variables and the difference in ideal behavior differs substantially between the two use cases for them. No one solution will really work for both.
2. Instead, many customers turn to our external integrations with various other secret management providers. However, the user experience isn't quite there: one typically has to work across several teams to enable OIDC ID Token authentication so pipelines can access secrets and the customer must organize the logistics of secret storage and rotation. Because GitLab plays a very minor role, mostly by providing identity to the pipelines, it lacks visibility into the actual secrets and the hierarchy administrators have set up.

In short, there is no easy-to-use, per-project dashboard for secrets and neither approach can provide this easily. And many times, a developer has to find many individuals in their organization with permissions to do one of these steps in order to onboard their new project.

In late 2023, GitLab started working on this problem, building their own design from the ground up. In May, my colleague, Jocelyn, published this in a blog post, considering the use of OpenBao to solve this problem.

OpenBao is a great choice for GitLab because we already support a HashiCorp Vault integration. For Premium and Ultimate tier customers, the Pipeline runner features native integration for fetching secrets from Vault: while all GitLab versions support issuing the OIDC ID Tokens, these paid plans offer support for a simplified pipeline syntax, improving the developer experience over manual CLI calls to Vault.

Being a mature secrets management solution already, OpenBao brings several useful properties to GitLab:

1. It has already been deployed at scale by various entities.
2. It has historically undergone audits and certifications useful for enterprise customers.
3. It is a self-contained service, which we can isolate from the rest of our environment and treat as trusted.

What we'd like to do is provide a single dashboard which lets us aggregate information into a project-specific view of secrets. Whereas previously audit logs might be separate between GitLab and a backend Vault instance, we can correlate them natively within the GitLab UI. Or, where access controls required understanding two separate permissions models--that of GitLab and that of Vault's ACL policies--GitLab will translate its own roles into OpenBao policies on behalf of the user.

At its core, our architecture is just like it was before, but with OpenBao replacing HashiCorp Vault and moving it from the customer's deployment to our management scope. It remains a separate service that pipeline workers talk to and customers will eventually be able to bring their own KMS to secure key material as well.

What is new is the interactions, typically driven by a user, between Rails and OpenBao for management purposes. When a user initiates any secrets management request, they do so via GitLab's Rail's API, which enforces initial authentication and holds privileged tokens for ACL management. In the future, we seek to add a User JWT issuer, allowing direct secret write operations by users without invoking Rails.

We thus get a clean trust separation, between three different entities:

1. GitLab Rails, which will perform various administrative actions on behalf of users,
2. Pipelines, which will execute jobs and fetch secrets, and
3. OpenBao, which is the source of truth for authorization and secret storage.

With official PostgreSQL support, OpenBao will be able to use the same database as Rails for smaller self-managed GitLab instances, avoiding the complexities of managing Raft which come with running Vault. Data will be encrypted before storing it in PostgreSQL and requires keys from a seal mechanism to decrypt.

The key to integrating OpenBao is to be opinionated about its data storage layout. Philosophically, OpenBao aligns with key/value databases: its API path structure usually mirrors the underlying data storage. ACL policies are groups of grants to specific capabilities–like read, write, delete–on specific API paths, with a default deny rule. After authentication, perhaps via a third-party engine, OpenBao issues a token, mapping an identity to a set of ACL policies. Thus, there is no concept of a long-term identity which has ownership of a secret.

The core of our design to integrate with OpenBao is a data layout which provides multi-tenancy and isolation of secrets. This architecture is familiar to many of our customers who have successfully integrated with their HashiCorp Vault clusters.

Each top-level tenant, usually the owner of a repository, has a separate path area in OpenBao. Within this space, they have an authentication mount for authorizing pipeline's access to secrets, with roles corresponding to each project. Each project has its own KVv2 secrets engine for storing that project's secrets, which contains custom metadata indicating the scope and contextual information like description.

GitLab Rails can read this metadata and provision ACL policies and JWT roles for all scopes when changes are necessary.

When support for Namespaces lands upstream, we'll seek to provision each tenants into their own namespaces and, long-term, we wish to use unique encryption keys with separate seal mechanisms to provide greater data isolation on GitLab.Com via per-tenant encryption keyrings.

Within the core management space is another authentication mount for authorizing Rail's own requests into OpenBao. The core of these requests are for provisioning new tenants and their mounts but also being the trusted entity that maintains the ACL policies. These ACL policies reflect the permissions and scopes of access designated by users for accessing these secrets: each environment or branch for a project is a separate policy, which contains read access to secrets within that scope.

Authentication occurs via a properties based model. OIDC ID Tokens issued by GitLab to pipeline jobs have claims reflecting contextual metadata, such as the repository, branch, initiating user, and environment. Each tenant's JWT mount matches these claims to a per-project role, with dynamic generation of ACL policies based on the properties present on the token.

Uniquely, this model lets us use OpenBao as the single source of truth for all operations: permissions are reflected via ACL roles, scopes and context are present in contextual metadata, and Rails is only present as a management engine and a trusted token issuer and pipeline provisioner and does not store data of its own.

So from a developer's perspective, all this work leads to the following two actions to use secrets in their pipelines:

1. Create a new secret via the native GitLab UI, specifying the scope of access and its value.
2. Access the secret from a pipeline's .gitlab-ci.yml definition file, using the secret name as the reference key.

And GitLab Rails and OpenBao will handle the rest.

At GitLab, we've been really excited about using OpenBao. If you are too, we welcome contributors of any sort! You can check out our roadmap, join our weekly community calls, or talk about OpenBao on social media or at conferences.

If you're a customer interested in the work GitLab is doing to build our native pipeline secrets solution, ask your account team about joining our beta program.

I'm happy to take questions now, but if you have any additional questions or want stickers or a picture with our cute mascot, BaoBao, find me afterwards in person or online!

Many thanks to Fatima for running the community calls in our absence!

photo credit: nicolas; pictured: alex

FOSDEM '25 was Alex's first time speaking at a conference and we are happy to report it was a success! The details of the talk were saved and will be published in a follow-up blog along with the recording when it is made available from the conference. It was great to meet so many people interested in identity and access management and OpenBao's role in solving secrets management problems, at GitLab and beyond.

After arriving in London, BaoBao was happy to find the conference venue was very well organized and everything was ready. And look, our logo is situated nicely on the list of table holders! BaoBao thanks OpenUK for generously sponsoring OpenBao's attendance this year and hopes to participate in the future.

During setup, BaoBao took a place on the table to get ready for the next two days of meeting people. When OpenTofu couldn't make it, Alex volunteered to represent them as well, which served as a great conversation starter. BaoBao knows that they'd do the same, after all, we're both OpenHashiForks!

After meeting the kind folks from Percona at FOSDEM, we encountered them again at State of Open, where they played hijinks with BaoBao! When Alex wandered off, they bestowed upon BaoBao a Valkey hat -- a good reminder to revive and rename our secrets engine! BaoBao appreciated the effort to avoid feeling chilly in the big conference center.

Later, OSA Con joined in the fun, placing a warning sticker that this booth talked too much about open source... BaoBao thought this disclaimer was much needed!

photo credit: nigel; pictured: alex and dotan

Finally, BaoBao and Alex got to meet Dotan Horovits, the CNCF Ambassador who made the necessary connections to get OpenBao invited to State of Open Con!

Both Alex and BaoBao had lots of fun networking and making friends with fans of OpenBao. Thank you to everyone who organized these events and helped them run so smoothly! A special thanks to all who offered to support OpenBao or consider it for use in their company!

From snowy Minnesota, BaoBao took its first ride to London Heathrow to begin its voyage to FOSDEM '25. With a brief layover, it explored King's Cross and the Coal Drops Yard. Rich with history, this area has long been used as a rail hub for the UK and now is connected to Europe via the Channel Tunnel.

From there, BaoBao boarded a train to Paris, France for some sightseeing and meetings with community members. BaoBao wishes that secrets management was as easy as traveling via European trains, but, with the help of the community, knows it will improve!

First up, the Eiffel Tower!

While it was rainy and dark, seeing the structure all lit up in person was very impressive! BaoBao thought the structure loomed larger than life, just like attackers in the shadows, targeting deployments using insecure secrets management practices...

The next day, BaoBao explored Paris a bit more, visiting the Notre Dame, the Pantheon, and finally, the Louvre! One of the largest museums in the world, filled with all types of art, from paintings to sculptures to fashion, this museum would take several visits to fully appreciate. BaoBao thinks there's art to open source too, and wishes it would be in its own museum some day!

BaoBao went out for dinner with its colleagues, Julien Cassignol (right), OpenBao TSC representative and CTPO of Wallix, and Dan Ghita (left), OpenBao Maintainer and backup Viaccess-Orca representative. BaoBao thinks its favorite part of open source is the friends you make along the way.

On Friday, BaoBao took the train from Paris to Brussels to join the fun that is FOSDEM. At dinner with friends old and new, BaoBao met Jan Martens, OpenBao Maintainer.

And now, BaoBao rests for the conference ahead -- and hopes to see you there!

I'm happy to share that OpenBao will be present at two conferences in February: FOSDEM '25 in Brussels and State of Open Con 2025 in London!

FOSDEM​

I will be giving a talk on Sunday, February 2nd, at 2:05 PM GMT+1 in room UA2.118 (Henriot) as part of the Identity and Access Management track. I'll give a brief history of the project, share how GitLab is using OpenBao, and ways you can become involved. Several OpenBao maintainers will be there, including Jan Martens, and stickers will be available!

This talk will also be live-streamed for anyone unable to make it.

State of Open Con​

Thanks to the generous help of Amanda Brock and Dotan Horovits, OpenBao has been provided a free non-profit table at State of Open Con from Tuesday, February 4th to Wednesday, February 5th. If you're curious about OpenBao and want to try some hands-on demos, stop by the booth--plenty of stickers will be available and we'll be contributing an OpenBao mascot (like pictured above) to the prize drawing on Tuesday!

If any community members are interested in helping to staff the booth, let us know as we have one free registration remaining.

Meetings​

If anyone is interested in meeting with me in either Brussels or London, please reach out via email.

In-Progress​

openbao#697 / vault#17189 - SSH CA Multi-Issuer​

Description: Vault and OpenBao have long had issues supporting proper rotation of CAs in both the SSH and PKI engines. Previously, an operator would either have to remount an entirely new engine, manually copying configuration and policy, or they'd have to destructively remove the existing CA and create a new one, creating a downtime window. While PKI got updates to add multi-issuer support in Vault v1.11, neither Vault nor OpenBao saw improvements to SSH CA rotation. This RFC by one our OpenBao Mentees, Gabriel, brings parity by adding multi-issuer support to the SSH engine for zero-downtime CA rotation.

How you can help: Gabriel is working hard on the implementation for this feature, but we'd appreciate feedback on the design from any users of the SSH CA!

openbao#753 - CEL for PKI Policy​

Description: The PKI Engine's role-based certificate validation method is inflexible: every possible certificate field and extension, complete with desired validation mechanism must be implemented in the engine itself, requiring a release for users to adopt. For instance, allowed_domains on a role today doesn't support setting default domains or rejecting requests with only a partial list of allowed domains. While functionally equivalent (in that, a user could request multiple certificates each with a subset of approved domains), it is hard for a PKI operator to ensure that all approved domains have been issued for in a single certificate. Fatima, another OpenBao Mentee, proposes to use Google's Common Expression Language (CEL), widely used in GCP and Kubernetes for validation and templating, to enforce issuance policy and template the final certificate from request parameters.

How you can help: Have experience using CEL or doing complex, company-specific PKI integrations? We'd love to hear from you about the proposed design!

openbao#787 - Add Namespace Support​

Description: Vault Enterpise supports Namespaces, a way of creating multi-tenancy and delegating permissions without running multiple clusters. Users have requested similar abilities with OpenBao, so a temporary working group was formed to create the design and initial implementation. This RFC, published by Peter, proposes API compatibility for consuming applications but suggests many future improvements to scalability and tenant isolation.

How you can help: While the initial implementation will be done by the Namespace WG, we welcome feedback on the design, testing of the feature, and designs and implementations for future enhancements.

openbao#549 / vault#5275 - Recursively List Keys​

Description: The most widely requested Vault feature with no solution is adding the ability to recursively list keys. Prior to transactions (for consistency) and pagination (for resource constraining expensive calls), this was hard to do safely. This RFC proposes introducing a new operation, with HTTP verb SCAN or via GET with ?scan=true, and ACL capability (scan) to allow plugin authors to introduce recursive list endpoints and operators to secure access to them.

How you can help: After the initial implementation is merged, it would be great to have feedback or PRs on additional endpoints to use this new operation.

ACL Improvements - openbao#769 / vault#5362 - Filter LIST Results and openbao#791 - Enforce List Pagination​

Description: The fourth-most widely requested Vault feature with no solution is filtering LIST results to only show accessible paths. This RFC proposes a new ACL policy parameter, list_scan_response_keys_filter_path, which contains a path to template with each list response item (from .keys) to check against the ACL system for visibility under the same token policy. While this is an expensive operation, a follow-up RFC proposes another parameter, pagination_limit, to allow policy authors to require usage of paginated lists (thereby reducing the load on path filtering).

How you can help: It would be great to have feedback on the templating design and how support for multiple paths could potentially behave (with an AND or OR conjunctions).

Completed​

openbao#296 - Transactional Storage​

Description: Recently merged and released in v2.1.0 was support for transactional storage across the entire OpenBao stack (from underlying physical storage backend to plugins). This allowed us to improve the scalability and fault-tolerance of OpenBao above and beyond what HashiCorp Vault had.

How you can help: A follow-up tracking issue invites contributions of places where transactions should be used. As always, we welcome testing of this feature.

Upcoming​

openbao#235 - Add XChaCha20-Poly1305 Barrier Encryption Support​

Description: OpenBao uses AES256-GCM96 as its barrier encryption algorithm. While suitably secure and FIPS compliant, this requires frequent key rotation to compensate for the 96-bit nonce (which is not collision resistant). Switching to XChaCha20-Poly1305 would allow us to maintain a smaller barrier keyring (as key rotation would not need to be done automatically) and avoid concerns over nonce collisions.

How you can help: While a proof of concept was proposed, we're looking for a volunteer to take over polishing the feature!

openbao#17 - Add ACME Support to the TLS Listener​

Description: While OpenBao supports PKI capabilities, due to a chicken-and-egg problem, it is hard to issue the TLS listener's own certificate via a CA stored in OpenBao. With auto-unseal and ACME support, however, it would be possible to do and greatly improve operator's experience when using other CAs as well.

How you can help: While a proof of concept was proposed, much polish was needed to complete this feature and we're looking for a volunteer to take over development!

We are thrilled to announce the availability of OpenBao v2.1.0, focused on safety and scalability improvements!

This release spent some time laying the groundwork for safety and scalability improvements for releases to come. With the help of the community, OpenBao will now take advantage of transactional storage semantics from its underlying data store, giving operators and plugin developers confidence in the consistency of storage writes. This storage safety allows us to focus on alternative storage layouts for improving scalability, for instance, increasing the maximum number of mount table entries past the single-entry limit.

This release also features contributions from many new and repeat contributors; thank you all!

Key highlights of the release​

Major features:

• Remove Mount Table Limits: Using transactional storage, we've split the auth and secret mount tables into separate storage entries, removing the requirement that the entire table fit into a single storage entry limited by max_entry_size. This allows potentially hundreds of thousands of mounts on a single scaled-up server.
• Transactional Storage: Plugin developers can now take advantage of safe storage modification APIs when the underlying physical storage supports them. The physical.TransactionalBackend and logical.TransactionalStorage types allow developers to begin read-only and writable transactions, committing or rolling back the desired changes.
• Transit: Support PKI CSR and certificate storage alongside key material. This allows callers to securely create keys and submit requests for certificates without the key material leaving Transit. Storage of the certificate on the key avoids the need for an additional K/V mount. Rotation of this certificate and its chain is also supported.
• physical/postgres: Reintroduce Postgres database for OpenBao storage, implementing paginated list support. This feature is currently in preview and breaking changes may occur.

Plugin improvements:

• auth/jwt: Allow templating ACL policies from data in claims on JWT or OIDC ID tokens.
• auth/oidc: Add a new oauth2_metadata configuration option to enable sending any of the tokens from the token issuer to the client.
• auth/oidc: Add a new callback_mode role option value device to use the oidc device flow instead of a callback, add a new poll_interval role option to control how often to poll for a response, and add a new callbackmode=device option to the oidc login method in the cli.
• auth/oidc: Add new callback_mode=direct role option to cause the oidc callback to be direct to the server instead of the client, and add a callbackmode=direct option to the oidc login method in the cli.
• docker: add /bin/vault symlink to docker images.
• rpm: Fix packaging to properly annotate configs entries for noreplace.
• secrets/kv: Implement transactions to prevent canceled operations from corrupting storage.
• secrets/pki: Use transactions for root generation, issuer import.
• secrets/pki: add not_before parameter to precisely define a certificate's "not before" field.
• secrets/pki: Add revoked_safety_buffer to control retention on revoked certificates separately from expired certificates.
• secrets/pki: Delete invalid certificates during tidy via tidy_invalid_certs=true if they cannot be parsed due to Go's x509 handling.
• secrets/pki: Support revoking expired certificates with the allow_expired_cert_revocation CRL configuration.
• storage/postgresql: Allow table creation to improve first-start UX.

Stay tuned for more great features to come!

Looking ahead​

Several feature-related working groups have spun up, including one around Namespaces which will bring welcomed multi-tenancy improvements in the future. OpenBao's mentees have also been making great improvements! Stay tuned for progress on multi-issuer SSH CA support and Common Expression Language (CEL)-based PKI issuance policy.

Changes are well underway for the next release, including usability enhancements to the PKI engine, improvements to the K/V engine based on paginated lists and transactional storage, additions to the ACL system to handle recursive listing, and an RFC for restricting the results of list operations to only visible entries.

If anyone has private forks of HashiCorp Vault, we are happy to collaborate around timely security fixes or syncing modifications to the core or plugins.

As always, we appreciate any and all contributions!