Apache Kafka database secrets engine

Apache Kafka is one of the supported plugins for the database secrets engine. This plugin issues dynamic SCRAM-SHA-256/512 credentials against an Apache Kafka cluster using the AdminClient API (via franz-go's kadm package). It also supports Static Roles.

See the database secrets engine docs for more information about setting up the database secrets engine.

Unlike the SQL-style database plugins, creation_statements is a single JSON document describing the SCRAM mechanism, iteration count, and optional ACLs for the new credential, rather than a semicolon-separated list of statements. Under the hood the plugin writes (and, on revoke, deletes) a SCRAM user record via the AlterUserSCRAMs AdminClient RPC and provisions ACL grants via the Kafka AdminClient CreateACLs RPC. When the credential lease is revoked or expires, both the SCRAM user and all associated ACLs are automatically deleted.

Capabilities

Plugin Name	Root Credential Rotation	Dynamic Roles	Static Roles	Username Customization
kafka-database-plugin	Yes	Yes	Yes	Yes

Setup

1. Enable the database secrets engine if it is not already enabled:

   $ bao secrets enable database
   Success! Enabled the database secrets engine at: database/

   By default, the secrets engine will enable at the name of the engine. To enable the secrets engine at a different path, use the -path argument.

2. Configure OpenBao with the proper plugin and connection information. The username/password root credentials must already exist as a SCRAM user on the cluster with permission to manage other users' SCRAM credentials:

   $ bao write database/config/kafka \
       plugin_name="kafka-database-plugin" \
       allowed_roles="producer" \
       brokers="broker1.example.com:9092,broker2.example.com:9092" \
       username="admin" \
       password="admin" \
       mechanism="SCRAM-SHA-256"
   Success! Data written to: database/config/kafka

3. Configure a role that maps a name in OpenBao to a SCRAM credential document to execute to create the database credential:

   $ bao write database/roles/producer \
       db_name="kafka" \
       creation_statements='{"mechanism":"SCRAM-SHA-256","iterations":4096}' \
       default_ttl="1h" \
       max_ttl="24h"
   Success! Data written to: database/roles/producer

   mechanism and iterations are both optional and default to SCRAM-SHA-256 and 4096 respectively when omitted.

Usage

After the secrets engine is configured and a user/machine has an OpenBao token with the proper permission, it can generate credentials.

1. Generate a new credential by reading from the /creds endpoint with the name of the role:

   $ bao read database/creds/producer
   Key                Value
   ---                -----
   lease_id           database/creds/producer/2f6a614c-4aa2-7b19-24b9-ad944a8d4de6
   lease_duration     1h
   lease_renewable    true
   password           SsnoaA-8Tv4t34f41baD
   username           v-token-producer-x7g2kf8s1a

   The returned username/password pair is a SCRAM credential on the Kafka cluster using the mechanism configured on the role (or the config's mechanism, by default SCRAM-SHA-256). Clients authenticate with it exactly as they would any other SASL/SCRAM Kafka user.

Kafka-specific notes

• AdminClient API, not the SQL-style statement model: The plugin manages SCRAM credentials via AlterUserSCRAMs, not SQL. creation_statements is a single JSON document, not a list of statements.
• Dynamic ACL management: creation_statements can include an acls array specifying resource permissions (e.g. for TOPIC, GROUP, CLUSTER) to grant to the issued credential via Kafka's CreateACLs AdminClient RPC. When the credential lease expires or is revoked, the plugin automatically removes all associated ACLs.
• PLAIN mechanism is not supported for root login: The plugin's root connection to the cluster always authenticates with SCRAM-SHA-256 or SCRAM-SHA-512; mechanism=PLAIN in database/config/:name is rejected.
• Revocation deletes both mechanisms: DeleteUser calls AlterUserSCRAMs for both SCRAM-SHA-256 and SCRAM-SHA-512 for the username, so a credential is fully removed regardless of which mechanism it was created with. A missing record is not treated as an error.
• Root rotation and static-role rotation query existing credentials: Both query the user's existing SCRAM credentials via DescribeUserSCRAMs and update the discovered mechanism(s) and iteration count, preserving their existing security parameters. If no SCRAM credentials exist, it falls back to the configured or default mechanism (SCRAM-SHA-256, 4096 iterations).

API

The full list of configurable options can be seen in the Kafka database plugin API page.

For more information on the database secrets engine's HTTP API please see the Database secrets engine API page.