Automated Deployment

The Data Streamhouse can be fully deployed and configured as part of an Infrastructure-as-Code (IaC) pipeline. This allows organizations to automate environment provisioning, connection setup, access control, and security configurations consistently across development, staging, and production environments.

This guide covers the key mechanisms required for automated deployments.

---

Dynamic Configuration Using Placeholders

When configuring Kafka or Amazon Kinesis connections, Portal supports environment variable placeholders to inject sensitive or environment-specific values dynamically.

Placeholder Syntax

Use the following syntax in any supported field:

${env:CON_VARIABLE_NAME}

Where CON_VARIABLE_NAME is the name of an environment variable exposed to the Portal process.

Important:

• The environment variable must start with CON_ (e.g., CON_KAFKA_USERNAME).

• Non-prefixed variables will not be substituted for security reasons.

Example

Kafka security configuration with placeholders:

org.apache.kafka.common.security.plain.PlainLoginModule required username="${env:CON_KAFKA_USERNAME}" password="${env:CON_KAFKA_PASSWORD}";

Container environment variables:

export CON_KAFKA_USERNAME="kafkauser"
export CON_KAFKA_PASSWORD="strongpassword123"

If a referenced environment variable is not set, the placeholder remains unchanged.

Supported Fields

Placeholders can be used in the following connection fields:

• Broker addresses (Kafka / Kinesis endpoints)

• Schema Registry URL

• Security configurations (SASL, truststores, keystores)

• Schema Registry security credentials

• Additional configuration fields

• Custom properties (JSON)

• JMX configuration (JSON)

---

Initializing Roles and Groups via Files

Instead of manually creating roles and groups through the Portal UI, Data Streamhouse supports initialization using YAML configuration files.

This enables full declarative configuration of access control as part of your IaC pipeline.

Configuration Environment Variables

Set the following environment variables during deployment:

DSH_PORTAL_INIT_ROLES_FILE=/path/to/roles.yaml
DSH_PORTAL_INIT_GROUPS_FILE=/path/to/groups.yaml

Important: On each startup, existing roles, groups, and user-group assignments will be cleared and replaced by the content of the specified YAML files.

Prerequisites

• LDAP or OpenID Connect synchronization must be active.

• User-group assignments must be managed externally (not manually inside Portal).

---

Roles File Structure (roles.yaml)

- name: "Role Name"
  description: "Role Description"
  rights:
    - action: "PermissionAction"
      resource: "ResourcePattern"
      effect: "Allow"

Example

- name: "Finance Team Role"
  description: "Access to all finance topics"
  rights:
    - action: "TopicAccessAll"
      resource: "1:finance*"
      effect: "Allow"

---

Groups File Structure (groups.yaml)

- name: "Group Name"
  externalMapping: "ExternalDirectoryGroupName"
  description: "Group Description"
  roles:
    - "Role Name"

Example

- name: "Finance Team QA"
  externalMapping: "us-finance-department-quality"
  description: "QA team for finance streams"
  roles:
    - "Finance Team Role"

Important:

• The externalMapping field must exactly match the external group name from LDAP or OpenID.

• Role names assigned to groups must match the role names defined in roles.yaml.

---

License Activation

Automated deployments must account for license activation to ensure the Portal setup is properly authorized.

Attention: License activation must be handled carefully during automated deployments.

Once activated, a license is tied to the Portal setup and stored in the connected database. If the database is re-created (e.g., using an ephemeral embedded database in testing environments), the license assignment will be lost.

Important: In such cases, Portal will not automatically reassign the license using online activation. Offline Activation or Activation via Public API must be used to enforce license reassignment.

For a full overview of supported activation methods and license compliance requirements, refer to the License Activation Scenarios article.

Recommendations

• Always deploy Portal with a persistent external database in production and staging environments.

• Automate license challenge-response activation via the Public API if using fully ephemeral environments.

• Track license assignments carefully to avoid under-licensed states or activation failures during automated deployments.

---

Best Practices for IaC Deployment

• Always use placeholders for secrets and environment-specific values.

• Use YAML configuration for groups and roles to maintain version control and repeatability.

• Leverage health endpoints (/live, /ready, /health) to integrate readiness checks into your deployment pipelines.

• If running behind an ingress or proxy, configure TLS and WebSocket forwarding correctly.

---

Public API for Advanced Provisioning

Data Streamhouse also offers a Public API that can be used to:

• Configure connections

• Manage catalog metadata

• Provision full data streaming environments programmatically

Note: Public API usage is documented separately.