Advanced Setup
For production environments requiring high availability (HA) and horizontal scalability, the Data Streamhouse Portal can be deployed as a cluster.
This configuration ensures:
No single point of failure
Continuous availability even during node failures
Dynamic scaling of additional Portal instances
Overview of Cluster Architecture
A Portal cluster consists of:
Master Nodes — Static nodes responsible for maintaining cluster stability and coordination.
Auto-Scaling Nodes — Dynamic instances that join the cluster at runtime to handle additional load.
Master Node Configuration
Master nodes form the backbone of the cluster. At least three master nodes are recommended to tolerate up to two simultaneous failures.
1. Define the Master Nodes
Decide which instances will act as masters. Example configuration:
Portal-1
portal-1
2551
Portal-2
portal-2
2552
Portal-3
portal-3
2553
All master nodes must know about each other.
2. Configure Each Master Node
Set the following environment variables on each master:
Example: Portal-1
Example: Portal-2
Example: Portal-3
Important: All masters must list the same
DSH_PORTAL_CLUSTER_NODES. Hostnames must resolve correctly inside the Kubernetes cluster or network.
Auto-Scaling Node Configuration
Once the master nodes are stable, you can dynamically launch additional Portal instances to scale horizontally.
Auto-scaling nodes need only one configuration:
When an auto-scaling node comes online:
It contacts the master nodes.
Registers itself into the cluster.
Receives the full cluster membership list through gossiping.
There is no need to manually configure DSH_PORTAL_CLUSTER_HOSTNAME or DSH_PORTAL_CLUSTER_PORT for auto-scaling nodes; these are determined automatically at runtime.
Best Practices for Cluster Deployments
Use fully qualified domain names (FQDNs) for all node references in
DSH_PORTAL_CLUSTER_NODES.Run an odd number of master nodes to optimize cluster quorum and failover (typically 3, 5, or 7).
Distribute master nodes across availability zones (if in a cloud environment) for maximum resilience.
Configure resource requests/limits properly to ensure predictable capacity.
Monitor cluster health through Portal’s
/healthendpoint and external monitoring systems.Test node failures regularly to verify failover behavior and cluster stability.
Example Cluster Configuration Summary
Master Node
Set DSH_PORTAL_CLUSTER_NODES, DSH_PORTAL_CLUSTER_HOSTNAME, DSH_PORTAL_CLUSTER_PORT
Auto-Scaling Node
Set DSH_PORTAL_CLUSTER_NODES only
Key Environment Variables for Cluster Mode
DSH_PORTAL_CLUSTER_NODES
Comma-separated list of all master nodes (host:port)
DSH_PORTAL_CLUSTER_HOSTNAME
Hostname of the node (masters only)
DSH_PORTAL_CLUSTER_PORT
Port for cluster communication (masters only)
Last updated
Was this helpful?