System Requirements

All components of the Data Streamhouse run in Linux-based environments, whether deployed on virtual machines, bare-metal servers, or as containers in Kubernetes or similar orchestration platforms.

Each component has specific resource requirements based on its function and expected workload.

Container Requirements for Portal

Portal serves as the control plane and user-facing interface. It requires more resources than other components due to data browsing, access control, and orchestration responsibilities.

• Minimum:

  • 2 vCPU

  • 8 GB RAM

  • Disk: 4 GB (SSD recommended)

  • External Postgres database

• Recommended for production:

  • 4+ vCPU

  • 16+ GB RAM (depending on expected concurrency)

Note: The number of concurrent users, parallel data exploration sessions, and API calls directly impact CPU usage. For environments with heavy ad-hoc query usage, multiple clusters or large datasets, scale CPU accordingly.

To fully utilize available memory, configure Portal's Java runtime settings explicitly using the DSH_PORTAL_OPTS environment variable (e.g., -Xms8g -Xmx12g). Without explicit configuration, the JVM may not allocate optimal memory for high-concurrency workloads.

Database Requirements

Portal requires an external PostgreSQL database to operate in production environments. The database stores:

• System configurations

• User management information

• Audit logs

• Metadata related to data streams and environments

Minimum Requirements

• Type: PostgreSQL 12 or higher

• Initial Disk Space: 1 GB (SSD recommended)

• Access: Full read and write permissions to the assigned schema (default: public)

Note: Database disk usage will grow over time, especially with active audit logging and large user bases. Monitor disk usage periodically and expand storage as needed.

Recommendations

• Run the database on dedicated infrastructure or a managed PostgreSQL service for production-grade resilience.

• Enable automatic backups and retention policies according to your organization's standards.

• Ensure low-latency network connectivity between Portal instances and the database.

Important: In-memory or temporary databases must not be used for production environments.

Container Requirements for Argus

Argus runs as a lightweight headless monitoring agent.

• Minimum:

  • 1 vCPU

  • 4 GB RAM

  • Disk: 4 GB (SSD recommended)

• Recommended:

  • 2 vCPU

  • 8 GB RAM (for large Kafka clusters or frequent health checks)

Container Requirements for Machina

Machina executes streaming applications and system processes. Resource requirements depend on the workload.

• Minimum:

  • 2 vCPU

  • 4 GB RAM

  • Disk: 4 GB (SSD recommended)

• Recommended:

  • 4+ vCPU

  • 8–16 GB RAM (depending on app complexity and parallelism)