This document describes Symphony Connectors for Cloud de Confiance by S3NS, which provide integration between IBM Spectrum Symphony and Cloud de Confiance by S3NS. This integration lets you use Cloud de Confiance by S3NS resources as flexible compute capacity for your Symphony clusters. Symphony then manages the cluster by using host factory. Host factory is a component of IBM Spectrum Symphony that extends your cluster with compute hosts from cloud providers like Cloud de Confiance by S3NS to meet resource demand in your Symphony clusters.
Use Symphony Connectors for Cloud de Confiance by S3NS to supplement your existing cloud infrastructure with flexible Cloud de Confiance by S3NS resources, particularly for high performance computing (HPC) workloads. Symphony Connectors for Cloud de Confiance by S3NS supports both Compute Engine and Google Kubernetes Engine services.
When to use Symphony Connectors for Cloud de Confiance by S3NS
Symphony Connectors for Cloud de Confiance by S3NS is designed to bridge the gap between Symphony's host factory service and Cloud de Confiance by S3NS. Use Symphony Connectors for Cloud de Confiance by S3NS in the following scenarios:
Cloud bursting: When your on-premises cluster resources are insufficient, dynamically scale your Symphony workloads into Cloud de Confiance by S3NS to access additional compute capacity. Symphony Connectors for Cloud de Confiance by S3NS lets you automatically provision resources as needed, integrated with existing Symphony job scheduling.
Hybrid cloud deployments: If you run Symphony workloads across both on-premises and Cloud de Confiance by S3NS infrastructure, you can use this solution to unify resource management through the Symphony host factory framework. This approach creates a consistent experience for job submission and monitoring, and flexible resource allocation.
Cost optimization: Use Cloud de Confiance by S3NS's flexible instance types and pricing models, such as Spot VMs (preemptible VMs), to run cost-sensitive workloads.
Development and testing: To provision temporary compute resources for development or testing workflows, use Symphony Connectors for Cloud de Confiance by S3NS rapidly set up and tear down environments. Costs are optimized through automatic resource cleanup.
Unified management: Use the host factory to create a consistent interface to manage both GKE pods and the Compute Engine instances that are used by Symphony.
Seamless enterprise integration: Use this solution to extend your existing Symphony deployments into Cloud de Confiance by S3NS without requiring significant architectural changes.
Learn how Symphony Connectors for Cloud de Confiance by S3NS works
Symphony Connectors for Cloud de Confiance by S3NS provide the necessary components to make Cloud de Confiance by S3NS a provider for host factory.
The following diagram shows the architecture of Symphony Connectors for Cloud de Confiance by S3NS:
As shown in the preceding diagram, this architecture contains three main groups: an IBM Spectrum Symphony environment, a Cloud de Confiance by S3NS host factory provider layer, and Cloud de Confiance by S3NS services. The Symphony environment contains the host factory and provider scripts. This architecture uses the Cloud de Confiance by S3NS host factory provider CLI tools layer to translate requests, allowing the Symphony scripts to control Compute Engine resources and a Kubernetes operator running in GKE.
The following sections explain the main components of Symphony Connectors for Cloud de Confiance by S3NS.
IBM Spectrum Symphony host factory
An IBM Spectrum Symphony service that manages the lifecycle of compute hosts from various providers based on workload demand. This Symphony component is the software that Cloud de Confiance by S3NS integrates with.
Provider scripts
The Symphony host factory service uses a set of shell scripts to integrate with a cloud provider. These scripts perform actions like requesting and returning resources.
By default, Symphony doesn't include a Host factory provider for Cloud de Confiance by S3NS.
The Cloud de Confiance by S3NS host factory provider enables the Symphony host factory framework to communicate with Cloud de Confiance by S3NS.
The following table shows the scripts that Symphony Connectors for Cloud de Confiance by S3NS uses:
| Name | Description | IBM documentation |
|---|---|---|
getAvailableTemplates.sh |
Retrieves the available
resource templates defined in the provider configuration. Host factory
calls getAvailableTemplates.sh to know what types of
resources it can request. |
getAvailableTemplates |
requestMachines.sh |
Provisions new compute resources from Cloud de Confiance by S3NS. It receives a JSON payload specifying the template and quantity. | requestMachines |
requestReturnMachines.sh |
Host factory deprovisions compute resources used by the Symphony cluster. It receives a JSON payload that identifies the machines to be returned. | requestReturnMachines |
getRequestStatus.sh |
Host factory polls the status of ongoing provisioning requests
initiated by requestMachines.sh. It receives a JSON payload
that contains request IDs. |
getRequestStatus |
getReturnRequests.sh |
Host factory polls the
status of ongoing deprovisioning requests initiated by
requestReturnMachines.sh. It receives a JSON payload that
contains request IDs. | getReturnRequests |
Cloud de Confiance by S3NS host factory provider
The host factory provider is the interface created by Google that integrates with IBM Spectrum Symphony's host factory framework. This Cloud de Confiance by S3NS plugin includes CLI tools that Symphony calls to request and manage resources in Cloud de Confiance by S3NS. CLI tools for both Compute Engine and GKE workloads are available. Symphony Connectors for Cloud de Confiance by S3NS refers to these CLI tools as providers:
- For Compute Engine workloads, you use the Compute Engine provider.
- For GKE workloads, you use the GKE provider. For GKE workloads, you need an additional operator. This Kubernetes operator runs within your GKE cluster to manage the lifecycle of the IBM Spectrum Symphony compute pods requested using the GKE provider.
After you install these CLI tools, you can also manually call the Compute Engine and GKE provider CLI tools using Python. The CLI tools are Python modules, making them executable using the Python interpreter. For more information, see the Running from Python documentation on GitHub.
Compute Engine provider
The Compute Engine provider (hf-gce) is a CLI tool designed to
integrate Symphony's host factory with
Compute Engine. After you install the Compute Engine provider, you
must configure it in
Symphony host factory. Once this configuration is complete, host factory can
automatically invokes scripts that use the hf-gce CLI.
The Compute Engine provider interacts directly with the Google Compute Engine API to manage the lifecycle of VM instances based on Symphony's demands and the configurations defined in the provider instance and templates. It handles the creation, monitoring, and termination of Compute Engine VMs, allowing them to join the Symphony cluster as compute nodes.
GKE provider
The GKE provider (hf-gke) is a CLI tool that connects
Symphony's host factory service with GKE. Host factory uses
scripts that call the hf-gke CLI.
The GKE provider interacts with the
GKE cluster's API to create, query, and delete Custom Resource
Definitions (CRDs), specifically GCPSymphonyResource and
MachineReturnRequest.
The Kubernetes operator running within the GKE cluster handles the actual Pod lifecycle management based on these custom resources. The GKE provider acts as the bridge between Symphony's resource requests and the Kubernetes custom resource-based orchestration.
Cloud de Confiance by S3NS services
You can connect to Compute Engine and Google Kubernetes Engine. There are no additional components for Compute Engine connections. For Google Kubernetes Engine connections, there is a Kubernetes operator.
Kubernetes operator
You only need this additional operator for GKE workloads that you run using the GKE provider. This operator runs within your GKE cluster to manage the lifecycle of Symphony compute Pods requested using the GKE provider. The GKE provider has the following features:
- Asynchronous operations: supports high-performance asynchronous processing, suitable for large-scale deployments.
- Preemption handling: includes automatic detection and management of Spot VM preemptions.
- Custom resource definitions: uses
GCPSymphonyResourceandMachineReturnRequestcustom resources to manage resources. - Resource cleanup: automates cleanup of completed resources, with configurable retention periods.
- Health monitoring: monitors the integration of the GKE provider with built-in health checks and status reporting.
Limitations
Symphony Connectors for Cloud de Confiance by S3NS has the following limitations:
- Cloud de Confiance by S3NS does not create, manage, schedule, or execute your IBM Spectrum Symphony workloads. You are responsible for all workload management within the Symphony software.
What's next
Visit the GitHub documentation to install and configure the Cloud de Confiance by S3NS host factory provider: