Advanced Workflow Patterns in Kontroler

After mastering the basics of Kontroler, you’ll likely want to implement more sophisticated workflow patterns to address complex business processes. This guide explores advanced patterns you can implement with Kontroler to make your workflows more powerful, resilient, and flexible.

Parallel Execution

One of Kontroler’s key strengths is its ability to efficiently execute steps in parallel, significantly reducing total execution time for suitable workloads.

Fan-Out Pattern

The fan-out pattern distributes tasks across multiple parallel workers:

apiVersion: kontroler.greedykomodo/v1alpha1
kind: DAG
metadata:
  name: fan-out-example
spec:
  task:
    - name: prepare-data
      image: data-processor:v1
      command: ["/scripts/prepare.sh"]
    
    # Fan-out to multiple parallel workers
    - name: process-shard-1
      image: data-processor:v1
      command: ["/scripts/process.sh", "--shard=1"]
      runAfter:
        - prepare-data
    
    - name: process-shard-2
      image: data-processor:v1
      command: ["/scripts/process.sh", "--shard=2"]
      runAfter:
        - prepare-data
    
    - name: process-shard-3
      image: data-processor:v1
      command: ["/scripts/process.sh", "--shard=3"]
      runAfter:
        - prepare-data
    
    # Fan-in to aggregate results
    - name: aggregate-results
      image: data-processor:v1
      command: ["/scripts/aggregate.sh"]
      runAfter:
        - process-shard-1
        - process-shard-2
        - process-shard-3

Workspaces: Sharing Data Between Tasks

One of the most powerful features of Kontroler is the ability to share persistent storage between tasks using workspaces. This enables data sharing and state persistence throughout your workflow.

Configuring a Workspace

Enable a workspace by adding the workspace configuration to your DAG:

apiVersion: kontroler.greedykomodo/v1alpha1
kind: DAG
metadata:
  name: workspace-example
spec:
  workspace:
    enable: true
    pvc:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
      storageClassName: standard
      volumeMode: Filesystem
  task:
    - name: "write-data"
      command: ["sh", "-c"]
      args: ["echo 'Hello from task 1' > /workspace/message.txt"]
    
    - name: "read-data"
      command: ["sh", "-c"]
      args: ["cat /workspace/message.txt"]
      runAfter: ["write-data"]

How Workspaces Work

1. PVC Creation: Kontroler automatically creates a PersistentVolumeClaim (PVC) based on your configuration
2. Mounting: Each task pod mounts the PVC at /workspace
3. Lifecycle: The PVC persists throughout the entire DAG execution
4. Sharing: All tasks can read from and write to the shared space

Custom Pod Templates

For advanced Kubernetes configurations, Kontroler supports customizing pods using podTemplate:

apiVersion: kontroler.greedykomodo/v1alpha1
kind: DAG
metadata:
  name: custom-pod-example
spec:
  task:
    - name: "advanced-job"
      image: "your-image:tag"
      podTemplate:
        volumes:
          - name: example-pvc
            persistentVolumeClaim:
              claimName: example-claim
        volumeMounts:
          - name: example-pvc
            mountPath: /data
        imagePullSecrets:
          - name: my-registry-secret
        securityContext:
          runAsUser: 1000
          runAsGroup: 3000
          fsGroup: 2000
        nodeSelector:
          disktype: ssd
        tolerations:
          - key: "key1"
            operator: "Equal"
            value: "value1"
            effect: "NoSchedule"

Pod Templates give you control over:

• Custom volume mounts
• Private registry authentication
• Security contexts
• Node selection and affinity
• Tolerations for special nodes
• Custom service accounts

Error Handling and Recovery

Robust error handling is critical for production workflows.

Retry Strategies

Configure retries with backoff:

apiVersion: kontroler.greedykomodo/v1alpha1
kind: DAG
metadata:
  name: retry-example
spec:
  task:
    - name: api-call
      backoff:
        limit: 5
      image: api-client:v1
      command: ["/bin/call-api", "--endpoint=https://api.example.com/data"]

Resource Management

Control resource consumption for efficient cluster utilization by defining specific resource requirements for tasks:

apiVersion: kontroler.greedykomodo/v1alpha1
kind: DAG
metadata:
  name: resource-aware
spec:
  task:
    - name: data-analysis
      image: data-analyzer:v1
      command: ["/bin/analyze", "--data=/input/large-dataset.csv"]
      resources:
        requests:
          memory: "4Gi"
          cpu: "2"
        limits:
          memory: "8Gi"
          cpu: "4"

Conclusion

These advanced patterns illustrate the power and flexibility of Kontroler for orchestrating complex workflows. By combining features like workspaces for data sharing, custom pod templates for Kubernetes integration, and sophisticated error handling, you can build robust workflow solutions that meet demanding business requirements.

The strength of Kontroler lies in how these capabilities can be combined to solve complex real-world orchestration challenges. As you become more familiar with these techniques, you’ll be able to design elegant workflow architectures that are both powerful and maintainable.

For more detailed documentation on these features, check our documentation for:

• Workspaces and data sharing
• Pod template configuration
• Resource management