Orchestrating AI Workflows with SAP AI Core

In the previous posts, we learned how to work with individual foundation models. But real-world AI applications often need more than simple prompts - they require chaining multiple operations, managing templates, filtering content, and coordinating complex workflows.

SAP AI Core's Orchestration Service solves this by providing a unified way to combine AI capabilities into sophisticated pipelines.

This post is part of a series about building AI-powered applications with SAP BTP:

Getting Started with SAP AI Core and the SAP AI SDK in CAP
Leveraging LLM Models and Deployments in SAP AI Core
Orchestrating AI Workflows with SAP AI Core (this post)
Document Grounding with RAG in SAP AI Core (coming soon)
Production-Ready AI Applications with SAP AI Core (coming soon)

What are we building?

In this post, we'll enhance our Support Ticket Intelligence System by:

Understanding the orchestration service architecture
Building multi-step AI workflows (classify → analyze → respond)
Implementing content filtering and guardrails
Using templates for consistent prompt management
Adding data masking for sensitive information

By the end, you'll be able to create sophisticated AI pipelines that combine multiple capabilities safely and efficiently.

What is the Orchestration Service?

The orchestration service is a layer on top of AI Core that lets you:

Feature	Description
LLM Module	Call foundation models with configurable parameters
Templating	Define reusable prompt templates with variables
Content Filtering	Block harmful or inappropriate content
Data Masking	Anonymize sensitive data before sending to LLMs
Grounding	Connect to knowledge bases (covered in next post)

The key benefit is that you define a pipeline configuration once, and the orchestration service handles the execution, filtering, and coordination automatically.

Prerequisites

Ensure you have:

Completed the setup from Part 1
A running AI Core deployment
The orchestration SDK installed:

Copy

npm install @sap-ai-sdk/orchestration

Step 1: Understanding Orchestration Architecture

The orchestration service works differently from direct LLM calls. Instead of calling a model directly, you:

Define a configuration specifying the modules to use
Send a request with your input data
The service executes the pipeline and returns results

Here's a simplified flow:

Copy

Input → [Content Filter] → [Template] → [LLM] → [Content Filter] → Output
              ↓                               ↓
        Block harmful              Block harmful
        input content             output content

Step 2: Basic Orchestration Setup

Let's create an orchestration client. Create /srv/lib/orchestration-client.js:

Copy

const { OrchestrationClient } = require('@sap-ai-sdk/orchestration');

/**
 * Client for SAP AI Core Orchestration Service
 */
class TicketOrchestrationClient {
  
  constructor() {
    // The client auto-discovers credentials from bound services or destinations
    this.client = new OrchestrationClient();
  }

  /**
   * Simple completion with orchestration
   */
  async complete(systemPrompt, userPrompt, options = {}) {
    const config = {
      llm: {
        model_name: options.model || 'gpt-4o',
        model_params: {
          max_tokens: options.maxTokens || 1000,
          temperature: options.temperature || 0.7
        }
      }
    };

    const response = await this.client.chatCompletion({
      orchestration_config: config,
      messages: [
        { role: 'system', content: systemPrompt },
        { role: 'user', content: userPrompt }
      ]
    });

    return {
      content: response.getContent(),
      usage: response.getTokenUsage()
    };
  }
}

module.exports = TicketOrchestrationClient;

The orchestration client provides a higher-level abstraction than the direct model clients, allowing you to add modules incrementally.

Step 3: Adding Prompt Templates

One powerful feature is templating - define prompts with placeholders that get filled at runtime.

Update the orchestration client:

Copy

const { OrchestrationClient } = require('@sap-ai-sdk/orchestration');

class TicketOrchestrationClient {
  
  constructor() {
    this.client = new OrchestrationClient();
  }

  /**
   * Generate ticket response using templates
   */
  async generateTicketResponse(ticket) {
    const config = {
      llm: {
        model_name: 'gpt-4o',
        model_params: {
          max_tokens: 800,
          temperature: 0.7
        }
      },
      // Define the template
      templating: {
        template: [
          {
            role: 'system',
            content: `You are a customer support agent for {{company_name}}.
            
Your guidelines:
- Be professional and empathetic
- Provide clear, actionable solutions
- Address the customer by name when known
- Keep responses concise but complete`
          },
          {
            role: 'user',
            content: `Please draft a response for this support ticket:

**Subject:** {{ticket_subject}}
**Priority:** {{ticket_priority}}
**Category:** {{ticket_category}}

**Customer Message:**
{{ticket_description}}

Generate a helpful response.`
          }
        ]
      }
    };

    // Provide template values
    const templateValues = {
      company_name: 'TechCorp Solutions',
      ticket_subject: ticket.subject,
      ticket_priority: ticket.priority || 'Medium',
      ticket_category: ticket.category || 'General',
      ticket_description: ticket.description
    };

    const response = await this.client.chatCompletion({
      orchestration_config: config,
      template_values: templateValues
    });

    return {
      content: response.getContent(),
      usage: response.getTokenUsage()
    };
  }
}

module.exports = TicketOrchestrationClient;

The template uses {{variable_name}} syntax for placeholders. This approach has several benefits:

Reusability: Define templates once, use with different data
Separation of concerns: Templates can be managed separately from code
Consistency: Ensure all responses follow the same structure
Testing: Easier to test with mock template values

Step 4: Implementing Content Filtering

Content filtering is crucial for production applications. It prevents harmful content from being sent to or received from the LLM.

Copy

const { OrchestrationClient } = require('@sap-ai-sdk/orchestration');

class TicketOrchestrationClient {
  
  constructor() {
    this.client = new OrchestrationClient();
  }

  /**
   * Generate response with content filtering
   */
  async generateSafeResponse(ticket) {
    const config = {
      llm: {
        model_name: 'gpt-4o',
        model_params: {
          max_tokens: 800,
          temperature: 0.7
        }
      },
      templating: {
        template: [
          {
            role: 'system',
            content: `You are a professional customer support agent.
Provide helpful, appropriate responses to customer inquiries.`
          },
          {
            role: 'user',
            content: `Respond to this ticket:
Subject: {{subject}}
Description: {{description}}`
          }
        ]
      },
      // Add content filtering
      filtering: {
        // Filter input before sending to LLM
        input: {
          filters: [
            {
              type: 'azure_content_safety',
              config: {
                Hate: 2,      // 0-6 scale, lower = stricter
                Violence: 2,
                Sexual: 2,
                SelfHarm: 2
              }
            }
          ]
        },
        // Filter output from LLM
        output: {
          filters: [
            {
              type: 'azure_content_safety',
              config: {
                Hate: 2,
                Violence: 2,
                Sexual: 2,
                SelfHarm: 2
              }
            }
          ]
        }
      }
    };

    try {
      const response = await this.client.chatCompletion({
        orchestration_config: config,
        template_values: {
          subject: ticket.subject,
          description: ticket.description
        }
      });

      return {
        content: response.getContent(),
        usage: response.getTokenUsage(),
        filtered: false
      };
    } catch (error) {
      // Content was filtered
      if (error.message?.includes('content_filter')) {
        console.warn('Content was filtered:', error.message);
        return {
          content: null,
          filtered: true,
          filterReason: error.message
        };
      }
      throw error;
    }
  }
}

The Azure Content Safety filter checks for:

Category	What it Detects
Hate	Discriminatory or hateful content
Violence	Violent content or threats
Sexual	Sexual or adult content
SelfHarm	Content promoting self-harm

The scale is 0-6 where lower numbers are stricter. A setting of 2 is moderately strict.

Step 5: Adding Data Masking

When processing tickets, you might encounter sensitive data like emails, phone numbers, or credit card numbers. Data masking anonymizes this before sending to the LLM.

Copy

const { OrchestrationClient } = require('@sap-ai-sdk/orchestration');

class TicketOrchestrationClient {
  
  constructor() {
    this.client = new OrchestrationClient();
  }

  /**
   * Generate response with data masking for privacy
   */
  async generatePrivacyAwareResponse(ticket) {
    const config = {
      llm: {
        model_name: 'gpt-4o',
        model_params: {
          max_tokens: 800,
          temperature: 0.7
        }
      },
      templating: {
        template: [
          {
            role: 'system',
            content: 'You are a customer support agent. Help the customer with their issue.'
          },
          {
            role: 'user',
            content: `Customer ticket:
{{description}}

Provide a helpful response.`
          }
        ]
      },
      // Add data masking
      masking: {
        masking_providers: [
          {
            type: 'sap_data_privacy_integration',
            method: 'pseudonymization',
            entities: [
              { type: 'profile-email' },
              { type: 'profile-phone' },
              { type: 'profile-person' },
              { type: 'profile-org' },
              { type: 'profile-location' }
            ]
          }
        ]
      }
    };

    const response = await this.client.chatCompletion({
      orchestration_config: config,
      template_values: {
        description: ticket.description
      }
    });

    // The response will have masked entities replaced back
    return {
      content: response.getContent(),
      usage: response.getTokenUsage()
    };
  }
}

With data masking enabled:

Input: "Contact me at john.doe@email.com or 555-123-4567"
Sent to LLM: "Contact me at [EMAIL_1] or [PHONE_1]"
LLM Response: "I'll reach out to you at [EMAIL_1]..."
Final Output: "I'll reach out to you at john.doe@email.com..."

The masking is transparent - your code works with real data, but the LLM only sees anonymized versions.

Step 6: Building Multi-Step Workflows

Now let's combine everything into a sophisticated ticket processing pipeline.

Create /srv/lib/ticket-pipeline.js:

Copy

const { OrchestrationClient } = require('@sap-ai-sdk/orchestration');

/**
 * Multi-step ticket processing pipeline
 */
class TicketProcessingPipeline {
  
  constructor() {
    this.client = new OrchestrationClient();
  }

  /**
   * Process a ticket through the complete pipeline:
   * 1. Classify the ticket
   * 2. Analyze sentiment and urgency
   * 3. Generate appropriate response
   */
  async processTicket(ticket) {
    console.log(`Processing ticket: ${ticket.ID}`);
    
    // Step 1: Classification
    const classification = await this._classifyTicket(ticket);
    console.log('Classification:', classification);
    
    // Step 2: Sentiment Analysis
    const sentiment = await this._analyzeSentiment(ticket);
    console.log('Sentiment:', sentiment);
    
    // Step 3: Generate Response (using classification and sentiment context)
    const response = await this._generateContextualResponse(
      ticket,
      classification,
      sentiment
    );
    
    return {
      ticketId: ticket.ID,
      classification,
      sentiment,
      suggestedResponse: response.content,
      tokenUsage: response.usage
    };
  }

  /**
   * Step 1: Classify the ticket
   */
  async _classifyTicket(ticket) {
    const config = {
      llm: {
        model_name: 'gpt-4o-mini',  // Fast model for classification
        model_params: {
          max_tokens: 150,
          temperature: 0.1  // Low temperature for consistency
        }
      },
      templating: {
        template: [
          {
            role: 'system',
            content: `Classify support tickets into categories. 
Respond ONLY with JSON: {"category": string, "priority": string, "confidence": number}

Categories: Technical Issue, Billing Question, Feature Request, Account Access, Bug Report, General Inquiry
Priorities: Critical, High, Medium, Low`
          },
          {
            role: 'user',
            content: 'Subject: {{subject}}\nDescription: {{description}}'
          }
        ]
      }
    };

    const response = await this.client.chatCompletion({
      orchestration_config: config,
      template_values: {
        subject: ticket.subject,
        description: ticket.description
      }
    });

    try {
      return JSON.parse(response.getContent());
    } catch {
      return { category: 'General Inquiry', priority: 'Medium', confidence: 0.5 };
    }
  }

  /**
   * Step 2: Analyze sentiment
   */
  async _analyzeSentiment(ticket) {
    const config = {
      llm: {
        model_name: 'gpt-4o-mini',
        model_params: {
          max_tokens: 100,
          temperature: 0.1
        }
      },
      templating: {
        template: [
          {
            role: 'system',
            content: `Analyze customer sentiment. 
Respond ONLY with JSON: {"sentiment": string, "urgency": string, "keywords": string[]}

Sentiments: Frustrated, Neutral, Positive
Urgency: Immediate, Soon, Normal`
          },
          {
            role: 'user',
            content: '{{description}}'
          }
        ]
      }
    };

    const response = await this.client.chatCompletion({
      orchestration_config: config,
      template_values: {
        description: ticket.description
      }
    });

    try {
      return JSON.parse(response.getContent());
    } catch {
      return { sentiment: 'Neutral', urgency: 'Normal', keywords: [] };
    }
  }

  /**
   * Step 3: Generate contextual response
   */
  async _generateContextualResponse(ticket, classification, sentiment) {
    // Adjust tone based on sentiment
    let toneGuideline = 'Be professional and helpful.';
    if (sentiment.sentiment === 'Frustrated') {
      toneGuideline = 'Be extra empathetic. Acknowledge their frustration. Apologize for any inconvenience.';
    } else if (sentiment.sentiment === 'Positive') {
      toneGuideline = 'Match their positive energy. Thank them for their patience/feedback.';
    }

    // Adjust urgency acknowledgment
    let urgencyNote = '';
    if (sentiment.urgency === 'Immediate' || classification.priority === 'Critical') {
      urgencyNote = 'Acknowledge the urgency and prioritize their issue.';
    }

    const config = {
      llm: {
        model_name: 'gpt-4o',  // Better model for response quality
        model_params: {
          max_tokens: 600,
          temperature: 0.7
        }
      },
      templating: {
        template: [
          {
            role: 'system',
            content: `You are a customer support agent.

Ticket Context:
- Category: {{category}}
- Priority: {{priority}}
- Customer Sentiment: {{sentiment}}

Tone Guidelines:
{{tone_guideline}}
{{urgency_note}}

Provide a helpful, complete response.`
          },
          {
            role: 'user',
            content: `Subject: {{subject}}

Customer Message:
{{description}}

Draft a response addressing their concern.`
          }
        ]
      },
      // Add content filtering for safety
      filtering: {
        input: {
          filters: [{ type: 'azure_content_safety', config: { Hate: 2, Violence: 2, Sexual: 2, SelfHarm: 2 } }]
        },
        output: {
          filters: [{ type: 'azure_content_safety', config: { Hate: 2, Violence: 2, Sexual: 2, SelfHarm: 2 } }]
        }
      }
    };

    const response = await this.client.chatCompletion({
      orchestration_config: config,
      template_values: {
        category: classification.category,
        priority: classification.priority,
        sentiment: sentiment.sentiment,
        tone_guideline: toneGuideline,
        urgency_note: urgencyNote,
        subject: ticket.subject,
        description: ticket.description
      }
    });

    return {
      content: response.getContent(),
      usage: response.getTokenUsage()
    };
  }
}

module.exports = TicketProcessingPipeline;

Step 7: Integrate Pipeline into the Service

Update /srv/ticket-service.js to use the pipeline:

Copy

const cds = require('@sap/cds');
const TicketProcessingPipeline = require('./lib/ticket-pipeline');

module.exports = class TicketService extends cds.ApplicationService {
  
  async init() {
    const { Tickets } = this.entities;
    const pipeline = new TicketProcessingPipeline();
    
    // Full pipeline processing
    this.on('processTicket', async (req) => {
      const { ticketId } = req.data;
      
      const ticket = await SELECT.one.from(Tickets).where({ ID: ticketId });
      if (!ticket) {
        return req.error(404, `Ticket ${ticketId} not found`);
      }
      
      try {
        const result = await pipeline.processTicket(ticket);
        
        // Update ticket with results
        await UPDATE(Tickets)
          .set({
            category: result.classification.category,
            priority: result.classification.priority,
            sentiment: result.sentiment.sentiment,
            aiResponse: result.suggestedResponse
          })
          .where({ ID: ticketId });
        
        return result;
      } catch (error) {
        console.error('Pipeline error:', error);
        return req.error(500, 'Failed to process ticket');
      }
    });
    
    // Auto-process new tickets
    this.after('CREATE', 'Tickets', async (ticket) => {
      setImmediate(async () => {
        try {
          const result = await pipeline.processTicket(ticket);
          
          await UPDATE(Tickets)
            .set({
              category: result.classification.category,
              priority: result.classification.priority,
              sentiment: result.sentiment.sentiment,
              aiResponse: result.suggestedResponse
            })
            .where({ ID: ticket.ID });
          
          console.log(`Auto-processed ticket ${ticket.ID}`);
        } catch (error) {
          console.error(`Auto-processing failed for ${ticket.ID}:`, error);
        }
      });
    });
    
    await super.init();
  }
};

Update the service definition /srv/ticket-service.cds:

Copy

using support.db as db from '../db/schema';

service TicketService @(path: '/api') {
  entity Tickets as projection on db.Tickets;
  
  // Full pipeline processing
  action processTicket(ticketId: UUID) returns {
    ticketId: UUID;
    classification: {
      category: String;
      priority: String;
      confidence: Double;
    };
    sentiment: {
      sentiment: String;
      urgency: String;
      keywords: array of String;
    };
    suggestedResponse: String;
  };
}

Step 8: Error Handling and Resilience

Production pipelines need robust error handling. Here's an enhanced version:

Copy

class ResilientPipeline extends TicketProcessingPipeline {
  
  constructor() {
    super();
    this.maxRetries = 3;
    this.retryDelay = 1000;
  }

  /**
   * Process with retry logic
   */
  async processTicketWithRetry(ticket) {
    let lastError;
    
    for (let attempt = 1; attempt <= this.maxRetries; attempt++) {
      try {
        return await this.processTicket(ticket);
      } catch (error) {
        lastError = error;
        console.warn(`Attempt ${attempt} failed:`, error.message);
        
        // Don't retry on content filter errors
        if (error.message?.includes('content_filter')) {
          throw error;
        }
        
        if (attempt < this.maxRetries) {
          await this._delay(this.retryDelay * attempt);
        }
      }
    }
    
    throw lastError;
  }

  /**
   * Process with fallback
   */
  async processTicketSafe(ticket) {
    try {
      return await this.processTicketWithRetry(ticket);
    } catch (error) {
      console.error('Pipeline failed, using fallback:', error.message);
      
      // Return basic fallback
      return {
        ticketId: ticket.ID,
        classification: {
          category: 'General Inquiry',
          priority: 'Medium',
          confidence: 0
        },
        sentiment: {
          sentiment: 'Neutral',
          urgency: 'Normal',
          keywords: []
        },
        suggestedResponse: 'Thank you for contacting support. A team member will review your ticket shortly.',
        fallbackUsed: true
      };
    }
  }

  _delay(ms) {
    return new Promise(resolve => setTimeout(resolve, ms));
  }
}

Step 9: Testing the Orchestration Pipeline

Create comprehensive tests in /test/requests.http:

Copy

### Create a frustrated customer ticket
POST http://localhost:4004/api/Tickets
Content-Type: application/json

{
  "subject": "URGENT: System down for 3 hours!",
  "description": "This is absolutely unacceptable! Our production system has been down for 3 hours and we're losing thousands of dollars every minute. I've called support 5 times and nobody can help. I need this fixed NOW or we're canceling our contract. My phone is 555-123-4567 and email is john.smith@company.com"
}

### Create a billing inquiry
POST http://localhost:4004/api/Tickets
Content-Type: application/json

{
  "subject": "Question about my invoice",
  "description": "Hi, I noticed a charge on my latest invoice that I don't recognize. Could you help me understand what the 'Premium Add-on Service' charge of $49.99 is for? I don't remember signing up for any add-ons. Thanks!"
}

### Create a feature request (positive)
POST http://localhost:4004/api/Tickets
Content-Type: application/json

{
  "subject": "Love the product! Feature suggestion",
  "description": "First, I want to say that your product has been amazing for our team's productivity! We've reduced our processing time by 50%. I had a small suggestion - would it be possible to add dark mode? It would be easier on the eyes during late work sessions. Keep up the great work!"
}

### Process a ticket through the full pipeline
POST http://localhost:4004/api/processTicket
Content-Type: application/json

{
  "ticketId": "YOUR-TICKET-ID"
}

### Get processed ticket
GET http://localhost:4004/api/Tickets(YOUR-TICKET-ID)

Monitoring and Debugging

Add logging to track pipeline performance:

Copy

class InstrumentedPipeline extends TicketProcessingPipeline {
  
  async processTicket(ticket) {
    const startTime = Date.now();
    const metrics = {
      ticketId: ticket.ID,
      steps: {}
    };
    
    try {
      // Classification
      let stepStart = Date.now();
      const classification = await this._classifyTicket(ticket);
      metrics.steps.classification = {
        duration: Date.now() - stepStart,
        result: classification
      };
      
      // Sentiment
      stepStart = Date.now();
      const sentiment = await this._analyzeSentiment(ticket);
      metrics.steps.sentiment = {
        duration: Date.now() - stepStart,
        result: sentiment
      };
      
      // Response
      stepStart = Date.now();
      const response = await this._generateContextualResponse(ticket, classification, sentiment);
      metrics.steps.response = {
        duration: Date.now() - stepStart,
        tokenUsage: response.usage
      };
      
      metrics.totalDuration = Date.now() - startTime;
      metrics.success = true;
      
      console.log('Pipeline metrics:', JSON.stringify(metrics, null, 2));
      
      return {
        ticketId: ticket.ID,
        classification,
        sentiment,
        suggestedResponse: response.content,
        tokenUsage: response.usage,
        metrics
      };
    } catch (error) {
      metrics.totalDuration = Date.now() - startTime;
      metrics.success = false;
      metrics.error = error.message;
      
      console.error('Pipeline failed:', JSON.stringify(metrics, null, 2));
      throw error;
    }
  }
}

Recap

In this post, we've built sophisticated AI workflows using SAP AI Core's orchestration service:

Orchestration basics: Understanding the service architecture and client setup
Templating: Creating reusable prompt templates with variables
Content filtering: Adding safety guardrails to prevent harmful content
Data masking: Protecting sensitive information like emails and phone numbers
Multi-step pipelines: Chaining classification, sentiment analysis, and response generation
Error handling: Building resilient pipelines with retries and fallbacks

Our Support Ticket System now processes tickets through a complete AI pipeline that:

Classifies tickets automatically
Detects customer sentiment and urgency
Generates contextually appropriate responses
Filters inappropriate content
Protects customer privacy

Next Steps

In the next post, Document Grounding with RAG in SAP AI Core, we'll learn how to:

Implement RAG (Retrieval-Augmented Generation)
Connect AI responses to your knowledge base
Upload and index company documents
Generate responses grounded in actual documentation

Stay tuned!

Orchestrating AI Workflows with SAP AI Core

What are we building?

What is the Orchestration Service?

Prerequisites

Step 1: Understanding Orchestration Architecture

Step 2: Basic Orchestration Setup

Step 3: Adding Prompt Templates

Step 4: Implementing Content Filtering

Step 5: Adding Data Masking

Step 6: Building Multi-Step Workflows

Step 7: Integrate Pipeline into the Service

Step 8: Error Handling and Resilience

Step 9: Testing the Orchestration Pipeline

Monitoring and Debugging

Recap

Next Steps

Resources

Published

Topics

Content