Our first batch of recipes have arrived! These recipes include step-by-step instructions for implementing the LLM Open Connector (readme, spec) with an API gateway, and each recipe features a different AI platform. More recipes are in the works featuring new AI platforms and system architectures!

Your contributions to the Cookbook and the open-source repository are welcome! Refer to the Contributing Guide and Code of Conduct to get started.

If you like the resources that you see here, consider adding a ⭐ on GitHub. It helps other people discover them!

See Also:

• Generative AI Developer Guide
• Einstein Platform GitHub Repo

The LLM Open Connector opens up a world of possibilities for bespoke AI applications for Salesforce customers and independent software vendors (ISVs). Check out our recipes to learn how to bring new models to Einstein Studio!

Prerequisites​

Before you begin, make sure that your local environment meets these prerequisites:

1. Node.js and npm are installed on your local machine.
2. You have a Heroku account. (Sign up at https://signup.heroku.com/.)
3. Heroku CLI is installed on your local machine (https://devcenter.heroku.com/articles/heroku-cli).
4. Git is installed on your local machine.
5. You have a Hugging Face account with an API key (https://huggingface.co/docs/hub/en/security-tokens).

Tutorial Video​

View a walkthrough on using any LLM with the Open Connector on the Salesforce Developers channel:

Set Up Your Local Environment​

1. Clone the einstein-platform repository to your local environment.

    git clone https://github.com/salesforce/einstein-platform.git
   

2. Create a directory for your project:

   mkdir llm-open-connector-hf
   cd llm-open-connector-hf
   

3. Locate the source files specific to Hugging Face connector. In the local einstein-platform project directory you cloned in step 1, navigate to the documentation/cookbook-assets/llm-open-connector-huggingface folder. Copy all the files in this folder.

4. Paste the llm-open-connector-huggingface files into the llm-open-connector-hf directory you created in step 2.

5. Install the required dependencies.

   npm install
   

Project Structure​

This section outlines the folder structure of the llm-open-connector-hf directory.

 llm-open-connector-hf/
 ├── config/
 │   └── index.js
 ├── controllers/
 │   └── chatController.js
 ├── middleware/
 │   └── index.js
 ├── routes/
 │   └── chat.js
 ├── utils/
 │   └── logger.js
 ├── .gitignore
 ├── .prettierrc
 ├── index.js
 ├── package.json
 └── Procfile

Project Description​

The following list provides a functional overview of each folder and file in the llm-open-connector-hf directory.

• config/: Holds configuration settings.
  • index.js: Exports configuration options for different environments.
• controllers/: Manages the request handling logic for the chat API.
  • chatController.js: Calls the chat completion API.
• middleware/: Contains custom middleware functions.
  • index.js: Includes functions for API key validation and error handling.
• routes/: Defines API routes.
  • chat.js: Contains endpoints for chat completion functionality.
• utils/: Utility functions for support tasks.
  • logger.js: A custom logger with data sanitization for secure logging.
• .gitignore: Lists files and directories that are ignored by Git, such as node_modules and sensitive files.
• .prettierrc: The configuration file for Prettier, which enforces consistent code formatting across the project.
• index.js: The main application file that initializes the server and routes.
• package.json: Contains metadata for the project and manages dependencies, scripts, and configurations.
• Procfile: Specifies process types and commands for deploying the app on platforms like Heroku.

Configure Your Local Environment​

1. Create a .env file in the root directory of the project and add these variables:

   PORT=3000
   HUGGING_FACE_API_KEY=your_hugging_face_api_key
   HUGGING_FACE_API_URL=https://api-inference.huggingface.co/models
   USE_THIRD_PARTY_ROUTER=false
   ALLOWED_ORIGINS=http://localhost:3000,https://your-salesforce-domain.com
   

   Replace your_hugging_face_api_key with your own Hugging Face API key and adjust the ALLOWED_ORIGINS as needed. Change ALLOWED_ORIGINS to allowlist your Salesforce domain.

   note

   The HUGGING_FACE_API_URL and USE_THIRD_PARTY_ROUTER configuration options are important when using third-party routers like Together AI.

   • For standard Hugging Face models, use HUGGING_FACE_API_URL=https://api-inference.huggingface.co and USE_THIRD_PARTY_ROUTER=false.
   • For third-party routers (e.g. Together AI), use HUGGING_FACE_API_URL=https://router.huggingface.co/together/v1/chat/completions and USE_THIRD_PARTY_ROUTER=true.

2. Make sure your .gitignore file includes the .env file to avoid accidentally committing sensitive information.

Test Your Application Locally​

1. Run your Node.js application locally:

   npm start
   

2. The server starts on the port specified in your .env file. The default port is 3000.

3. Test the endpoints using a tool like cURL or Postman to ensure they're working correctly. Here are two examples:

   a. Using a standard Hugging Face model (Mixtral):

   curl -X POST http://localhost:3000/chat/completions \
   -H 'Content-Type: application/json' \
   -H 'api-key: <use your HuggingFace access token here>' \
   -d '{
     "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
     "messages": [
       {
         "content": "What is the capital of Italy?",
         "role": "user"
       }
     ],
     "max_tokens": 100,
     "temperature": 0.7,
     "n": 2,
     "parameters": {
       "top_p": 0.9
     }
   }'
   

   b. Using Qwen through Together AI (requires setting HUGGING_FACE_API_URL=https://router.huggingface.co/together/v1/chat/completions and USE_THIRD_PARTY_ROUTER=true):

   curl -X POST http://localhost:3000/chat/completions \
   -H 'Content-Type: application/json' \
   -H 'api-key: <use your Together AI access token here>' \
   -d '{
     "model": "Qwen/Qwen2.5-7B-Instruct-Turbo",
     "messages": [
       {
         "content": "What is the capital of Italy?",
         "role": "user"
       }
     ],
     "max_tokens": 100,
     "temperature": 0.7,
     "n": 2,
     "parameters": {
       "top_p": 0.9
     }
   }'
   

Prepare for Heroku Deployment​

1. Initialize a Git repository in your project directory:

   git init
   

2. Add your files to the repository:

   git add .
   

3. Commit your changes:

   git commit -m "Initial commit"
   

Update Your Default Branch​

To switch the default branch used to deploy apps from master to main, follow these steps:

1. Create a branch locally:

   git checkout -b main
   

2. Delete the old default branch locally:

   git branch -D master
   

   Now, the local environment only knows about the main branch.

3. Reset the git repository on the Heroku Platform:

   • Use the heroku-reset command from the heroku-repo CLI plugin. This command does not impact the running application.

   important

   Communicate this change with your team. If other developers are unaware of the reset and push to master, the reset will be overwritten.

4. To switch the default branch in GitHub, refer to this article: Setting the Default Branch.

Deploy to Heroku​

1. Login to the Heroku CLI:

   heroku login
   

2. Create a Heroku app:

   heroku create your-app-name
   

   Replace your-app-name with a unique name for your application.

3. Set the API_KEY config var on Heroku:

   heroku config:set HUGGING_FACE_API_KEY=your_api_key_here -a your-app-name
   

   Replace your_api_key_here with your own API key.

4. Set the ALLOWED_ORIGINS config var on Heroku:

   heroku config:set ALLOWED_ORIGINS=your_salesforce_domain_here -a your-app-name
   

   Replace your_salesforce_domain_here with your own Salesforce domain.

5. Deploy your app to Heroku:

   git push heroku main
   

Test Your Deployed Application​

You can test your deployed application using the deployed API endpoint. Use a tool like cURL or Postman to test the endpoints of your Node.js app:

• Chat Completions: POST https://your-app-name.herokuapp.com/chat/completions

To test the chat/completions endpoint with cURL, run this command:

curl -X POST https://still-beach-80840-58cbb07ae5f4.herokuapp.com/chat/completions \
-H 'Content-Type: application/json' \
-H 'api-key: <use your HuggingFace access token here>' \
-d '{
  "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
  "messages": [
    {
      "content": "What is the capital of Italy?",
      "role": "user"
    }
  ],
  "max_tokens": 100,
  "temperature": 0.7,
  "n": 2,
  "parameters": {
    "top_p": 0.9
  }
}'

Bring Your Connected Endpoint to Salesforce Model Builder​

Follow the instructions in this developer blog to use your model in Model Builder. When you activate your model, you can use it within Prompt Builder, the Models API, and for building actions using prompt templates in Agent Builder. All these methods provide built-in security offered by the Einstein Trust Layer.

Current Features​

The following list outlines the features included in this recipe:

• Integration with the Hugging Face Serverless Inference API for models that support Chat Completion
• Express server with advanced security configurations
• CORS configuration with customizable allowed origins
• Rate limiting to prevent abuse
• API key validation for protected routes
• Comprehensive error handling and sanitized logging
• Helmet.js integration for enhanced security headers
• Chat completion controller with input validation and response reshaping
• Optimized message processing:
  • Concatenates multiple system messages into a single message as required by some LLMs
  • Preserves the order of user and assistant messages

API Endpoints​

• POST /chat/completions: Send a chat message and receive an AI-generated response.
  • The endpoint optimizes message processing by concatenating system messages.
  • The following example provides two system messages:

    {
      "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "system", "content": "Always be polite."},
        {"role": "user", "content": "Hello!"},
        {"role": "assistant", "content": "Hi there!"},
        {"role": "user", "content": "How are you?"}
      ],
      "model": "gpt-3.5-turbo",
      "max_tokens": 150
    }
    
  • The API processes the system messages into:

    {
      "messages": [
        {"role": "system", "content": "You are a helpful assistant.\nAlways be polite."},
        {"role": "user", "content": "Hello!"},
        {"role": "assistant", "content": "Hi there!"},
        {"role": "user", "content": "How are you?"}
      ],
      "model": "gpt-3.5-turbo",
      "max_tokens": 150
    }
    

Current Security Measures​

The recipe's security measures include a Helmet.js configuration, a CORS configuration, rate limiting, API key validation, and sanitized logging.

The following list briefly highlights the current security measures:

• A Helmet.js configuration with strict security settings:
  • Content Security Policy (CSP) with restrictive directives
  • Cross-Origin Embedder Policy
  • Cross-Origin Opener Policy
  • Cross-Origin Resource Policy
  • DNS Prefetch Control
  • Expect-CT header
  • Frameguard to prevent clickjacking
  • HTTP Strict Transport Security (HSTS)
  • IE No Open
  • X-Content-Type-Options nosniff
  • Origin-Agent-Cluster header
  • Permitted Cross-Domain Policies
  • Referrer-Policy
  • X-XSS-Protection
• A CORS configuration restricts allowed origins.
• Rate limits: 100 requests per 15 minutes per IP.
• API key validation provides protected routes.
• Sanitized logging prevents accidental exposure of sensitive data.

Important Considerations​

1. This cookbook uses the mistralai/Mixtral-8x7B-Instruct-v0.1 model in the example cURL requests. You can replace it with any supported model. For a list of supported model IDs, see Hugging Face Chat Completion.
2. Monitor your usage to manage costs, especially if you expect high traffic.
3. Error handling and input validation must be improved for production use.
4. Consider implementing further security measures for production use.

Conclusion​

This cookbook demonstrates how to set up an LLM Open Connector using Hugging Face Serverless API for Chat Completion endpoints for various models. For production purposes, remember to use Hugging Face's dedicated Interference Endpoints feature. Always optimize your implementation based on your specific requirements and expected usage patterns.

Prerequisites​

1. AWS Account with appropriate permissions
2. AWS CLI. For installation instructions, see Install or update to the latest version of the AWS CLI and Set up the AWS CLI.
3. Python 3.9 or later
4. AWS CDK installed (npm install -g aws-cdk)
5. Access to AWS Bedrock (you may need to request access if it's not enabled in your account)

Set Up AWS Bedrock and Model Access​

Before deploying and using the LLM Open Connector, you need to set up access to AWS Bedrock and enable the specific model you want to use. Follow these steps:

Enable AWS Bedrock:

• Go to the AWS Bedrock console.
• If you see a "Get started" button, click it to enable Bedrock for your account.
• If Bedrock is already enabled, you'll see the Bedrock dashboard.

Request access to models:

• In the Bedrock console, go to "Model access" in the left navigation pane.
• Find the model you want to use (for example, Llama 3.1 70B Instruct).
• Click the menu (three dots) and select "Request model access".
• Verify that the model is selected and click Next.
• Read and accept the terms and conditions, then click Submit.
• Wait for your request to be approved. This is usually instant for most models.

Installation​

1. Create a new local directory for your project and navigate to it:

   mkdir aws-bedrock-llama-connector
   cd aws-bedrock-llama-connector
   

2. Download from the einstein-platform repository:

   • lib/aws_bedrock_llama_connector_stack.py
   • lambda/handler.py
   • app.py
   • cdk.json
   • index.html
   • requirements.txt

3. Copy the downloaded files into your project directory.

4. Create a virtual environment and install Python dependencies:

   python -m venv venv
   source venv/bin/activate  # On Windows, use `venv\Scripts\activate`
   pip install -r requirements.txt
   

Project Structure​

Your project structure should now look like this:

aws-bedrock-llama-connector/
├── lib/
│   └── aws_bedrock_llama_connector_stack.py
├── lambda/
│   └── handler.py
├── app.py
├── cdk.json
├── index.html
└── requirements.txt

Deployment​

1. Configure AWS CLI with your credentials:

   aws configure
   

2. Bootstrap your AWS environment (if you haven't already):

   cdk bootstrap
   

3. Deploy the stack:

   cdk deploy
   

4. After deployment, CDK will output the API Gateway URL. You can use this URL to make requests to your Llama model.

Usage​

To use the deployed API, send a POST request to the /chat/completions endpoint with the following JSON structure:

{
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "Tell me a joke about programming." }
  ],
  "temperature": 0.7
}

You can use tools like cUrl, Postman, or any programming language to make HTTP requests to your API.

Example using cUrl​

Here's an example cUrl command to test your API:

curl -X POST https://your-api-id.execute-api.your-region.amazonaws.com/prod/chat/completions \
-H "Content-Type: application/json" \
-d '{
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Tell me a joke about programming."}
  ],
  "temperature": 0.7
}'

Replace https://your-api-id.execute-api.your-region.amazonaws.com/prod with the actual URL of your deployed API Gateway, which you can find in the AWS Console or in the output of the cdk deploy command.

This cUrl command sends a POST request to your API with the specified JSON payload. The API should respond with a JSON object containing the generated text from the Llama model.

Note: If you're using Windows Command Prompt, you may need to escape the double quotes in the JSON payload differently. For PowerShell or bash on Windows (e.g., Git Bash), the command should work as written above.

Testing with a Web UI​

Use the included index.html file to test your API gateway with a web UI.

Before using the web UI, edit the file and replace https://your-api-id.execute-api.your-region.amazonaws.com/prod with the actual URL of your deployed API Gateway, which you can find in the AWS Console or in the output of the cdk deploy command.

Notes​

1. This cookbook uses the meta.llama3-1-70b-instruct-v1:0 model as a placeholder. You can replace it with any supported model that you have been granted access to in Amazon Bedrock. For a list of supported model IDs, see Amazon Bedrock model IDs.
2. The token count in the response is set to -1 as AWS Bedrock doesn't provide token usage information. You may need to implement your own token counting logic if required.
3. Error handling and input validation should be improved for production use.
4. Consider disabling CORS.
5. Consider implementing authentication and rate limiting for your API Gateway.
6. Monitor your AWS usage to manage costs, especially if you expect high traffic.

Conclusion​

This cookbook demonstrates how to set up an LLM Open Connector using AWS Bedrock, Lambda, and API Gateway. It provides a serverless, scalable solution for integrating Llama models into your applications. Remember to optimize your implementation based on your specific requirements and expected usage patterns.

Prerequisites​

Before you begin, make sure that your local environment meets these prerequisites.

1. Python 3.9 or later installed on your local machine
2. A Heroku account (sign up at https://signup.heroku.com/)
3. Heroku CLI installed (https://devcenter.heroku.com/articles/heroku-cli)
4. Git installed on your local machine
5. A Groq API key (sign up at https://console.groq.com/)

Set Up Your Local Environment​

1. Create a new directory for your project:

   mkdir llm-open-connector
   cd llm-open-connector
   

2. Create a virtual environment and activate it:

   python -m venv venv
   source venv/bin/activate  # On Windows, use `venv\Scripts\activate`
   

3. Download these files from the einstein-platform repository:

   • .gitignore
   • app.py
   • index.html
   • Procfile
   • requirements.txt
   • runtime.txt

4. Copy the downloaded files into your project directory.

5. Install the required packages:

   pip install -r requirements.txt
   

Configure Your Local Environment​

1. For local testing, create a .env file in your project directory and add your Groq API key:

   GROQ_API_KEY=your_groq_api_key_here
   

   Replace your_groq_api_key_here with your actual Groq API key.

2. Make sure your .gitignore file includes the .env file to avoid accidentally committing sensitive information.

Test Your Application Locally​

1. Run your Flask application:

   python app.py
   

2. Your app should now be running on http://127.0.0.1:5000/.

3. Test the endpoints using a tool like cURL or Postman to ensure they're working correctly.

To test the chat/completions endpoint, run this cURL command:

curl -X POST http://127.0.0.1:5000/chat/completions \
-H "Content-Type: application/json" \
-d '{
  "model": "llama3-8b-8192",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of Canada?"}
  ],
  "max_tokens": 50,
  "temperature": 0.7,
  "n": 1
}'

Prepare for Heroku Deployment​

1. Initialize a Git repository in your project directory:

   git init
   

2. Add your files to the repository:

   git add .
   

3. Commit your changes:

   git commit -m "Initial commit"
   

Update Your Default Branch​

To switch the default branch used to deploy apps from master to main, follow these steps:

1. Create a new branch locally:

   git checkout -b main
   

2. Delete the old default branch locally:

   git branch -D master
   

   Now, the local environment only knows about the main branch.

3. Reset the git repository on the Heroku Platform:

   • Use the heroku-reset command from the heroku-repo CLI plugin.
   • This will not impact the running application.

   Note: Communicate this change with your team. If other developers are unaware of the reset, they might push to master, overwriting the reset.

4. To switch the default branch in GitHub, refer to this article: Setting the Default Branch.

Deploy to Heroku​

1. Make sure you're logged in to the Heroku CLI:

   heroku login
   

2. Create a new Heroku app:

   heroku create your-app-name
   

   Replace your-app-name with a unique name for your application.

3. Set the GROQ_API_KEY config var on Heroku:

   heroku config:set GROQ_API_KEY=your_groq_api_key_here -a your-app-name
   

   Replace your_groq_api_key_here with your actual Groq API key.

4. Deploy your app to Heroku:

   git push heroku main
   

5. Open your deployed app:

   heroku open
   

Your LLM Open Connector should now be deployed and accessible via the Heroku URL.

Test Your Deployed Application​

You can test your deployed application in two ways:

1. Using the example UI:

   • Open your browser and navigate to https://your-app-name.herokuapp.com
   • You'll see a simple interface where you can input prompts and get responses from the LLM
   • Try different prompts and get super fast responses!

2. Using API endpoints: Use a tool like cURL or Postman to test the endpoints of your Flask app:

   • Chat Completions: POST https://your-app-name.herokuapp.com/chat/completions

Conclusion​

You have successfully created and deployed an LLM Open Connector using the Groq API and deployed it to Heroku! This connector adheres to the Salesforce LLM Open Connector API specification, allowing for seamless integration with the Einstein AI Platform using the BYOLLM feature.

With this connector, you can bring new foundation models like Llama 3 into Einstein Studio that take advantage of Groq's fast inference platform.

Remember to monitor your usage and costs associated with the Groq API, and consider implementing additional security measures, such as rate limiting, CORS restrictions, and user authentication, before using this connector in a production environment.

Prerequisites​

Before you begin, make sure that your local environment meets these prerequisites.

1. Python 3.9 or later installed on your local machine
2. A Heroku account (sign up at https://signup.heroku.com/)
3. Heroku CLI installed (https://devcenter.heroku.com/articles/heroku-cli)
4. Git installed on your local machine
5. A SambaNova API key (sign up at https://sambanova.ai/)

Set Up Your Local Environment​

1. Create a new directory for your project:

   mkdir llm-open-connector
   cd llm-open-connector
   

2. Create a virtual environment and activate it:

   python -m venv venv
   source venv/bin/activate  # On Windows, use `venv\Scripts\activate`
   

3. Download these files from the einstein-platform repository:

   • .gitignore
   • app.py
   • index.html
   • Procfile
   • requirements.txt
   • runtime.txt

4. Copy the downloaded files into your project directory.

5. Install the required packages:

   pip install -r requirements.txt
   

Configure Your Local Environment​

1. For local testing, create a .env file in your project directory and add your API key:

   API_KEY=your_api_key_here
   

   Replace your_api_key_here with your actual API key.

2. Make sure your .gitignore file includes the .env file to avoid accidentally committing sensitive information.

Test Your Application Locally​

1. Run your Flask application:

   python app.py
   

2. Your app should now be running on http://127.0.0.1:5000/.

3. Test the endpoints using a tool like cURL or Postman to ensure they're working correctly.

To test the chat/completions endpoint, run this cURL command:

curl -X POST http://127.0.0.1:5000/chat/completions \
-H "Content-Type: application/json" \
-d '{
  "model": "Meta-Llama-3.1-8B-Instruct",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of Canada?"}
  ],
  "temperature": 0.7,
}'

Prepare for Heroku Deployment​

1. Initialize a Git repository in your project directory:

   git init
   

2. Add your files to the repository:

   git add .
   

3. Commit your changes:

   git commit -m "Initial commit"
   

Update Your Default Branch​

To switch the default branch used to deploy apps from master to main, follow these steps:

1. Create a new branch locally:

   git checkout -b main
   

2. Delete the old default branch locally:

   git branch -D master
   

   Now, the local environment only knows about the main branch.

3. Reset the git repository on the Heroku Platform:

   • Use the heroku-reset command from the heroku-repo CLI plugin.
   • This will not impact the running application.

   Note: Communicate this change with your team. If other developers are unaware of the reset, they might push to master, overwriting the reset.

4. To switch the default branch in GitHub, refer to this article: Setting the Default Branch.

Deploy to Heroku​

1. Make sure you're logged in to the Heroku CLI:

   heroku login
   

2. Create a new Heroku app:

   heroku create your-app-name
   

   Replace your-app-name with a unique name for your application.

3. Set the API_KEY config var on Heroku:

   heroku config:set API_KEY=your_api_key_here -a your-app-name
   

   Replace your_api_key_here with your actual API key.

4. Deploy your app to Heroku:

   git push heroku main
   

5. Open your deployed app:

   heroku open
   

Your LLM Open Connector should now be deployed and accessible via the Heroku URL.

Test Your Deployed Application​

You can test your deployed application in two ways:

1. Using the example UI:

   • Open your browser and navigate to https://your-app-name.herokuapp.com
   • You'll see a simple interface where you can input prompts and get responses from the LLM
   • Try different prompts and get super fast responses!

2. Using API endpoints: Use a tool like cURL or Postman to test the endpoints of your Flask app:

   • Chat Completions: POST https://your-app-name.herokuapp.com/chat/completions

Conclusion​

You have successfully created and deployed an LLM Open Connector using the SambaNova API and deployed it to Heroku! This connector adheres to the Salesforce LLM Open Connector API specification, allowing for seamless integration with the Einstein AI Platform using the BYOLLM feature.

With this connector, you can bring new foundation models like Llama 3 into Einstein Studio that take advantage of SambaNova's fast inference platform.

Remember to monitor your usage and costs associated with the SambaNova API, and consider implementing additional security measures, such as rate limiting, CORS restrictions, and user authentication, before using this connector in a production environment.

Prerequisites​

• A Salesforce org with Einstein Generative AI and Data Cloud enabled.
• A Salesforce Einstein Studio Environment.

Step 1. Create Your watsonx Instance​

Create an account.

1. If you don’t already have one, create a watsonx trial account.

   If you have an existing, non-trial watsonx account, you need to follow these additional steps:

   1. In IBM Cloud, set up IBM Cloud Object Storage for use with IBM watsonx.
   2. Set up the Watson Studio and Watson Machine Learning services.
   3. Create a new project from the IBM watsonx console.

Find your Project ID value.

1. In your sandbox, click the Manage tab and copy your Project ID value.
2. Store the project-id value. You’ll need this value along with an API key and a region-id in step three.

Create an IBM Cloud API Key.

1. In the IBM Cloud console, go to Manage > Access (IAM).
2. In the sidebar, select API Keys.
3. Click Create.
4. Enter a name and description for your API key.
5. Then, click Create.
6. Store your API key in a safe location.

Get the region value for the watsonx instance.

1. From the watsonx home page, check the region in the header bar.

2. Store the region-id value.

Step 2. Verify Your Open Connector Implementation (Optional)​

IBM automatically provides a working implementation of the Open Connector for watsonx that you can use to test this workflow. If you want to create your own implementation for production use cases, follow these steps. Otherwise, skip to step three.

1. Create your own connector implementation for watsonx. For directions and code, see IBM's watsonx connector repo.

Step 3. Create a BYOLLM Connection to the watsonx Model in Einstein Studio​

Before you connect your Open Connector implementation to Einstein Studio, you need three pieces of information from Step 1: project-id, API key, and region-id.

1. Log in to your Salesforce org as an admin user and open Einstein Studio.
2. Click Model Library.
3. Click Add Foundation Model.

4. In Model Builder, click Connect to your LLM.

5. Click Next.
6. Enter the details of your watsonx instance.
   • Name: IBM Granite (or your own preferred name)
   • URL: if you are using the connector hosted by IBM, get the URL from IBM's watsonx connector repo. Otherwise, use the URL from the connector instance that you have created from the code and documentation in the watsonx repo. When filling out the URL value, be sure to use the project-id and region-id that you copied from step 1.
   • Authentication: Key Based
   • Auth Header: X-IBM-API-KEY
   • Key: [your IBM API key]
   • Model Name/ID: ibm/granite-13b-chat-v2 (Or the specific model ID you want to connect to, refer to watsonx model IDs from your IBM watsonx console.)
   • Token Limit: 8,192 (Or the specified model's maximum context length. Refer to the watsonx model IDs from your IBM watsonx console.)

7. Click Connect.
8. Click Name and Connect.

Step 4. Create a Configured Model​

Before you can use your connected LLM, you need to create a configured model.

1. In the Model Library in Einstein Studio, select Configure Foundation Model.
2. Select Create Model.
3. Choose your connected LLM from the “Foundation Model” dropdown.
4. Configure the model parameters.
5. Test the model performance in the model playground.
6. After testing, click Save in the bottom right.
7. Navigate to the "Generative" tab of the Einstein Studio Models page and verify that your watsonx configured model appears.
8. Select your watsonx model to view the configuration details or create a prompt template.

You can now use your LLM wherever you can use Einstein Studio generative models. For instance, you can build prompt templates using Prompt Builder.

See Also​

• External: IBM watsonx Repo
• Interactive Demo: Use watsonx AI models from Salesforce
• Salesforce Help: Add a Foundation Model

This recipe implements an example of Cerebras Inference; however, the high-level process is the same for all models and providers.

High-level Process​

There are four high-level steps for connecting your MuleSoft model endpoint to the LLM Open Connector.

Prerequisites + Tutorial Video​

Before you begin, review the prerequisites:

1. You have a MuleSoft Account (Sign up here).
2. You have Anypoint Code Builder (ACB) or Anypoint Studio installed. The instructions in this recipe are based on Anypoint Studio.
3. You have Inference Connector installed.
4. You have a Cerebras account with an API key.

View a step-by-step tutorial video that covers an implementation similar to the one covered in this recipe:

Step 1: Download the API Specification for the LLM Open Connector​

1. Download LLM Open Connector API Spec.

2. Rename the file from llm-open-connector.yml to llm-open-connector.yaml.

Step 2: Import the API Specification into Anypoint Design Center​

1. Log in to your MuleSoft account.

2. Go to Anypoint Design Center.

3. Import a new API specification from a file using these values:

   • Project Name: Cerebras-LLM-Provider,
   • API Specification: Select REST API,
   • File upload: llm-open-connector.yaml. Be sure to use the renamed file.

4. Click Import.

5. Verify that the API specification successfully imported.

6. Change termsOfService: "" to termsOfService: "-".

7. Remove servers: and the example url.

servers:
  - url: https://bring-your-own-llm.example.com

7. Click Publish.

8. Provide versioning information:

   • Asset version: 1.0.0
   • API version: v1
   • Lifecycle State: Stable

9. Click Publish to Exchange.

Step 3: Implement the API Specification​

This cookbook uses Anypoint Studio to implement the API Specification. If you prefer, you can also implement the spec with Anypoint Code Builder.

Import API Specification into Studio​

1. Open Anypoint Studio and create a Mule Project.

2. Name the project and import an API spec:

   • Project Name: cerebras-llm-provider,
   • Import a published API: Select the Cerebras-LLM-Provider API spec from the previous step.

3. Click Finish.

Add the Inference Connector to Your Project​

1. If you have not installed the Inference Connector, install it before you start.
2. Add the Inference Connector dependency to the Mule Project.

<dependency>
  <groupId>com.mulesoft.connectors</groupId>
  <artifactId>mac-inference-chain</artifactId>
  <version>0.1.0</version>
  <classifier>mule-plugin</classifier>
</dependency>

3. Make sure the Inference Connector is present in the Mule Palette.

Implement the Chat Completions Endpoint​

1. Go to the scaffolded flow post:\chat\completions:application\json:llm-open-connector-config.

2. Drag and drop Chat completions operation from the Inference Connector into the Flow.

3. Provide the Inference connector configuration for Cerebras.

4. Parametrize all properties needed by the LLM Open Connector API Spec. .

5. In the Chat completions operation, enter: payload.messages.

6. Before the Chat completions operation, add the Set Variable operation with the name model and enter in the expression value payload.model.

7. After the Chat completions operation, add the Transform Message operation and provide the mapping in this code block:

%dw 2.0
output application/json
---
{
	id: "chatcmpl-" ++ now() as Number as String,
	created: now() as Number,
	usage: {
		completion_tokens: attributes.tokenUsage.outputCount,
		prompt_tokens: attributes.tokenUsage.inputCount,
		total_tokens: attributes.tokenUsage.totalCount
	},
	model: vars.model,
	choices: [
    	{
      finish_reason: "stop",
      index: 0,
      message: {
        content: payload.response default "",
        role: "assistant"
      }
    }
  ],
	object: "chat.completion"
}

8. Save the project.

Test Locally​

1. Start the Mule Application.

2. Go to the API Console.

3. Enter the API Key and following payload:

{
  "messages": [
    {
      "content": "What is the capital of Switzerland?",
      "role": "user",
      "name": ""
    }
  ],
  "model": "llama3.1-8b",
  "max_tokens": 500,
  "n": 1,
  "temperature": 0.7,
  "parameters": {
    "top_p": 0.5
  }
}

4. Validate the result. Make sure the values are mapped correctly for token usage.

{
  "id": "chatcmpl-1732373228",
  "created": 1732373228,
  "usage": {
    "completion_tokens": 8,
    "prompt_tokens": 42,
    "total_tokens": 50
  },
  "model": "llama3.1-8b",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "The capital of Switzerland is Bern.",
        "role": "assistant"
      }
    }
  ],
  "object": "chat.completion"
}

Step 4. Deploy to Anypoint CloudHub​

1. After the application is tested successfully, deploy it to Anypoint CloudHub.

2. Right-click your Mule Project and navigate to Anypoint Platform > Deploy to CloudHub.

3. Choose the environment you want to deploy to.

4. Enter the required values:

   • App name: cerebras-llm-provider
   • Deployment target: Shared Space (Cloudhub 2.0)
   • Replica Count: 1
   • Replica Size: Micro (0.1 vCore)

5. Click Deploy Application.

6. Wait until the application is deployed.

Create a Configuration in Model Builder​

After your API is running on CloudHub, you need to add the model endpoint to Model Builder.

1. In Salesforce, open Data Cloud.

2. Navigate to Einstein Studio.

3. Click Add Foundation Model, and click Connect to your LLM.

4. Click Next.

5. Enter the required values:

   • Name: Cerebras-LLM-Provider
   • URL: <cloudhub_url>/api
   • Model: A model name is required. For this recipe, choose between llama3.1-70b or llama3.1-8b.

6. Click Connect.

7. Create two configurations for each supported model:

   • llama3.1-70b
   • llama3.1-8b

Important Considerations​

1. This cookbook uses Cerebras models llama3.1-70b and llama3.1-8b.
2. When configuring in Model Builder, you need to provide a default value for the model. In this recipe the model name is parametrized, so a value is required.
3. The API is deployed under the governance of the MuleSoft Anypoint Platform. As a result:
   • You can monitor the application by viewing logs and errors.
   • You can apply additional security through Anypoint's API management capabilities.
   • You can deploy multiple replicas to scale horizontally and vertically.

Conclusion​

This cookbook demonstrates how to set up an LLM Open Connector using MuleSoft for the Chat Completion endpoints of Cerebras. This recipe is a sandbox implementation, and it's not production ready. For production use, please optimize your implementation based on your specific requirements and expected usage patterns.

Prerequisites​

1. Anypoint Platform account (create a free trial here).
2. MuleSoft's IDE: either Anypoint Code Builder (ACB) or Anypoint Studio. This recipe is based on ACB.
3. Ollama locally running the model of your choice (for example, llama3).
4. Ngrok to expose your local Ollama instance to the public cloud.
5. A REST client application to test the API calls. For example, cURL or Postman.

For troubleshooting, you can compare your Mule app with this GitHub repo.

Expose the Ollama URL​

1. Verify your model is up and running (locally) in Ollama using ollama list or ollama ps.

2. Run this command to start ngrok:

   ngrok http 11434 --host-header="localhost:11434"
   

3. Copy the address from the Forwarding field. This address is the URL you need to connect to Ollama from the Mule app.

Create and Publish the API Specification​

1. Download the llm-open-connector.yml file from the einstein-platform GitHub repository.

2. Rename the extension of the file from yml to yaml.

3. Log in to your Anypoint Platform account.

4. Navigate to Design Center.

5. Click Create + > Import from File.

6. Fill out the spec details:

   • Project name: LLM Open Connector
   • Import type: API specification REST API
   • Import file: The llm-open-connector.yaml file from steps 1 and 2. (Ensure that the file extension is .yaml.)

7. Click Import.

8. Add a character inside termsOfService (for example, termsOfService: "-").

9. Remove the servers section, including the example url:

   servers:
       - url: https://bring-your-own-llm.example.com
   

10. Click Publish in the top right of the screen.

11. Provide versioning information:

    • Asset version: 1.0.0
    • API version: v1
    • LifeCycle State: Stable

12. Click Publish to Exchange.

13. After it is published, you can click the Exchange link to see the preview of the published asset, or click Close.

14. Exit Design Center.

Implement the Mule App​

1. Head to the IDE of your choice. In this recipe, we are using Anypoint Code Builder, the MuleSoft IDE powered by Visual Studio Code.

2. In ACB, click Implement an API.

   note

   If you haven't signed in to your Anypoint Platform account through ACB, it asks you to sign in. Follow the prompts to sign in.

3. Fill out the project details:

   • Project Name: ollama-llm-provider
   • Project Location: choose any folder to keep this project
   • Search an API Specification from Exchange to implement it: the LLM Open Connector we just published
   • Mule Runtime: 4.8.0
   • Java Version: 17

4. Click Create Project and wait for it to be fully processed.

Maven​

1. After it is processed, open the pom.xml file and add the Maven dependency:

   <dependency>
       <groupId>com.mulesoft.connectors</groupId>
       <artifactId>mule4-aichain-connector</artifactId>
       <version>1.0.0</version>
       <classifier>mule-plugin</classifier>
   </dependency>
   

2. Change the version in line 6:

   <version>1.0.0</version>
   

3. Copy the groupId (in dependencies) for the llm-open-connector artifact. The groupId is a number similar to: d62b8a81-f143-4534-bb89-3673ad61ah01. This number is your organization ID.

4. Paste this organization ID in the groupId field at the top of the file, replacing com.mycompany:

   <groupId>d62b8a81-f143-4534-bb89-3673ad61ah01</groupId>
   

LLM Config​

1. Create a file under src/main/resources named llm-config.json.

2. Paste this code into the llm-config.json file:

   {
       "OLLAMA": {
           "OLLAMA_BASE_URL": "https://11-13-23-16-11.ngrok-free.app"
       }
   }
   

3. Replace the example URL with the one you copied from ngrok in step 3 of Expose the Ollama URL.

Mule Flow​

1. Open the ollama-llm-provider.xml file under src/main/mule.

2. Add this code under apikit:config and before <flow name="llm-open-connector-main">.

   <ms-aichain:config configType="Configuration Json" filePath='#[(mule.home default "") ++ "/apps/" ++ (app.name default "") ++ "/llm-config.json"]' llmType="OLLAMA" modelName="#[vars.inputPayload.model default 'llama3']" name="MAC_Config" temperature="#[vars.inputPayload.temperature default 1]" maxTokens="#[vars.inputPayload.max_tokens default 500]"></ms-aichain:config>
   
   note

   If your model is not llama3, make sure you upload the default value in the previous line (for example, modelName="#[vars.inputPayload.model default 'llama3']").

3. Locate the last flow in the file (for example, post:\chat\completions:application\json:llm-open-connector-config).

4. Remove the logger and add this code to the flow.

   <logger doc:name="Logger" doc:id="ezzhif" message="#[payload]" />
   <ee:transform doc:name="Transform" doc:id="dstcls">
       <ee:message>
           <ee:set-payload><![CDATA[(payload.messages filter ($.role == "user"))[0].content default payload.messages[0].content]]></ee:set-payload>
       </ee:message>
       <ee:variables>
           <ee:set-variable variableName="inputPayload">
               <![CDATA[payload]]>
           </ee:set-variable>
       </ee:variables>
   </ee:transform>
   <ms-aichain:chat-answer-prompt config-ref="MAC_Config" doc:id="mmoptd" doc:name="Chat answer prompt" />
   <ee:transform doc:name="Transform" doc:id="czdqgi">
       <ee:message>
           <ee:set-payload>
                   <![CDATA[output application/json
   ---
   {
       id: uuid( ),
       choices: [
           {
               finish_reason: "stop",
               index: 0,
               message: {
                   content: payload.response default "",
                   role: "assistant"
               }
           }
       ],
       created: now() as Number,
       model: vars.inputPayload.model default "",
       object: "chat.completion",
       usage: {
           completion_tokens: (attributes.tokenUsage.outputCount default 0) as Number,
           prompt_tokens: (attributes.tokenUsage.inputCount default 0) as Number,
           total_tokens: (attributes.tokenUsage.totalCount default 0) as Number
       }
   }]]>
           </ee:set-payload>
       </ee:message>
   </ee:transform>
   <logger doc:name="Logger" doc:id="rzhuiw" message="#[payload]" />
   

5. Save all the files.

Deploy to CloudHub 2.0​

1. With the ollama-llm-provider.xml file open, click Deploy to CloudHub in the top right of the screen (the rocket 🚀 icon).

2. Select CloudHub 2.0.

3. Select Cloudhub-US-East-2 if you have a free trial account based on the US cloud. Otherwise, select any of the regions available to you.

   note

   To verify the regions you have access to in your Anypoint Platform account, you can do so from the Runtime Manager by creating a deployment and checking the available options.

4. Select Sandbox as the environment.

5. A new file named deploy_ch2.json is created under src/main/resources. You can change the data in this file if needed. By default, it contains:

   {
       "applicationName": "ollama-llm-provider",
       "runtime": "4.8.0",
       "replicas": 1,
       "replicaSize": "0.1",
       "deploymentModel": "rolling"
   }
   

6. In the message that appears in the bottom right, click Deploy.

7. In the next prompt, select the latest Mule runtime version available. For this recipe, you can select 4.8.1:6e-java17 if available. This action changes the runtime field in the deploy_ch2.json file.

8. Your asset is first published to Exchange as an Application. Afterwards, the deployment to CloudHub 2.0 starts. When you receive the message: 'ollama-llm-provider' is deployed to 'Sandbox' Env in CloudHub 2.0, the deployment has successfully started in Anypoint Platform.

9. Navigate to your Anypoint Platform account and open Runtime Manager (or click Open in Runtime Manager in ACB).

10. When your application's status appears as 🟢 Running, it's ready to start receiving requests. Copy the public endpoint URL (for example, https://ollama-llm-provider-69s5jr.5se6i9-2.usa-e2.cloudhub.io).

11. Add /api at the end of the URL. This URL is used to call the API.

Test the Deployed Mule App​

You can call the previous URL using tools like cURL or Postman. Here is an example cURL command; make sure you replace the example URL with your own.

curl -X POST https://ollama-llm-provider-69s5jr.5se6i9-2.usa-e2.cloudhub.io/api/chat/completions \
-H 'Content-Type: application/json' \
-d '{
  "messages": [
    {
      "content": "What is the capital of Canada?",
      "role": "user",
      "name": "Alex"
    }
  ],
  "model": "llama3",
  "max_tokens": 0,
  "n": 1,
  "temperature": 1,
  "parameters": {
    "top_p": 0.5
  }
}'

Bring Your Connected Endpoint to Salesforce Model Builder​

Follow the instructions in this developer blog to use your model in Model Builder. When you activate your model, you can use it within Prompt Builder, the Models API, and for building actions using prompt templates in Agent Builder. All these methods provide built-in security offered by the Einstein Trust Layer.

Conclusion​

This cookbook demonstrates how to set up the LLM Open Connector using Ollama locally for Chat Completion endpoints for various models. This option is recommended for local testing since there are no credit limits. Another scenario is if you want to test your own model locally using Ollama. The implementation in this recipe is not recommended for production use unless you plan to host your model on-premises.

Overview​

This document outlines detailed steps for using LLM Open Connector with Vertex AI on Google Cloud.

Instead of building custom, one-off integrations for each LLM vendor, the LLM Open Connector provides a standard API-based approach on the OpenAI schema. LLM Open Connector provides a simple and quick path for teams to connect any model—be it enterprise-grade, open-source, or custom-trained—to Salesforce.

In this example, we will deploy a fine-tuned Gemini model from Google on Vertex AI Model Garden, use Google Cloud Run Functions (Gen 2) for serverless compute, providing a scalable and maintainable solution that connects Salesforce to Vertex AI's model endpoints. This approach is particularly suited for Agentforce chat completion use cases, enabling advanced conversational AI capabilities within Salesforce.

Tutorial Video​

View a step-by-step tutorial video that covers this Vertex AI recipe on YouTube:

Background Information​

This section introduces the products and features covered in the cookbook. If you're already familiar with LLM Open Connector, Google Cloud Platform (GCP), and Vertex AI feel free to skip to the next section.

1. Google Cloud equivalents to AWS services.

   • AWS Bedrock: GCP's equivalent for hosting foundation models is Vertex AI. Vertex AI provides access to various models, including Google's own and other partner models. All models can be found in Model Garden
   • AWS Lambda: GCP's serverless compute platform is Cloud Functions. This will host your code that interacts with the LLM.
   • AWS API Gateway: GCP's API management service is Cloud Endpoints or Cloud Functions (HTTP). These services create the API endpoint that the LLM Open Connector will call in Salesforce.

2. Vertex AI offers a range of foundation models. This cookbook uses Gemini 1.5 Pro.

   • Text models, like PaLM 2, offer different sizes (e.g., Bison, Gecko) and are optimized for various tasks and performance requirements.
   • Code models, for code generation and related tasks.
   • Chat models, optimized for conversational interactions.

High-level Solution Components​

1. A fine-tuned Gemini model deployed on Vertex AI (required for the cookbook). An end user can use any model as-is from Model Garden for their use case.
2. A Cloud Run Function that serves as a middleware, handling authentication and request/response transformation.
3. Secret Manager for secure API key storage.
4. Salesforce Einstein Studio configuration for the LLM Open Connector.

High-level Solution Flow​

In this scenario, we will deploy a fine-tuned Gemini Model in Google Cloud Platform and leverage Cloud Run Functions to authenticate the Data Cloud Open LLM Connector and invoke the Vertex Model endpoint. This setup is for Agentforce chat completion.

Prerequisites​

Before beginning the implementation, review the following requirements.

1. Google Cloud Account with appropriate permissions. Required project editor or equivalent roles.

   note

   If you’re new to Google Cloud, new customers get $300 in free credits to run, test, and deploy workloads as of 02/11/2025.

2. A Google Cloud project with billing enabled.

   • On the project selector page in the Google Cloud console, select or create a Google Cloud project.
   • Ensure that billing is enabled for your Google Cloud project.

3. Google Cloud SDK (gcloud CLI). For installation instructions, see Install the gcloud CLI and Set up gcloud CLI.

4. Python 3.9 or later.

5. Access to Vertex AI. You may need to enable the Vertex AI API in your project via the Google Cloud Console if it's not already enabled.

6. Enable all APIs required for this cookbook:

   • Vertex AI API
   • Cloud Run API
   • Secret Manager API
   • Cloud Build API
   • Any additional APIs as required by Google Cloud UI.

Implementation Steps​

1. Fine-tune Your Gemini Model​

Fine-tuning a Gemini model allows you to customize its responses for your specific use case. Here's how to set it up:

1. Access Vertex AI Model Garden:

   • Go to the Google Cloud Console.
   • Go to Vertex AI > Model Garden.
   • Search for "Gemini".
   • Select "gemini-1.5-pro" (non-streaming version).

2. Prepare your training data. The training data must be in JSONL format, with each line containing a complete conversation example. The following template represents the required structure:

   {"contents": [{"role": "user", "parts": [{"text": "Your question here"}]}, {"role": "model", "parts": [{"text": "Expected response here"}]}]}
   

   Refer to the following test data sample:

   {"contents": [{"role": "user", "parts": [{"text": "Based on sensor data, what is the recommended maintenance interval for CNC Lathe Alpha?"}]}, {"role": "model", "parts": [{"text": "CNC Lathe Alpha should be maintained every 5 months, especially if sensor readings show vibration levels above 2.5 m/s² or temperatures exceeding 80°C."}]}]}
   
   {"contents": [{"role": "user", "parts": [{"text": "What maintenance schedule does CNC Milling Beta require according to the latest IoT data?"}]}, {"role": "model", "parts": [{"text": "CNC Milling Beta should be scheduled for maintenance every 6 months. Abnormal spindle temperatures above 85°C or increased vibrations may indicate the need for earlier service."}]}]}
   
   {"contents": [{"role": "user", "parts": [{"text": "From the IoT sensor analysis, what maintenance interval is recommended for Hydraulic Press Delta?"}]}, {"role": "model", "parts": [{"text": "Hydraulic Press Delta is recommended to be maintained every 4 months. Sudden drops in hydraulic pressure or unexpected increases in acoustic emissions should prompt earlier service."}]}]}
   
   {"contents": [{"role": "user", "parts": [{"text": "According to sensor readings, how often should Robotic Arm Omega be serviced?"}]}, {"role": "model", "parts": [{"text": "Robotic Arm Omega should be serviced every 7 months, with earlier checks if joint torque variability exceeds 10% or if motor temperatures spike unexpectedly."}]}]}
   
   {"contents": [{"role": "user", "parts": [{"text": "What maintenance interval does the sensor data suggest for Injection Molder Sigma?"}]}, {"role": "model", "parts": [{"text": "Injection Molder Sigma should undergo maintenance every 8 months. Notable increases in heating element temperature or pressure fluctuations indicate the need for an earlier service."}]}]}
   
   {"contents": [{"role": "user", "parts": [{"text": "Based on current IoT readings, what maintenance frequency is recommended for Conveyor Belt Gamma?"}]}, {"role": "model", "parts": [{"text": "Conveyor Belt Gamma is best maintained every 3 months, especially if sensor data shows inconsistent belt speeds or abnormal motor vibrations."}]}]}
   
   {"contents": [{"role": "user", "parts": [{"text": "What does the IoT data recommend for the maintenance schedule of Packaging Machine Epsilon?"}]}, {"role": "model", "parts": [{"text": "Packaging Machine Epsilon should be maintained every 6 months. Spikes in vibration levels or minor misalignments detected by sensors suggest that prompt maintenance may be necessary."}]}]}
   
   {"contents": [{"role": "user", "parts": [{"text": "How often should Press Brake Zeta be serviced according to sensor trends?"}]}, {"role": "model", "parts": [{"text": "Press Brake Zeta should be serviced every 5 months. Consistent monitoring of force output and temperature is crucial; significant deviations indicate that earlier service might be required."}]}]}
   
   {"contents": [{"role": "user", "parts": [{"text": "Based on sensor insights, what is the recommended maintenance interval for Laser Cutter Theta?"}]}, {"role": "model", "parts": [{"text": "Laser Cutter Theta is recommended to be maintained every 4 months. Increases in ambient dust levels or fluctuations in laser power readings are critical indicators for timely maintenance."}]}]}
   
   {"contents": [{"role": "user", "parts": [{"text": "What maintenance schedule should be followed for Drilling Machine Kappa based on IoT data?"}]}, {"role": "model", "parts": [{"text": "Drilling Machine Kappa should be maintained every 6 months, with special attention if there are sudden spikes in vibration or drilling temperature that may signal an urgent service need."}]}]}
   

3. Create and configure the tuning job:

   • Go to Vertex AI → Training.
   • Click Create Training Job.
   • Select your fine-tuning dataset.
   • Configure the hyperparameters (start with defaults).
   • Start the training job.

4. Deploy the fine-tuned model:

   • After you complete training, go to the model in Vertex AI.
   • Click Deploy.
   • Select region (e.g., us-central1).
   • Store the following information. You'll need these values later.
     • Endpoint Name
     • Endpoint ID
     • Location
     • Project ID

2. Create and Configure Your Secrets in Google Cloud​

The LLM Open Connector requires an API key for authentication. We'll store this securely in Google Cloud Secret Manager.

1. Go to Secret Manager:

   • Open Google Cloud Console.
   • Go to Security → Secret Manager
   • Click CREATE SECRET.

2. Configure the secret:

   • Name: llm_connector_api_key
   • Upload a pre-generated API key — you'll need this when configuring your model in Salesforce.
   • Click CREATE SECRET.
   note

   For the cookbook, we set this key to never expire. GCP takes care of rotating the encryption key. However, a different Cloud Run function can be set to rotate the API key. When a new key is generated in GCP, the Open LLM Connector API key needs to be updated.

3. Verify the secret creation. You should see your secret listed in Secret Manager. The "Resource ID" will be in the format: projects/{PROJECT_ID}/secrets/llm_connector_api_key.

4. Configure IAM Permissions. Ensure your Cloud Run Function's service account has the following roles:

   • Secret Manager Secret Accessor
   • Vertex AI User

We recommend creating a dedicated Cloud Run function service account. Provide all required permissions to this service account.

3. Implement the Cloud Run Function​

The Cloud Run Function serves as middleware between Salesforce and Vertex AI. Follow the steps to implement it. You can download asset files from the einstein-platform repo.

1. Create the Function Code with at least the minimum required settings.
   • Trigger type: HTTPS
   • Choose a dedicated service account that has access to the secrets.
   • Runtime python 3.10
2. Configure environment variables:
   • Project ID
   • Location
   • Endpoint ID (fine-tuned model endpoint)
3. Create a new directory for your function and add the following files:
   • main.py: openllmconnectormain.py
   • requirements.txt:

     functions-framework==3.*
     google-cloud-aiplatform==1.*
     google-cloud-secret-manager==2.*
     flask==2.*
     
4. Test and deploy the function.

4. Create a Configured Model in Salesforce​

1. In your Salesforce Org, open Data Cloud.
2. Go to Einstein Studio, go to Model Builder.
3. Click New Configuration.
4. Select "LLM Open Connector".
5. Enter Configuration Details:
   • Endpoint URL: https://{REGION}-{PROJECT_ID}.cloudfunctions.net/llm\_connector/chat/completions
   • Auth Header: api-key
   • Auth Key: {VALUE_FROM_SECRET_MANAGER}
   • Model Name: {FINETUNED_MODEL_NAME}
6. After the connection is successful, select and test the fine-tuned model in Model Garden.
7. Adjust temperature settings and create a chat completions version for use in Salesforce.

Congratulations! Your Vertex AI model is now ready to use!

See Also​

• LLM Open Connector + AWS

Prerequisites​

1. If you don't already have one, create an xAI account.
2. Create a team in the xAI Cloud Console.
3. Purchase some xAI API credits.
4. Create an API key.

Integrate with Agentforce​

xAI is fully compatible with the Open Connector API specification, so you can follow the LLM Open Connector product guide. Follow the steps in this post for an overview.

Step 1. Add a Foundation Model​

Navigate to Einstein Studio in Data Cloud.

Step 2. Connect Your LLM​

Select the option to bring your own model.

Step 3. Enter Your xAI API Key​

Enter your xAI details to create a new model connection in Model Builder.

• Add https://api.x.ai/v1/chat/completions to the URL field.
• Add your API secret to the Auth Key field.
• Add grok-4 to the Model Name field.

Step 4. Create A Configured Model​

Before you can use your connected model, you need to create a configured model. Select grok-4 in the Model Library and click New Configuration.

Test your model in the the Model Playground. When you're ready, you can save your configuration and create the model for use in Salesforce.

Step 5. Start Using Grok​

You can now use Grok 4 via Salesforce's Open Connector! You can leverage Grok’s advanced reasoning in your Agentforce application to handle complex requests and streamline workflows.

To monitor consumption, purchase additional credits, and manage your Grok models, use the xAI Cloud console.