Llama API | Batch Inference › Chat Completion

Batch Inference

batch_inference

Methods

Chat Completion -> { completion_message_batch }

post/alpha/batch-inference/chat-completion

Parameters

X-LlamaStack-Client-Version: string

Optional

X-LlamaStack-Provider-Data: string

Optional

Response fields

completion_message_batch: Array<{ content, role, stop_reason, 1 more... }>

Request example

curl http://any-hosted-llama-stack.com/alpha/batch-inference/chat-completion \
    -H 'Content-Type: application/json' \
    -d '{
      "messages_batch": [
        [
          {
            "content": "string",
            "role": "user"
          }
        ]
      ],
      "model": "model"
    }'

200Example

{
  "completion_message_batch": [
    {
      "content": "string",
      "role": "assistant",
      "stop_reason": "end_of_turn",
      "tool_calls": [
        {
          "arguments": {
            "foo": "string"
          },
          "call_id": "call_id",
          "tool_name": "brave_search"
        }
      ]
    }
  ]
}

Completion ->

BatchCompletion

post/alpha/batch-inference/completion