Interface InferenceParams

Describes the parameters for making an inference request.

InferenceParams

interface InferenceParams {
    extra?: Record<string, any>;
    grammar?: string;
    images?: string[];
    max_tokens?: number;
    min_p?: number;
    model?: ModelConf;
    repeat_penalty?: number;
    stop?: string[];
    stream?: boolean;
    temperature?: number;
    template?: string;
    tfs?: number;
    top_k?: number;
    top_p?: number;
}

Properties

extra? grammar? images? max_tokens? min_p? model? repeat_penalty? stop? stream? temperature? template? tfs? top_k? top_p?

Properties

Optional extra

extra?: Record<string, any>

Extra parameters to include in the payload

Optional grammar

grammar?: string

The gnbf grammar to use for grammar-based sampling.

Optional images

images?: string[]

Optional max_tokens

max_tokens?: number

The number of predictions to return.

Optional min_p

min_p?: number

The minimum probability for a token to be considered, relative to the probability of the most likely token.s

Optional model

model?: ModelConf

The model configuration details for inference.

Optional repeat_penalty

repeat_penalty?: number

Adjusts penalty for repeated tokens.

Optional stop

stop?: string[]

List of stop words or phrases to halt predictions.

Optional stream

stream?: boolean

Indicates if results should be streamed progressively.

Optional temperature

temperature?: number

Adjusts randomness in sampling; higher values mean more randomness.

Optional template

template?: string

The template to use, for the backends that support it.

Optional tfs

tfs?: number

Set the tail free sampling value.

Optional top_k

top_k?: number

Limits the result set to the top K results.

Optional top_p

top_p?: number

Filters results based on cumulative probability.

Settings

Member Visibility

Theme

On This Page

extragrammarimagesmax_tokensmin_pmodelrepeat_penaltystopstreamtemperaturetemplatetfstop_ktop_p