LocalLm types documentation
    Preparing search index...

    Interface InferenceParams

    Describes the parameters for making an inference request.

    InferenceParams

    const inferenceParams: InferenceParams = {
      stream: true,
      model: { name: 'gpt-3', ctx: 2048 },
      template: 'default',
      max_tokens: 150,
      top_k: 50,
      top_p: 0.9,
      min_p: 0.01,
      temperature: 0.7,
      repeat_penalty: 1.2,
      tfs: 0.8,
      stop: ['###'],
      grammar: 'default_grammar',
      images: ['data:image/png;base64,...']
    };
    interface InferenceParams {
        extra?: Record<string, any>;
        grammar?: string;
        images?: string[];
        max_tokens?: number;
        min_p?: number;
        model?: ModelConf;
        repeat_penalty?: number;
        schema?: Record<string, any>;
        stop?: string[];
        stream?: boolean;
        temperature?: number;
        template?: string;
        tfs?: number;
        top_k?: number;
        top_p?: number;
        tsGrammar?: string;
    }
    Index

    Properties

    extra? grammar? images? max_tokens? min_p? model? repeat_penalty? schema? stop? stream? temperature? template? tfs? top_k? top_p? tsGrammar?

    Properties

    extra?: Record<string, any>

    Extra parameters to include in the payload

    grammar?: string

    The gnbf grammar to use for grammar-based sampling.

    images?: string[]

    The base64 images data (for multimodal models).

    max_tokens?: number

    The number of predictions to return.

    min_p?: number

    The minimum probability for a token to be considered, relative to the probability of the most likely token.

    model?: ModelConf

    The model configuration details for inference.

    repeat_penalty?: number

    Adjusts penalty for repeated tokens.

    schema?: Record<string, any>

    A json schema to format the output.

    stop?: string[]

    List of stop words or phrases to halt predictions.

    stream?: boolean

    Indicates if results should be streamed progressively.

    temperature?: number

    Adjusts randomness in sampling; higher values mean more randomness.

    template?: string

    The template to use, for the backends that support it.

    tfs?: number

    Set the tail free sampling value.

    top_k?: number

    Limits the result set to the top K results.

    top_p?: number

    Filters results based on cumulative probability.

    tsGrammar?: string

    A Typescript interface to be converted to a gnbf grammar to use for grammar-based sampling.

    Settings

    Member Visibility

    On This Page

    Properties