What Does a Penalty Value Like 1.5 Mean for Frequency and Presence in Token Generation?

When it comes to frequency and presence penalties, the numerical value isn't a probability threshold like Top P. Instead, it acts as a scaling factor that adjusts the model’s logit calculations for each token.

What Does a Value Like 1.5 Mean?

• Scaling Factor: A value of 1.5 means that for each token in the generation process, a penalty of 1.5 (multiplied by its frequency or a binary indicator for presence) is subtracted from its log probability.
• Effect on Token Sampling:
  • For frequency penalty, tokens that have been used often will receive a larger decrease in their probability, making them less likely to be chosen again.
  • For presence penalty, any token that has already appeared (even once) receives a fixed penalty, reducing its chance to be repeated.
• Relative Strength: A 1.5 penalty is moderate to strong. This is higher than a value like 0.2 or 0.5, meaning the effect of discouraging repetition will be more pronounced. However, because these adjustments are applied to the log probabilities, the final impact on token selection is intertwined with the baseline probabilities produced by the model.

Important Distinctions

• Not a Percentage: Unlike Top P, which limits the selection to a cumulative probability mass (e.g., 70%), the penalties adjust how much more “costly” it is to repeat tokens.
• Tuning Behavior: These parameters are primarily a tool for controlling repetitiveness and steering the conversation toward more novel completions. The effect of a specific value (like 1.5) may vary based on the context of the prompt and other sampling parameters.

In summary, a value like 1.5 doesn’t limit the available options by percentage but instead applies a stronger discouragement to tokens that have already appeared, pushing the model to potentially choose less frequent or new tokens in the output.