ftgate

ftgate is a robust tool designed for performing regression checks on fine-tuned models, specifically focusing on tool calling. It answers the critical question of whether the byte data sent during runtime corresponds accurately to what the model was initially trained on, and whether it effectively calls the correct tools even after processes such as export and quantization.

Key Features

• Accurate Byte Comparison
  The ftgate template allows for three-sided byte comparison, ensuring that the output of llama-server matches the exact training bytes, which is crucial for reliable tool-calling prompts.

• Error Diagnosis
  Extensive measurement findings highlight issues such as incorrect data representations and system prompt additions, which can impair model performance. For instance, the Ollama provider inadvertently alters the model's expected input structure, leading to potential inaccuracies.

• Dataset Linting
  With ftgate data, the dataset is linted using the model's own tokenizer and template. It checks for common pitfalls such as empty assistant targets, incorrect role ordering, and schema compliance, thus ensuring data quality from the onset.

• Tool Calling Validation
  The utility of ftgate tools evaluates whether the tools are still accessible post-finetuning. It utilizes promptfoo for intelligent evaluation, ensuring that all expected arguments are present and correctly formatted in JSON, offering detailed feedback on any failures encountered.

Usage Example

Setting up and utilizing the ftgate functionality is straightforward. Here is a basic usage example:

pip install ftgate
ftgate template --model Qwen/Qwen2.5-0.5B-Instruct \
                --llama http://localhost:8080 \
                --ollama qwen2.5:0.5b-instruct \
                --dataset train.jsonl --sample 20 --diff

CI Integration

ftgate also includes provisions for continuous integration. Users can seamlessly integrate ftgate checks into their workflows using GitHub Actions or pre-commit hooks to maintain dataset integrity as new changes are made.

Example configurations for GitHub Actions and pre-commit hooks are provided, ensuring that users can adopt best practices in maintaining their projects.

Conclusion

ftgate presents a critical suite of tools that enhance the reliability and accuracy of machine learning models post-finetuning, with a focus on ensuring that tool calls remain true to specifications. This makes it an invaluable resource for developers working with fine-tuned models in various applications.