Documentation for qwen 2.5 models Alibaba <article> #quickstart. md # Quickstart This guide helps you quickly start using Qwen2.5. We provide examples of [Hugging Face Transformers](https://github. com /huggingface/transformers) as well as [ModelScope](https://github. com /modelscope/modelscope), and [vLLM](https://github. com /vllm-project/vllm) for deployment. You can find Qwen2.5 models in the [Qwen2.5 collection](https://huggingface. co /collections/Qwen/qwen25-66e81a666513e518adb90d9e) at Hugging Face Hub. ## Hugging Face Transformers & ModelScope To get a quick start with Qwen2.5 , we advise you to try with the inference with `transformers` first. Make sure that you have installed `transformers>=4.37.0`. We advise you to use Python 3.10 or higher, and PyTorch 2.3 or higher. :::{dropdown} Install `transformers` * Install with `pip`: ```bash pip install transformers -U ``` * Install with `conda`: ```bash conda install conda-forge::transformers ``` * Install from source: ```bash pip install git+https://github. com /huggingface/transformers ``` ::: The following is a very simple code snippet showing how to run Qwen2.5 -7B-Instruct: ```python from transformers import AutoModelForCausalLM, AutoTokenizer model_name = "Qwen/Qwen2.5 -7B-Instruct" model = AutoModelForCausalLM. from_pretrained ( model_name, torch_dtype="auto", device_map="auto" ) tokenizer = AutoTokenizer. from_pretrained (model_name) prompt = "Give me a short introduction to large language model." messages = [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt}, ] text = tokenizer. apply_chat_template ( messages, tokenize=False, add_generation_prompt=True, ) model_inputs = tokenizer([text], return_tensors="pt").to(model. device ) generated_ids = model. generate ( **model_inputs, max_new_tokens=512, ) generated_ids = [ output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs. input_ids , generated_ids) ] response = tokenizer. batch_decode (generated_ids, skip_special_tokens=True)[0] ``` As you can see, it's just standard usage for casual LMs in `transformers`! ### Streaming Generation Streaming mode for model chat is simple with the help of `TextStreamer`. Below we show you an example of how to use it: ```python... # Reuse the code before `model. generate ()` in the last code snippet from transformers import TextStreamer streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True) generated_ids = model. generate ( **model_inputs, max_new_tokens=512, streamer=streamer, ) ``` It will print the text to the console or the terminal as being generated. ### ModelScope To tackle with downloading issues, we advise you to try [ModelScope](https://github. com /modelscope/modelscope). Before starting, you need to install `modelscope` with `pip`. `modelscope` adopts a programmatic interface similar (but not identical) to `transformers`. For basic usage, you can simply change the first line of code above to the following: ```python from modelscope import AutoModelForCausalLM, AutoTokenizer ``` For more information, please refer to [the documentation of `modelscope`](https://www. modelscope. cn/docs). ## vLLM for Deployment To deploy Qwen2.5 , we advise you to use vLLM. vLLM is a fast and easy-to-use framework for LLM inference and serving. In the following, we demonstrate how to build a OpenAI-API compatible API service with vLLM. First, make sure you have installed `vllm>=0.4.0`: ```bash pip install vllm ``` Run the following code to build up a vLLM service. Here we take Qwen2.5 -7B-Instruct as an example: ```bash python -m vllm. entrypoints. openai. api_server --model Qwen/Qwen2.5 -7B-Instruct ``` with `vllm>=0.5.3`, you can also use ```bash vllm serve Qwen/Qwen2.5 -7B-Instruct ``` Then, you can use the [create chat interface](https://platform. openai. com/docs/api-reference/chat/completions/create) to communicate with Qwen: ```bash curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen2.5 -7B-Instruct", "messages": [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} ], "temperature": 0.7 , "top_p": 0.8 , "repetition_penalty": 1.05 , "max_tokens": 512 }' ``` or you can use Python client with `openai` Python package as shown below: ```python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai_api_key = "EMPTY" openai_api_base = "http://localhost:8000/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base, ) chat_response = client. chat. completions. create ( model="Qwen/Qwen2.5 -7B-Instruct", messages=[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."}, ], temperature=0.7 , top_p=0.8 , max_tokens=512, extra_body={ "repetition_penalty": 1.05 , }, ) print("Chat response:", chat_response) ``` For more information, please refer to [the documentation of `vllm`](https://docs. vllm. ai/en/stable/). ## Next Step Now, you can have fun with Qwen2.5 models. Would love to know more about its usages? Feel free to check other documents in this documentation. <article> #concepts. md # Key Concepts ## Qwen Qwen (Chinese: 通义千问; pinyin: _Tongyi Qianwen_) is the large language model and large multimodal model series of the Qwen Team, Alibaba Group. Qwen is capable of natural language understanding, text generation, vision understanding, audio understanding, tool use, role play, playing as AI agent, etc. Both language models and multimodal models are pre-trained on large-scale multilingual and multimodal data and post-trained on quality data for aligning to human preferences. There is the proprietary version hosted exclusively at [Alibaba Cloud \[zh\]](https://help. aliyun. com/zh/model-studio/developer-reference/tongyi-qianwen-llm/) and the open-weight version. The spectrum for the open-weight models spans over - Qwen: the language models - [Qwen](https://github. com /QwenLM/Qwen): 1.8B , 7B, 14B, and 72B models - [Qwen1.5 ](https://github. com /QwenLM/Qwen1.5 /tree/v1.5 ): 0.5B , 1.8B , 4B, 14BA2.7B , 7B, 14B, 32B, 72B, and 110B models - [Qwen2](https://github. com /QwenLM/Qwen2/tree/v2.0 ): 0.5B , 1.5B , 7B, 57A14B, and 72B models - [Qwen2.5 ](https://github. com /QwenLM/Qwen2.5 /): 0.5B , 1.5B , 3B, 7B, 14B, 32B, and 72B models - Qwen-VL: the vision-language models - [Qwen-VL](https://github. com /QwenLM/Qwen-VL): 7B-based models - [Qwen2-VL](https://github. com /QwenLM/Qwen2-VL): 2B, 7B, and 72B-based models - Qwen-Audio: the audio-language models - [Qwen-Audio](https://github. com /QwenLM/Qwen-Audio): 7B-based model - [Qwen2-Audio](https://github. com /QwenLM/Qwen2-Audio): 7B-based models - CodeQwen/Qwen-Coder: the language models for coding - [CodeQwen1.5 ](https://github. com /QwenLM/CodeQwen1.5 ): 7B models - [Qwen2.5 -Coder](https://github. com /QwenLM/Qwen2-Coder): 7B models - Qwen-Math: the language models for mathematics - [Qwen2-Math](https://github. com /QwenLM/Qwen2-Math): 1.5B , 7B, and 72B models - [Qwen2.5 -Math](https://github. com /QwenLM/Qwen2.5 -Math): 1.5B , 7B, and 72B models **In this document, our focus is Qwen, the language models.** ## Causal Language Models Causal language models, also known as autoregressive language models or decoder-only language models, are a type of machine learning model designed to predict the next token in a sequence based on the preceding tokens. In other words, they generate text one token at a time, using the previously generated tokens as context. The "causal" aspect refers to the fact that the model only considers the past context (the already generated tokens) when predicting the next token, not any future tokens. Causal language models are widely used for various natural language processing tasks involving text completion and generation. They have been particularly successful in generating coherent and contextually relevant text, making them a cornerstone of modern natural language understanding and generation systems. **Takeaway: Qwen models are causal language models suitable for text completion.** :::{dropdown} Learn more about language models They are three main kinds of models that are commonly referred to as language models in deep learning: - Sequence-to-sequence models: T5 and the likes Sequence-to-sequence models use both an encoder to capture the entire input sequence and a decoder to generate an output sequence. They are widely used for tasks like machine translation, text summarization, etc. - Bidirectional models or encoder-only models: BERT and the likes Bidirectional models can access both past and future context in a sequence during training. They cannot generate sequential outputs in real-time due to the need for future context. They are widely used as embedding models and subsequently used for text classification. - Casual language models or decoder-only models: GPT and the likes Causal language models operate unidirectionally in a strictly forward direction, predicting each subsequent word based only on the previous words in the sequence. This unidirectional nature ensures that the model's predictions do not rely on future context, making them suitable for tasks like text completion and generation. ::: ### Pre-training & Base models Base language models are foundational models trained on extensive corpora of text to predict the next word in a sequence. Their main goal is to capture the statistical patterns and structures of language, enabling them to generate coherent and contextually relevant text. These models are versatile and can be adapted to various natural language processing tasks through fine-tuning. While adept at producing fluent text, they may require in-context learning or additional training to follow specific instructions or perform complex reasoning tasks effectively. For Qwen models, the base models are those without "-Instruct" indicators, such as Qwen2.5 -7B and Qwen2.5 -72B. **Takeaway: Use base models for in-context learning, downstream fine-tuning, etc.** ### Post-training & Instruction-tuned models Instruction-tuned language models are specialized models designed to understand and execute specific instructions in conversational styles. These models are fine-tuned to interpret user commands accurately and can perform tasks such as summarization, translation, and question answering with improved accuracy and consistency. Unlike base models, which are trained on large corpora of text, instruction-tuned models undergo additional training using datasets that contain examples of instructions and their desired outcomes, often in multiple turns. This kind of training makes them ideal for applications requiring targeted functionalities while maintaining the ability to generate fluent and coherent text. For Qwen models, the instruction-tuned models are those with the "-Instruct" suffix, such as Qwen2.5 -7B-Instruct and Qwen2.5 -72B-Instruct. [^instruct-chat] **Takeaway: Use instruction-tuned models for conducting tasks in conversations, downstream fine-tuning, etc.** [^instruct-chat]: Previously, they are known as the chat models and with the "-Chat" suffix. Starting from Qwen2, the name is changed to follow the common practice. For Qwen, "-Instruct" and "-Chat" should be regarded as synonymous. ## Tokens & Tokenization Tokens represent the fundamental units that models process and generate. They can represent texts in human languages (regular tokens) or represent specific functionality like keywords in programming languages (control tokens [^special]). Typically, a tokenizer is used to split text into regular tokens, which can be words, subwords, or characters depending on the specific tokenization scheme employed, and furnish the token sequence with control tokens as needed. The vocabulary size, or the total number of unique tokens a model recognizes, significantly impacts its performance and versatility. Larger language models often use sophisticated tokenization methods to handle the vast diversity of human language while keeping the vocabulary size manageable. Qwen use a relatively large vocabulary of 151,646 tokens in total. [^special]: Control tokens can be called special tokens. However, the meaning of special tokens need to be interpreted based on the contexts: special tokens may contain extra regular tokens. **Takeaway: Tokenization method and vocabulary size is important.** ### Byte-level Byte Pair Encoding Qwen adopts a subword tokenization method called Byte Pair Encoding (BPE), which attempts to learn the composition of tokens that can represent the text with the fewest tokens. For example, the string " tokenization" is decomposed as " token" and "ization" (note that the space is part of the token). Especially, the tokenization of Qwen ensures that there is no unknown words and all texts can be transformed to token sequences. There are 151,643 tokens as a result of BPE in the vocabulary of Qwen, which is a large vocabulary efficient for diverse languages. As a rule of thumb, 1 token is 3~4 characters for English texts and 1.5 ~1.8 characters for Chinese texts. **Takeaway: Qwen processes texts in subwords and there are no unknown words.** :::{dropdown} Learn more about tokenization in Qwen Qwen uses byte-level BPE (BBPE) on UTF-8 encoded texts. It starts by treating each byte as a token and then iteratively merges the most frequent pairs of tokens occurring the texts into larger tokens until the desired vocabulary size is met. In byte-level BPE, minimum 256 tokens are needed to tokenize every piece of text and avoid the out of vocabulary (OOV) problem. In comparison, character-level BPE needs every Unicode character in its vocabulary to avoid OOV and the Unicode Standard contains 154,998 characters as of Unicode Version 16.0. One limitation to keep in mind for byte-level BPE is that the individual tokens in the vocabulary may not be seemingly semantically meaningful or even valid UTF-8 byte sequences, and in certain aspects, they should be viewed as a text compression scheme. ::: ### Control Tokens & Chat Template Control tokens and chat templates both serve as mechanisms to guide the model's behavior and outputs. Control tokens are special tokens inserted into the sequence that signifies meta information. For example, in pre-training, multiple documents may be packed into a single sequence. For Qwen, the control token "<|endoftext|>" is inserted after each document to signify that the document has ended and a new document will proceed. Chat templates provide a structured format for conversational interactions, where predefined placeholders or prompts are used to elicit responses from the model that adhere to a desired dialogue flow or context. Different models may use different kinds of chat template to format the conversations. It is crucial to use the designated one to ensure the precise control over the LLM's generation process. Qwen uses the following format (ChatML[^chatml]), making use of control tokens to format each turn in the conversations ```text <|im_start|>{{role}} {{content}}<|im_end|> ``` The user input take the role of `user` and the model generation takes the role of `assistant`. Qwen also supports the meta message that instruct the model to perform specific actions or generate text with certain characteristics, such as altering tone, style, or content, which takes the role of `system` and the content defaults to "You are Qwen, created by Alibaba Cloud. You are a helpful assistant." The following is a full example: ```text <|im_start|>system You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|> <|im_start|>user hello<|im_end|> <|im_start|>assistant Hello! How can I assist you today? Let me know if you have any questions or need help with something. I'm here to help!<|im_end|> <|im_start|>user Explain large language models like I'm 5.<|im_end|> <|im_start|>assistant Sure, I'd be happy to explain large language models in a simple way! Imagine that you have a friend who is really good at playing with words. This friend has read so many books and talked to so many people that they know how to use words in all sorts of different ways. They can tell stories, answer questions, and even make up new words if they need to. Now imagine that this friend is actually a computer program, called a "large language model". It's been trained on lots and lots of text, like books, articles, and conversations, so it knows how to use words just like your word-loving friend does. When you ask the model a question or give it a task, it uses all the knowledge it's learned to come up with a response that makes sense. Just like your friend might sometimes make mistakes or say things in a funny way, the large language model isn't perfect either. But it's still really impressive, because it can understand and generate human language in a way that was once thought impossible for machines!<|im_end|><|endoftext|> ``` Starting from Qwen2.5 , the Qwen model family including multimodal and specialized models will use a unified vocabulary, which contains control tokens from all subfamilies. There are 22 control tokens in the vocabulary of Qwen2.5 , making the vocabulary size totaling 151,665: - 1 general: `<|endoftext|>` - 2 for chat: `<|im_start|>` and `<|im_end|>` - 2 for tool use: `<tool_call>` and `</tool_call>` - 11 for vision - 6 for coding **Takeaway: Qwen uses ChatML with control tokens for chat template.** [^chatml]: For historical reference only, ChatML is first described by the OpenAI Python SDK. The last available version is [this](https://github. com /openai/openai-python/blob/v0.28.1/chatml. md ). Please also be aware that that document lists use cases intended for OpenAI models. For Qwen2.5 models, please only use as in our guide. ## Length Limit As Qwen models are causal language models, in theory there is only one length limit of the entire sequence. However, since there is often packing in training and each sequence may contain multiple individual pieces of texts. **How long the model can generate or complete ultimately depends on the use case and in that case how long each document (for pre-training) or each turn (for post-training) is in training.** For Qwen2.5 , the packed sequence length in training is 32,768 tokens.[^yarn] The maximum document length in pre-training is this length. The maximum message length for user and assistant is different in post-training. In general, the assistant message could be up to 8192 tokens. [^yarn]: The sequence length can be extended to 131,072 tokens for Qwen2.5 -7B, Qwen2.5 -14B, Qwen2.5 -32B, and Qwen2.5 -72B models with YaRN. Please refer to the model card on how to enable YaRN in vLLM. **Takeaway: Qwen2.5 models can process texts of 32K or 128K tokens and up to 8K tokens can be assistant output.** <article> #function_call. md --- myst: number_code_blocks: ["python3"] --- # Function Calling ## Preface Function calling with large language models is a huge and evolving topic. It is particularly important for AI applications: - either for AI-native applications that strive to work around the shortcomings of current AI technology, - or for existing applications that seeks the integration of AI technology to improve performance, user interaction and experience, or efficiency. This guide will not delve into those discussions or which role an LLM should play in an application and the related best practice. Those views are reflected in the design of AI application frameworks: from LangChain to LlamaIndex to QwenAgent. Instead, we will talk about how Qwen2.5 can be used to support function calling and how it can be used to achieve your goals, from the inference usage for developing application to the inner workings for hardcore customizations. In this guide, - We will first demonstrate how to use function calling with Qwen2.5. - Then, we will introduce the technical details on functional calling with Qwen2.5 , which are mainly about the templates. Before starting, there is one thing we have not yet introduced, that is... ## What is function calling? :::{Note} There is another term "tool use" that may be used to refer to the same concept. While some may argue that tools are a generalized form of functions, at present, their difference exists only technically as different I/O types of programming interfaces. ::: Large language models (LLMs) are powerful things. However, sometimes LLMs by themselves are simply not capable enough. - On the one hand, LLMs have inherent modeling limitations. For one, they do not know things that are not in their training data, which include those happened after their training ended. In addition, they learn things in the way of likelihood, which suggests that they may not be precise enough for tasks with fixed rule sets, e. g., mathematical computation. - On the other hand, it is not easy to use LLMs as a Plug-and-Play service programmatically with other things. LLMs mostly talk in words that are open to interpretation and thus ambiguous, while other software or applications or systems talk in code and through programming interfaces that are pre-defined and fixed and structured. To this end, function calling establishes a common protocol that specifies how LLMs should interact with the other things. The procedure is mainly as follows: 1. The application provides a set of functions and the instructions of the functions to an LLM. 2. The LLM choose to or not to, or is forced to use one or many of the functions, in response to user queries. 3. If the LLM chooses to use the functions, it states how the functions should be used based on the function instructions. 4. The chosen functions are used as such by the application and the results are obtained, which are then given to the LLM if further interaction is needed. They are many ways for LLMs to understand and follow this protocol. As always, the key is prompt engineering or an internalized template known by the model. Qwen2.5 were pre-trained with various types of templates that could support function calling, so that users can directly make use of this procedure. ## Inference with Function Calling :::{note} Please be aware that the inference usage is subject to change as the frameworks and the Qwen models evolve. ::: As function calling is essentially implemented using prompt engineering, you could manually construct the model inputs for Qwen2 models. However, frameworks with function calling support can help you with all that laborious work. In the following, we will introduce the usage (via dedicated function calling chat template) with - **Qwen-Agent**, - **Hugging Face transformers**, - **Ollama**, and - **vLLM**. If you are familiar with the usage of OpenAI API, you could also directly use the OpenAI-compatible API services for Qwen2.5. However, not all of them support function calling for Qwen2.5. Currently, supported solutions include the self-hosted service by [Ollama](https://github. com /ollama/ollama/blob/main/docs/openai. md ) or [vLLM](https://docs. vllm. ai/en/stable/serving/openai_compatible_server. html #tool-calling-in-the-chat-completion-api) and the cloud service of [ModelStudio \[zh\]](https://help. aliyun. com/zh/model-studio/developer-reference/compatibility-of-openai-with-dashscope#97e2b45391x08). If you are familiar with application frameworks, e. g., LangChain, you can also use function calling abilities in Qwen2.5 via ReAct Prompting. ### The Example Case Let's also use an example to demonstrate the inference usage. We assume **Python 3.11 ** is used as the programming language. **Scenario**: Suppose we would like to ask the model about the temperature of a location. Normally, the model would reply that it cannot provide real-time information. But we have two tools that can be used to obtain the current temperature of and the temperature at a given date of a city respectively, and we would like the model to make use of them. To set up the example case, you can use the following code: :::{dropdown} Preparation Code :name: prepcode ```python import json def get_current_temperature(location: str, unit: str = "celsius"): """Get current temperature at a location. Args: location: The location to get the temperature for, in the format "City, State, Country". unit: The unit to return the temperature in. Defaults to "celsius". (choices: ["celsius", "fahrenheit"]) Returns: the temperature, the location, and the unit in a dict """ return { "temperature": 26.1 , "location": location, "unit": unit, } def get_temperature_date(location: str, date: str, unit: str = "celsius"): """Get temperature at a location and date. Args: location: The location to get the temperature for, in the format "City, State, Country". date: The date to get the temperature for, in the format "Year-Month-Day". unit: The unit to return the temperature in. Defaults to "celsius". (choices: ["celsius", "fahrenheit"]) Returns: the temperature, the location, the date and the unit in a dict """ return { "temperature": 25.9 , "location": location, "date": date, "unit": unit, } def get_function_by_name(name): if name == "get_current_temperature": return get_current_temperature if name == "get_temperature_date": return get_temperature_date TOOLS = [ { "type": "function", "function": { "name": "get_current_temperature", "description": "Get current temperature at a location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": 'The location to get the temperature for, in the format "City, State, Country".', }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": 'The unit to return the temperature in. Defaults to "celsius".', }, }, "required": ["location"], }, }, }, { "type": "function", "function": { "name": "get_temperature_date", "description": "Get temperature at a location and date.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": 'The location to get the temperature for, in the format "City, State, Country".', }, "date": { "type": "string", "description": 'The date to get the temperature for, in the format "Year-Month-Day".', }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": 'The unit to return the temperature in. Defaults to "celsius".', }, }, "required": ["location", "date"], }, }, }, ] MESSAGES = [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant. Current Date: 2024-09-30"}, {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow?"}, ] ``` ::: In particular, the tools should be described using JSON Schema and the messages should contain as much available information as possible. You can find the explanations of the tools and messages below: :::{dropdown} Example Tools The tools should be described using the following JSON: ```json [ { "type": "function", "function": { "name": "get_current_temperature", "description": "Get current temperature at a location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The location to get the temperature for, in the format \"City, State, Country\"." }, "unit": { "type": "string", "enum": [ "celsius", "fahrenheit" ], "description": "The unit to return the temperature in. Defaults to \"celsius\"." } }, "required": [ "location" ] } } }, { "type": "function", "function": { "name": "get_temperature_date", "description": "Get temperature at a location and date.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The location to get the temperature for, in the format \"City, State, Country\"." }, "date": { "type": "string", "description": "The date to get the temperature for, in the format \"Year-Month-Day\"." }, "unit": { "type": "string", "enum": [ "celsius", "fahrenheit" ], "description": "The unit to return the temperature in. Defaults to \"celsius\"." } }, "required": [ "location", "date" ] } } } ] ``` For each **tool**, it is a JSON object with two fields: - `type`: a string specifying the type of the tool, currently only `"function"` is valid - `function`: an object detailing the instructions to use the function For each **function**, it is a JSON object with three fields: - `name`: a string indicating the name of the function - `description`: a string describing what the function is used for - `parameters`: [a JSON Schema](https://json-schema. org /learn/getting-started-step-by-step) that specifies the parameters the function accepts. Please refer to the linked documentation for how to compose a JSON Schema. Notable fields include `type`, `required`, and `enum`. Most frameworks use the tool format and some may use the function format. Which one to use should be obvious according to the naming. ::: :::{dropdown} Example Messages Our query is `What's the temperature in San Francisco now? How about tomorrow?`. Since the model does not know what the current date is, let alone tomorrow, we should provide the date in the inputs. Here, we decide to supply that information in the system message after the default system message `You are Qwen, created by Alibaba Cloud. You are a helpful assistant.`. You could append the date to user message in your application code. ```json [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant. Current Date: 2024-09-30"}, {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow?"} ] ``` ::: ### Qwen-Agent [Qwen-Agent](https://github. com /QwenLM/Qwen-Agent) is actually a Python Agent framework for developing AI applications. Although its intended use cases are higher-level than efficient inference, it does contain the **canonical implementation** of function calling for Qwen2.5. It provides the function calling ability for Qwen2.5 to an OpenAI-compatible API through templates that is transparent to users. {#note-official-template} It's worth noting that since a lot of stuff can be done under the scene with application frameworks, currently the official function calling implementation for Qwen2.5 is very flexible and beyond simple templating, making it hard to adapt it other frameworks that use less capable templating engines. Before starting, let's make sure the latest library is installed: ```bash pip install -U qwen-agent ``` For this guide, we are at version v0.0.10. #### Preparing Qwen-Agent can wrap an OpenAI-compatible API that does not support function calling. You can serve such an API with most inference frameworks or obtain one from cloud providers like DashScope or Together. Assuming there is an OpenAI-compatible API at `http://localhost:8000/v1`, Qwen-Agent provides a shortcut function `get_chat_model` to obtain a model inference class with function calling support: ```python from qwen_agent. llm import get_chat_model llm = get_chat_model({ "model": "Qwen/Qwen2.5 -7B-Instruct", "model_server": "http://localhost:8000/v1", "api_key": "EMPTY", }) ``` In the above, `model_server` is the `api_base` common used in other OpenAI-compatible API clients. It is advised to provide the `api_key` (but not via plaintext in the code), even if the API server does not check it, in which case, you can set it to anything. For model inputs, the common message structure for system, user, and assistant history should be used: ```python messages = MESSAGES[:] # [ # {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant. Current Date: 2024-09-30"}, # {"role": "user", "content": "What's the temperature in San Francisco now? How about tomorrow?"} # ] ``` We add the current date to the system message so that the "tomorrow" in the user message is anchored. It can also be added to the user message if one desires. At the time, Qwen-Agent works with functions instead of tools. This requires a small change to our tool descriptions, that is, extracting the function fields: ```python functions = [tool["function"] for tool in TOOLS] ``` #### Tool Calls and Tool Results To interact with the model, the `chat` method should be used: ```python for responses in llm. chat ( messages=messages, functions=functions, extra_generate_cfg=dict(parallel_function_calls=True), ): pass messages. extend (responses) ``` In the above code, the `chat` method receives the `messages`, the `functions`, and an `extra_generate_cfg` parameter. You can put sampling parameters, such as `temperature`, and `top_p`, in the `extra_generate_cfg`. Here, we add to it a special control `parallel_function_calls` provided by Qwen-Agent. As its name suggests, it will enable parallel function calls, which means that the model may generate multiple function calls for a single turn as it deems fit. The `chat` method returns a generator of list, each of which may contain multiple messages. Since we enable `parallel_function_calls`, we should get two messages in the responses: ```python [ {'role': 'assistant', 'content': '', 'function_call': {'name': 'get_current_temperature', 'arguments': '{"location": "San Francisco, CA, USA", "unit": "celsius"}'}}, {'role': 'assistant', 'content': '', 'function_call': {'name': 'get_temperature_date', 'arguments': '{"location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}'}}, ] ``` As we can see, Qwen-Agent attempts to parse the model generation in an easier to use structural format. The details related to function calls are placed in the `function_call` field of the messages: - `name`: a string representing the function to call - `arguments`: a JSON-formatted string representing the arguments the function should be called with Note that Qwen2.5 -7B-Instruct is quite capable: - It has followed the function instructions to add the state and the country to the location. - It has correctly induced the date of tomorrow and given in the format required by the function. Then comes the critical part -- checking and applying the function call: ```python3 for message in responses: if fn_call := message. get ("function_call", None): fn_name: str = fn_call['name'] fn_args: dict = json. loads (fn_call["arguments"]) fn_res: str = json. dumps (get_function_by_name(fn_name)(**fn_args)) messages. append ({ "role": "function", "name": fn_name, "content": fn_res, }) ``` To get tool results: - line 1: We should iterate the function calls in the order the model generates them. - line 2: We can check if a function call is needed as deemed by the model by checking the `function_call` field of the generated messages. - line 3-4: The related details including the name and the arguments of the function can also be found there, which are `name` and `arguments` respectively. - line 6: With the details, one should call the function and obtain the results. Here, we assume there is a function named [`get_function_by_name`](#prepcode) to help us get the related function by its name. - line 8-12: With the result obtained, add the function result to the messages as `content` and with `role` as `"function"`. Now the messages are ```python [ {'role': 'system', 'content': 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant. Current Date: 2024-09-30'}, {'role': 'user', 'content': "What's the temperature in San Francisco now? How about tomorrow?"}, {'role': 'assistant', 'content': '', 'function_call': {'name': 'get_current_temperature', 'arguments': '{"location": "San Francisco, CA, USA", "unit": "celsius"}'}}, {'role': 'assistant', 'content': '', 'function_call': {'name': 'get_temperature_date', 'arguments': '{"location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}'}}, {'role': 'function', 'name': 'get_current_temperature', 'content': '{"temperature": 26.1 , "location": "San Francisco, CA, USA", "unit": "celsius"}'}, {'role': 'function', 'name': 'get_temperature_date', 'content': '{"temperature": 25.9 , "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}'}, ] ``` #### Final Response Finally, run the model again to get the final model results: ```python for responses in llm. chat (messages=messages, functions=functions): pass messages. extend (responses) ``` The final response should be like ```python {'role': 'assistant', 'content': 'Currently, the temperature in San Francisco is approximately 26.1 °C. Tomorrow, on 2024-10-01, the temperature is forecasted to be around 25.9 °C.'} ``` ### Hugging Face transformers Since function calling is based on prompt engineering and templates, `transformers` supports it with its tokenizer utilities, in particular, the `tokenizer. apply_chat_template ` method, which hides the sophistication of constructing the model inputs, using the Jinja templating engine. However, it means that users should handle the model output part on their own, which includes parsing the generated function call message. The blog piece [_Tool Use, Unified_](https://huggingface. co /blog/unified-tool-use) is very helpful in understanding its design. Be sure to take a look. Tool use API is available in transformers since v4.42.0. Before starting, let's check that: ```bash pip install "transformers>4.42.0" ``` For this guide, we are at version v4.44.2. #### Preparing For Qwen2.5 , the chat template in `tokenizer_config. json ` has already included support for the Hermes-style tool use. We simply need to load the model and the tokenizer: ```python from transformers import AutoModelForCausalLM, AutoTokenizer model_name_or_path = "Qwen/Qwen2.5 -7B-Instruct" tokenizer = AutoTokenizer. from_pretrained (model_name_or_path) model = AutoModelForCausalLM. from_pretrained ( model_name_or_path, torch_dtype="auto", device_map="auto", ) ``` The inputs are the same with those in [the preparation code](#prepcode): ```python tools = TOOLS messages = MESSAGES[:] ``` In `transformers`, you can also directly use Python functions as tools with certain constraints[^get_json_schema_note]: ```python tools = [get_current_temperature, get_temperature_date] ``` [^get_json_schema_note]: `transformers` will use `transformers. utils. get_json_schema` to generate the tool descriptions from Python functions. There are some gotchas with `get_json_schema`, and it is advised to check [its doc \[v4.44.2\]](https://github. com /huggingface/transformers/blob/v4.44.2/src/transformers/utils/chat_template_utils. py #L183-L288) before relying on it. - The function should use Python type hints for parameter types and has a Google-style docstring for function description and parameter descriptions. - Supported types are limited, since the types needs to be mapped to JSON Schema. In particular, `typing. Literal ` is not supported. You can instead add `(choices:...)` at the end of a parameter description, which will be mapped to a `enum` type in JSON Schema. Please be aware that all the returned results in the examples in the linked docstring are actually the content of the `function` field in the actual returned results. #### Tool Calls and Tool Results To construct the input sequence, we should use the `apply_chat_template` method and then let the model continue the texts: ```python text = tokenizer. apply_chat_template (messages, tools=tools, add_generation_prompt=True, tokenize=False) inputs = tokenizer(text, return_tensors="pt").to(model. device ) outputs = model. generate (**inputs, max_new_tokens=512) output_text = tokenizer. batch_decode (outputs)[0][len(text):] ``` The output texts should be like ```text <tool_call> {"name": "get_current_temperature", "arguments": {"location": "San Francisco, CA, USA"}} </tool_call> <tool_call> {"name": "get_temperature_date", "arguments": {"location": "San Francisco, CA, USA", "date": "2024-10-01"}} </tool_call><|im_end|> ``` Now we need to do two things: 1. Parse the generated tool calls to a message and add them to the messages, so that the model knows which tools are used. 2. Obtain the results of the tools and add them to the messages, so that the model knows the results of the tool calls. In `transformers`, the tool calls should be a field of assistant messages. Let's use a simple function called `try_parse_tool_calls` to parse the tool calls: {#parse-function} ```python import re def try_parse_tool_calls(content: str): """Try parse the tool calls.""" tool_calls = [] offset = 0 for i, m in enumerate(re. finditer (r"<tool_call>\n(.+)?\n</tool_call>", content)): if i == 0: offset = m. start () try: func = json. loads (m. group (1)) tool_calls. append ({"type": "function", "function": func}) if isinstance(func["arguments"], str): func["arguments"] = json. loads (func["arguments"]) except json. JSONDecodeError as e: print(f"Failed to parse tool calls: the content is {m. group (1)} and {e}") pass if tool_calls: if offset > 0 and content[:offset].strip(): c = content[:offset] else: c = "" return {"role": "assistant", "content": c, "tool_calls": tool_calls} return {"role": "assistant", "content": re. sub (r"<\|im_end\|>$", "", content)} ``` This function does not cover all possible scenarios and thus is prone to errors. But it should suffice for the purpose of this guide. :::{note} The template in the `tokenizer_config. json ` assumes that the generated content alongside tool calls is in the same message instead of separate assistant messages, e. g., ```json { "role": "assistant", "content": "To obtain the current temperature, I should call the functions `get_current_temperate`.", "tool_calls": [ {"type": "function", "function": {"name": "get_current_temperature", "arguments": {"location": "San Francisco, CA, USA", "unit": "celsius"}}} ] } ``` instead of ```json [ { "role": "assistant", "content": "To obtain the current temperature, I should call the functions `get_current_temperate`.", }, { "role": "assistant", "content": "", "tool_calls": [ {"type": "function", "function": {"name": "get_current_temperature", "arguments": {"location": "San Francisco, CA, USA", "unit": "celsius"}}} ] } ] ``` This is implemented roughly in `try_parse_tool_calls` but keep that in mind if you are writing your own tool call parser. ::: ```python messages. append (try_parse_tool_calls(output_text)) if tool_calls := messages[-1].get("tool_calls", None): for tool_call in tool_calls: if fn_call := tool_call. get ("function"): fn_name: str = fn_call["name"] fn_args: dict = fn_call["arguments"] fn_res: str = json. dumps (get_function_by_name(fn_name)(**fn_args)) messages. append ({ "role": "tool", "name": fn_name, "content": fn_res, }) ``` The messages now should be like ```python [ {'role': 'system', 'content': 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant. Current Date: 2024-09-30'}, {'role': 'user', 'content': "What's the temperature in San Francisco now? How about tomorrow?"}, {'role': 'assistant', 'content': '', 'tool_calls': [ {'type': 'function', 'function': {'name': 'get_current_temperature', 'arguments': {'location': 'San Francisco, CA, USA'}}}, {'type': 'function', 'function': {'name': 'get_temperature_date', 'arguments': {'location': 'San Francisco, CA, USA', 'date': '2024-10-01'}}}, ]}, {'role': 'tool', 'name': 'get_current_temperature', 'content': '{"temperature": 26.1 , "location": "San Francisco, CA, USA", "unit": "celsius"}'}, {'role': 'tool', 'name': 'get_temperature_date', 'content': '{"temperature": 25.9 , "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}'}, ] ``` The messages are similar to those of Qwen-Agent, but there are some major differences: - Tools instead of functions - Parallel calls are by default - Multiple tool calls as a list in a single assistant message, instead of multiple messages. - The function arguments are parsed into a dict if it is a valid JSON-formatted string. #### Final Response Then it's time for the model to generate the actual response for us based on the tool results. Let's query the model again: ```python text = tokenizer. apply_chat_template (messages, tools=tools, add_generation_prompt=True, tokenize=False) inputs = tokenizer(text, return_tensors="pt").to(model. device ) outputs = model. generate (**inputs, max_new_tokens=512) output_text = tokenizer. batch_decode (outputs)[0][len(text):] ``` The output_text should be like ``` The current temperature in San Francisco is approximately 26.1 °C. Tomorrow, on October 1, 2024, the temperature is expected to be around 25.9 °C.<|im_end|> ``` Add the result text as an assistant message and the final messages should be ready for further interaction: ```python messages. append (try_parse_tool_calls(output_text)) ``` ### Ollama Ollama is a set of tools for serving LLMs locally. It also relies on its template implementation to support function calling. Different from transformers, which is written in Python and uses the Jinja template whose syntax is heavily inspired by Django and Python, Ollama, which is mostly written in Go, uses Go's [text/template](https://pkg. go. dev/text/template) packages. In addition, Ollama implements internally a helper function so that it can automatically parse the generated tool calls in texts to structured messages if the format supported. You could check the [Tool support](https://ollama. com /blog/tool-support) blog post first. Tool support has been available in Ollama since v0.3.0. You can run the following to check the Ollama version: ```bash ollama -v ``` If lower than expected, follow [the official instructions](https://ollama. com /download) to install the latest version. In this guide, we will aslo use [ollama-python](https://github. com /ollama/ollama-python), before starting, make sure it is available in your environment: ```bash pip install ollama ``` For this guide, the `ollama` binary is at v0.3.9 and the `ollama` Python library is at v0.3.2. #### Preparing The messages structure used in Ollama is the same with that in `transformers` and the template in [Qwen2.5 Ollama models](https://ollama. com /library/qwen2.5 ) has supported tool use. The inputs are the same with those in [the preparation code](#prepcode): ```python tools = TOOLS messages = MESSAGES[:] model_name = "qwen2.5 :7b" ``` Note that you cannot pass Python functions as tools directly and `tools` has to be a `dict`. #### Tool Calls and Tool Results We can use the `ollama. chat ` method to directly query the underlying API: ```python import ollama response = ollama. chat ( model=model_name, messages=messages, tools=tools, ) ``` The main fields in the response could be: ```python { 'model': 'qwen2.5 :7b', 'message': { 'role': 'assistant', 'content': '', 'tool_calls': [ {'function': {'name': 'get_current_temperature', 'arguments': {'location': 'San Francisco, CA, USA'}}}, {'function': {'name': 'get_temperature_date', 'arguments': {'date': '2024-10-01', 'location': 'San Francisco, CA, USA'}}}, ], }, } ``` Ollama's tool call parser has succeeded in parsing the tool results. If not, you may refine [the `try_parse_tool_calls` function above](#parse-function). Then, we can obtain the tool results and add them to the messages. The following is basically the same with `transformers`: ```python messages. append (response["message"]) if tool_calls := messages[-1].get("tool_calls", None): for tool_call in tool_calls: if fn_call := tool_call. get ("function"): fn_name: str = fn_call["name"] fn_args: dict = fn_call["arguments"] fn_res: str = json. dumps (get_function_by_name(fn_name)(**fn_args)) messages. append ({ "role": "tool", "name": fn_name, "content": fn_res, }) ``` The messages are now like ```python [ {'role': 'system', 'content': 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant. Current Date: 2024-09-30'}, {'role': 'user', 'content': "What's the temperature in San Francisco now? How about tomorrow?"}, {'role': 'assistant', 'content': '', 'tool_calls': [ {'function': {'name': 'get_current_temperature', 'arguments': {'location': 'San Francisco, CA, USA'}}}, {'function': {'name': 'get_temperature_date', 'arguments': {'date': '2024-10-01', 'location': 'San Francisco, CA, USA'}}}, ]}, {'role': 'tool', 'name': 'get_current_temperature', 'content': '{"temperature": 26.1 , "location": "San Francisco, CA, USA", "unit": "celsius"}'}, {'role': 'tool', 'name': 'get_temperature_date', 'content': '{"temperature": 25.9 , "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}'}, ] ``` #### Final Response The rest are easy: ```python response = ollama. chat ( model=model_name, messages=messages, tools=tools, ) messages. append (response["message"]) ``` The final message should be like the following: ```python {'role': 'assistant', 'content': 'The current temperature in San Francisco is approximately 26.1 °C. For tomorrow, October 1st, 2024, the forecasted temperature will be around 25.9 °C.'} ``` (heading-target)= ### vLLM vLLM is a fast and easy-to-use library for LLM inference and serving. It uses the tokenizer from `transformers` to format the input, so we should have no trouble preparing the input. In addition, vLLm also implements helper functions so that generated tool calls can be parsed automatically if the format is supported. Tool support has been available in `vllm` since v0.6.0. Be sure to install a version that supports tool use. For more information, check the [vLLM documentation](https://docs. vllm. ai/en/stable/serving/openai_compatible_server. html #tool-calling-in-the-chat-completion-api). For this guide, we are at version v0.6.1.post2. We will use the OpenAI-Compatible API by `vllm` with the API client from the `openai` Python library. #### Preparing For Qwen2.5 , the chat template in tokenizer_config. json has already included support for the Hermes-style tool use. We simply need to start a OpenAI-compatible API with vLLM: ```bash vllm serve Qwen/Qwen2.5 -7B-Instruct --enable-auto-tool-choice --tool-call-parser hermes ``` The inputs are the same with those in [the preparation code](#prepcode): ```python tools = TOOLS messages = MESSAGES[:] ``` Let's also initialize the client: ```python from openai import OpenAI openai_api_key = "EMPTY" openai_api_base = "http://localhost:8000/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base, ) model_name = "Qwen/Qwen2.5 -7B-Instruct" ``` #### Tool Calls and Tool Results We can use the create chat completions endpoint to query the model: ```python response = client. chat. completions. create ( model=model_name, messages=messages, tools=tools, temperature=0.7 , top_p=0.8 , max_tokens=512, extra_body={ "repetition_penalty": 1.05 , }, ) ``` vLLM should be able to parse the tool calls for us, and the main fields in the response (`response. choices [0]`) should be like ```python Choice( finish_reason='tool_calls', index=0, logprobs=None, message=ChatCompletionMessage( content=None, role='assistant', function_call=None, tool_calls=[ ChatCompletionMessageToolCall( id='chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15', function=Function(arguments='{"location": "San Francisco, CA, USA"}', name='get_current_temperature'), type='function', ), ChatCompletionMessageToolCall( id='chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501', function=Function(arguments='{"location": "San Francisco, CA, USA", "date": "2024-10-01"}', name='get_temperature_date'), type='function', ), ], ), stop_reason=None, ) ``` Note that the function arguments are JSON-formatted strings, which Qwen-Agent follows but `transformers` and Ollama differs. As before, chances are that there are corner cases where tool calls are generated but they are malformed and cannot be parsed. For production code, we should try parsing by ourselves. Then, we can obtain the tool results and add them to the messages as shown below: ```python messages. append (response. choices [0].message. model_dump ()) if tool_calls := messages[-1].get("tool_calls", None): for tool_call in tool_calls: call_id: str = tool_call["id"] if fn_call := tool_call. get ("function"): fn_name: str = fn_call["name"] fn_args: dict = json. loads (fn_call["arguments"]) fn_res: str = json. dumps (get_function_by_name(fn_name)(**fn_args)) messages. append ({ "role": "tool", "content": fn_res, "tool_call_id": call_id, }) ``` It should be noted that the OpenAI API uses `tool_call_id` to identify the relation between tool results and tool calls. The messages are now like ```python [ {'role': 'system', 'content': 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant. Current Date: 2024-09-30'}, {'role': 'user', 'content': "What's the temperature in San Francisco now? How about tomorrow?"}, {'content': None, 'role': 'assistant', 'function_call': None, 'tool_calls': [ {'id': 'chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15', 'function': {'arguments': '{"location": "San Francisco, CA, USA"}', 'name': 'get_current_temperature'}, 'type': 'function'}, {'id': 'chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501', 'function': {'arguments': '{"location": "San Francisco, CA, USA", "date": "2024-10-01"}', 'name': 'get_temperature_date'}, 'type': 'function'}, ]}, {'role': 'tool', 'content': '{"temperature": 26.1 , "location": "San Francisco, CA, USA", "unit": "celsius"}', 'tool_call_id': 'chatcmpl-tool-924d705adb044ff88e0ef3afdd155f15'}, {'role': 'tool', 'content': '{"temperature": 25.9 , "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"}', 'tool_call_id': 'chatcmpl-tool-7e30313081944b11b6e5ebfd02e8e501'}, ] ``` #### Final Response Let's call the endpoint again to seed the tool results and get response: ```python response = client. chat. completions. create ( model=model_name, messages=messages, tools=tools, temperature=0.7 , top_p=0.8 , max_tokens=512, extra_body={ "repetition_penalty": 1.05 , }, ) messages. append (response. choices [0].message. model_dump ()) ``` The final response (`response. choices [0].message. content `) should be like ```text The current temperature in San Francisco is approximately 26.1 °C. For tomorrow, the forecasted temperature is around 25.9 °C. ``` ### Discussions Now, we have introduced how to conduct inference with function calling using Qwen2 in three different frameworks! Let's make a brief comparison. | Item | OpenAI API | Hugging Face transformers | Ollama | vLLM | Qwen-Agent | | :----- | :---: | :---: | :---: | :---: | :---: | | Type | HTTP API | Python Library | HTTP API | HTTP API | Python Library | | Inference Backend | - | PyTorch | llama. cpp | PyTorch | HTTP API | | Templating Backend | - | Jinja | Go `text/template` | Jinja | Python | | Tools/Functions | Tools | Tools | Tools | Tools | Functions | | Parallel Calls | Default Yes (Configurable) | Yes | Yes | Yes | Default No (Configurable) | | Call Format | Single assistant message with `tool_calls` | Single assistant message with `tool_calls` | Single assistant message with `tool_calls` | Single assistant message with `tool_calls` | Multiple assistant messages with `function_call` | | Call Argument Format | string | object | object | string | string | | Call Result Format | Multiple tool messages with `content` | Multiple tool messages with `content` | Multiple tool messages with `content` | Multiple tool messages with `content` | Multiple function messages with `content` | There are some details not shown in the above table: - OpenAI API comes with Python, Node. js , Go, and. NET SDKs. It also follows the OpenAPI standard. - Ollama comes with Python and Node. js SDKs. It has OpenAI-compatible API at a different base url that can be accessed using OpenAI API SDK. - Qwen-Agent as an application framework can call the tools automatically for you, which is introduced in [the Qwen-Agent guide](./qwen_agent). In addition, there are more on the model side of function calling, which means you may need to consider more things in production code: - **Accuracy of function calling**: When it comes to evaluate the accuracy of function calling, there are two aspects: (a) whether the correct functions (including no functions) are selected and (b) whether the correct function arguments are generated. It is not always the case that Qwen2.5 will be accurate. Function calling can involve knowledge that is deep and domain-specific. Sometimes, it doesn't fully understand the function and select the wrong one by mistake. Sometimes, it can fall into a loop and require calling the same function again and again. Sometimes, it will fabricate required function arguments instead of asking the user for input. To improve the function calling accuracy, it is advised to first try prompt engineering: does a more detailed function description help? can we provide instructions and examples to the model in the system message? If not, finetuning on your own data could also improve performance. - **Protocol consistency**: Even with the proper function calling template, the protocol may break. The model may generate extra texts to tool calls, e. g., explanations. The generated tool call may be invalid JSON-formatted string but a representation of a Python dict The generated tool call may be valid JSON but not conforms to the provided JSON Schema. For those kinds of issues, while some of them could be addressed with prompt engineering, some are caused by the nature of LLMs and can be hard to resolve in a general manner by LLMs themselves. While we strive to improve Qwen2.5 in this regard, edge cases are unlikely to be eliminated completely. ## Function Calling Templates The template design for function calling often includes the following aspects: - How to describe the functions to the model, so that the model understands what they are and how to use them. - How to prompt the model, so that it knows that functions can be used and in what format to generate the function calls. - How to tell a function call generation from others in generated text, so that we can extract the calls from the generated texts and actually make the calls. - How to incorporate the function results to the text, so that the model can tell them from its own generation and make connection among the calls and the results. For experienced prompt engineers, it should be possible to make any LLM support function calling, using in-context learning techniques and with representative examples, though with varied accuracy and stability depending on how "zero-shot" the task at hand is. ### Starting from ReAct Prompting For example, ReAct Prompting can be used to implement function calling with an extra element of planning: - **Thought**: the overt reasoning path, analyzing the functions and the user query and saying it out "loud" - **Action**: the function to use and the arguments with which the function should be called - **Observation**: the results of the function In fact, Qwen2 is verse in the following variant of ReAct Prompting (similar to LangChain ReAct) to make the intermediate texts more structured: ``` Answer the following questions as best you can. You have access to the following tools: {function_name}: Call this tool to interact with the {function_name_human_readable} API. What is the {function_name_human_readable} API useful for? {function_desciption} Parameters: {function_parameter_descriptions} {argument_formatting_instructions} {function_name}: Call this tool to interact with the {function_name_human_readable} API. What is the {function_name_human_readable} API useful for? {function_desciption} Parameters: {function_parameter_descriptions} {argument_formatting_instructions} Use the following format: Question: the input question you must answer Thought: you should always think about what to do Action: the action to take, should be one of [{function_name},{function_name}] Action Input: the input to the action Observation: the result of the action... (this Thought/Action/Action Input/Observation can be repeated zero or more times) Thought: I now know the final answer Final Answer: the final answer to the original input question Begin! Question: {query} Thought: {some_text} Action: {function_name} Action Input: {function_arguments} Observation: {function_results} Final Answer: {response} ``` As you can see, there is no apparent user/assistant conversation structure in the template. The model will simply continue the texts. One should write the code to actively detect which step the model is at and in particular to add the observations in the process, until the Final Answer is generated. However, as most programming interfaces accept the message structure, there should be some kind of adapter between the two. [The ReAct Chat Agent](https://github. com /QwenLM/Qwen-Agent/blob/v0.0.10/qwen_agent/agents/react_chat. py ) in Qwen-Agent facilitates this kind of conversion. ### Qwen2 Function Calling Template As a step forward, the official Qwen2 function calling template is in the vein of the ReAct Prompting format but focuses more on - differentiating the keywords like `Question`, `Thought`, `Action`, etc., from generation, - simplifying the process, - supporting better multi-turn conversation, and - adding controls for specialized usage. An equivalent example would be ``` <|im_start|>system {system message} ## Tools You have access to the following tools: ### {function_name_human_readable} {function_name}: {function_description} Parameters: {function_parameter_descriptions} {argument_formatting_instructions} ### {function_name_human_readable} {function_name}: {function_description} Parameters: {function_parameter_descriptions} {argument_formatting_instructions} ## When you need to call a tool, please insert the following command in your reply, which can be called zero or multiple times according to your needs: ✿FUNCTION✿: The tool to use, should be one of [{function_name},{function_name}] ✿ARGS✿: The input of the tool ✿RESULT✿: Tool results ✿RETURN✿: Reply based on tool results. Images need to be rendered as <|im_end|> <|im_start|>user {query}<|im_end|> <|im_start|>assistant ✿FUNCTION✿: {function_name} ✿ARGS✿: {function_arguments} ✿RESULT✿: {function_result} ✿RETURN✿:{response}<|im_end|> ``` Let's first list the obvious differences: - Keywords (`✿FUNCTION✿`, `✿ARGS✿`, etc.) seem rare in ordinary text and more semantically related to function calling, but not special tokens yet. - Thought is omitted. This could affect accuracy for some use cases. - Use the system-user-assistant format for multi-turn conversations. Function calling prompting is moved to the system message. How about adding controls for specialized usage? The template actually has the following variants: - Language: the above is for non-Chinese language; there is another template in Chinese. - Parallel Calls: the above is for non-parallel calls; there is another template for parallel calls. In the canonical implementation in Qwen-Agent, those switches are implemented in Python, according to the configuration and current input. The actual text with _parallel calls_ should be like the following: ```text <|im_start|>system You are Qwen, created by Alibaba Cloud. You are a helpful assistant. Current Date: 2024-09-30 ## Tools You have access to the following tools: ### get_current_temperature get_current_temperature: Get current temperature at a location. Parameters: {"type": "object", "properties": {"location": {"type": "string", "description": "The location to get the temperature for, in the format \"City, State, Country\"."}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "The unit to return the temperature in. Defaults to \"celsius\"."}}, "required": ["location"]} Format the arguments as a JSON object. ### get_temperature_date get_temperature_date: Get temperature at a location and date. Parameters: {"type": "object", "properties": {"location": {"type": "string", "description": "The location to get the temperature for, in the format \"City, State, Country\"."}, "date": {"type": "string", "description": "The date to get the temperature for, in the format \"Year-Month-Day\"."}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "The unit to return the temperature in. Defaults to \"celsius\"."}}, "required": ["location", "date"]} Format the arguments as a JSON object. ## Insert the following command in your reply when you need to call N tools in parallel: ✿FUNCTION✿: The name of tool 1, should be one of [get_current_temperature,get_temperature_date] ✿ARGS✿: The input of tool 1 ✿FUNCTION✿: The name of tool 2 ✿ARGS✿: The input of tool 2... ✿FUNCTION✿: The name of tool N ✿ARGS✿: The input of tool N ✿RESULT✿: The result of tool 1 ✿RESULT✿: The result of tool 2... ✿RESULT✿: The result of tool N ✿RETURN✿: Reply based on tool results. Images need to be rendered as <|im_end|> <|im_start|>user What's the temperature in San Francisco now? How about tomorrow?<|im_end|> <|im_start|>assistant ✿FUNCTION✿: get_current_temperature ✿ARGS✿: {"location": "San Francisco, CA, USA"} ✿FUNCTION✿: get_temperature_date ✿ARGS✿: {"location": "San Francisco, CA, USA", "date": "2024-10-01"} ✿RESULT✿: {"temperature": 26.1 , "location": "San Francisco, CA, USA", "unit": "celsius"} ✿RESULT✿: {"temperature": 25.9 , "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"} ✿RETURN✿: The current temperature in San Francisco is approximately 26.1 °C. For tomorrow, October 1st, 2024, the forecasted temperature will be around 25.9 °C.<|im_end|> ``` This template is hard to adapt it for other frameworks that use less capable templating engines. But it is doable at least partially for Jinja, which is Python-oriented after all. We didn't use it because using the template in `transformers` leads to more changes to the inference usage, which are not very common for beginners. For the interested, you can find the Jinja template and key points on usage below: :::{dropdown} Qwen2 Function Calling Jinja Template ```jinja {%- if messages[0]["role"] == "system" %} {%- set system_message = messages[0]["content"] %} {%- set loop_messages = messages[1:] %} {%- else %} {%- set system_message = "You are a helpful assistant." %} {%- set loop_messages = messages %} {%- endif %} {%- if parallel_tool_calls is undefined %} {%- set parallel_tool_calls = false %} {%- endif %} {%- if language is undefined or language != "zh" %} {%- set language = "en" %} {%- endif %} {{- "<|im_start|>system\n" + system_message|trim }} {%- if tools is defined %} {{- " # 工具 ## 你拥有如下工具： " if language == "zh" else " ## Tools You have access to the following tools: " }} {%- set functions = tools|map(attribute="function")|list %} {%- set function_names = functions|map(attribute="name")|join(",") %} {%- for function in functions %} {{- "### " + function. name + " " + function. name + ": " + function. description + (" 输入参数：" if language == "zh" else " Parameters: ") + function. parameters |tojson + (" 此工具的输入应为JSON对象。 " if language == "zh" else " Format the arguments as a JSON object. ") }} {%- endfor %} {%- if parallel_tool_calls and language == "zh" %} {{- "## 你可以在回复中插入以下命令以并行调用N个工具： ✿FUNCTION✿: 工具1的名称，必须是[" + function_names + "]之一\n✿ARGS✿: 工具1的输入\n✿FUNCTION✿: 工具2的名称\n✿ARGS✿: 工具2的输入\n...\n✿FUNCTION✿: 工具N的名称\n✿ARGS✿: 工具N的输入\n✿RESULT✿: 工具1的结果\n✿RESULT✿: 工具2的结果\n...\n✿RESULT✿: 工具N的结果\n✿RETURN✿: 根据工具结 果进行回复，需将图片用渲染出来" }} {%- elif parallel_tool_calls %} {{- "## Insert the following command in your reply when you need to call N tools in parallel: ✿FUNCTION✿: The name of tool 1, should be one of [" + function_names + "]\n✿ARGS✿: The input of tool 1\n✿FUNCTION✿: The name of tool 2\n✿ARGS✿: The input of tool 2\n...\n✿FUNCTION✿: The name of tool N\n✿ARGS✿: The input of tool N\n✿RESULT✿: The result of tool 1\n✿RESULT✿: The result of tool 2\n...\n✿RESULT✿: The result of tool N\n✿RETURN✿: Reply based on tool results. Images need to be rendered as " }} {%- elif language == "zh" %} {{- "## 你可以在回复中插入零次、一次或多次以下命令以调用工具： ✿FUNCTION✿: 工具名称，必须是[" + function_names + "]之一。\n✿ARGS✿: 工具输入\n✿RESULT✿: 工具结果\n✿RETURN✿: 根据工具结果进行回复，需将图片用渲染出来" }} {%- else %} {{- "## When you need to call a tool, please insert the following command in your reply, which can be called zero or multiple times according to your needs: ✿FUNCTION✿: The tool to use, should be one of [" + function_names + "]\n✿ARGS✿: The input of the tool\n✿RESULT✿: Tool results\n✿RETURN✿: Reply based on tool results. Images need to be rendered as " }} {%- endif %} {%- endif %} {{- "<|im_end|>" }} {%- for message in loop_messages %} {%- if message. role == "user" %} {{- "\n<|im_start|>" + message. role + "\n" + message. content + "<|im_end|>" }} {%- if loop. last and add_generation_prompt %} {{- "\n<|im_start|>assistant\n" }} {%- endif %} {%- elif message. role == "tool" %} {{- "✿RESULT✿: " + message. content + "\n" }} {%- if loop. last and add_generation_prompt %} {{- "✿RETURN✿:" }} {%- endif %} {%- elif message. role == "assistant" and message. tool_calls is defined %} {%- if loop. previtem. role == "user" %} {{- "\n<|im_start|>assistant\n" }} {%- endif %} {%- for function in message. tool_calls |map(attribute="function") %} {{- "✿FUNCTION✿: " + function. name + "\n✿ARGS✿: " + function. arguments |tojson + "\n" }} {%- endfor %} {%- elif message. role == "assistant" %} {%- if loop. previtem. role == "user" %} {{- "\n<|im_start|>assistant\n" }} {%- elif loop. previtem. role == "tool" %} {{- "✿RETURN✿:" }} {%- endif %} {{- message. content }} {%- if loop. nextitem is undefined or loop. nextitem. role == "user" %} {{- "<|im_end|>" }} {%- endif %} {%- else %} {{- "\n<|im_start|>" + message. role + "\n" + message. content + "<|im_end|>" }} {%- endif %} {%- endfor %} ``` To use this template in `transformers`: - Switches can be enabled by passing them to the `apply_chat_template` method, e. g., `tokenizer. apply_chat_template (messages, tools=tools, add_generation_prompt=True, parallel_tool_call=True, language="zh", tokenize=False)`. By default, it is for English non-parallel function calling. - The tool arguments should be a Python `dict` instead of a JSON-formatted object `str`. - Since the generation needs to be stopped at `✿RESULT✿` or else the model will generate fabricated tool results, we should add it to `stop_strings` in `generation_config`: ```python model. generation_config. stop_strings = ["✿RESULT✿:", "✿RETURN✿:"] ``` - As a result of using `stop_strings`, you need to pass the tokenizer to `model. generate ` as `model. generate (**inputs, tokenizer=tokenizer, max_new_tokens=512)`. - `response`, i. e., the model generation based on the tool calls and tool results, may contain a leading space. You should not strip it for the model. It is resulted from the tokenization and the template design. - The `try_parse_tool_calls` function should also be modified accordingly. ::: ### Qwen2.5 Function Calling Templates For `transformers` and Ollama, we have also used templates that are easier to implement with Jinja or Go. They are variants of [the Nous Research's Hermes function calling template](https://github. com /NousResearch/Hermes-Function-Calling#prompt-format-for-function-calling). The Jinja template and the Go template should produce basically the same results. They final text should look like the following: ```text <|im_start|>system You are Qwen, created by Alibaba Cloud. You are a helpful assistant. Current Date: 2024-09-30 # Tools You may call one or more functions to assist with the user query. You are provided with function signatures within <tools></tools> XML tags: <tools> {"type": "function", "function": {"name": "get_current_temperature", "description": "Get current temperature at a location.", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The location to get the temperature for, in the format \"City, State, Country\"."}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "The unit to return the temperature in. Defaults to \"celsius\"."}}, "required": ["location"]}}} {"type": "function", "function": {"name": "get_temperature_date", "description": "Get temperature at a location and date.", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The location to get the temperature for, in the format \"City, State, Country\"."}, "date": {"type": "string", "description": "The date to get the temperature for, in the format \"Year-Month-Day\"."}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "The unit to return the temperature in. Defaults to \"celsius\"."}}, "required": ["location", "date"]}}} </tools> For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags: <tool_call> {"name": <function-name>, "arguments": <args-json-object>} </tool_call><|im_end|> <|im_start|>user What's the temperature in San Francisco now? How about tomorrow?<|im_end|> <|im_start|>assistant <tool_call> {"name": "get_current_temperature", "arguments": {"location": "San Francisco, CA, USA"}} </tool_call> <tool_call> {"name": "get_temperature_date", "arguments": {"location": "San Francisco, CA, USA", "date": "2024-10-01"}} </tool_call><|im_end|> <|im_start|>user <tool_response> {"temperature": 26.1 , "location": "San Francisco, CA, USA", "unit": "celsius"} </tool_response> <tool_response> {"temperature": 25.9 , "location": "San Francisco, CA, USA", "date": "2024-10-01", "unit": "celsius"} </tool_response><|im_end|> <|im_start|>assistant The current temperature in San Francisco is approximately 26.1 °C. Tomorrow, on October 1, 2024, the temperature is expected to be around 25.9 °C.<|im_end|> ``` While the text may seem different from the previous one, the basic prompting structure is still the same. There are just more structural tags and more JSON-formatted strings. --- There is one thing we haven't talked about: how should functions be described to the LLMs. In short, you could describe them as you would normally describe them in an API documentation, as long as you can effectively parse, validate, and execute the tool calls generated by the models. The format with JSON Schema appears a valid and common choice. ## Finally In whichever way you choose to use function calling with Qwen2.5 , keep in mind that the limitation and the perks of prompt engineering applies: - It is not guaranteed that the model generation will always follow the protocol even with proper prompting or templates. Especially, for the templates that are more complex and relies more on the model itself to think and stay on track than the ones that are simpler and relies on the template and the use of control or special tokens. The latter one, of course, requires some kind of training. In production code, be prepared that if it breaks, countermeasures or rectifications are in place. - If in certain scenarios, the generation is not up to expectation, you can refine the template to add more instructions or constraints. While the templates mentioned here are general enough, they may not be the best or the most specific or the most concise for your use cases. The ultimate solution is fine-tuning using your own data. Have fun prompting! <article> #qwen_agent. md Qwen-Agent ========== `Qwen-Agent <https://github. com /QwenLM/Qwen-Agent>`__ is a framework for developing LLM applications based on the instruction following, tool usage, planning, and memory capabilities of Qwen. It also comes with example applications such as Browser Assistant, Code Interpreter, and Custom Assistant. Installation ------------.. code:: bash git clone https://github. com /QwenLM/Qwen-Agent. git cd Qwen-Agent pip install -e./ Developing Your Own Agent ------------------------- Qwen-Agent provides atomic components such as LLMs and prompts, as well as high-level components such as Agents. The example below uses the Assistant component as an illustration, demonstrating how to add custom tools and quickly develop an agent that uses tools... code:: py import json import os import json5 import urllib. parse from qwen_agent. agents import Assistant from qwen_agent. tools. base import BaseTool, register_tool llm_cfg = { # Use the model service provided by DashScope: 'model': 'qwen-max', 'model_server': 'dashscope', # 'api_key': 'YOUR_DASHSCOPE_API_KEY', # It will use the `DASHSCOPE_API_KEY' environment variable if 'api_key' is not set here. # Use your own model service compatible with OpenAI API: # 'model': 'Qwen/Qwen2.5 -7B-Instruct', # 'model_server': 'http://localhost:8000/v1', # api_base # 'api_key': 'EMPTY', # (Optional) LLM hyperparameters for generation: 'generate_cfg': { 'top_p': 0.8 } } system = 'According to the user\'s request, you first draw a picture and then automatically run code to download the picture ' + \ 'and select an image operation from the given document to process the image' # Add a custom tool named my_image_gen： @register_tool('my_image_gen') class MyImageGen(BaseTool): description = 'AI painting (image generation) service, input text description, and return the image URL drawn based on text information.' parameters = [{ 'name': 'prompt', 'type': 'string', 'description': 'Detailed description of the desired image content, in English', 'required': True }] def call(self, params: str, **kwargs) -> str: prompt = json5.loads (params)['prompt'] prompt = urllib. parse. quote(prompt) return json. dumps ( {'image_url': f'https://image. pollinations. ai/prompt/{prompt}'}, ensure_ascii=False) tools = ['my_image_gen', 'code_interpreter'] # code_interpreter is a built-in tool in Qwen-Agent bot = Assistant(llm=llm_cfg, system_message=system, function_list=tools, files=[os. path. abspath('doc. pdf ')]) messages = [] while True: query = input('user question: ') messages. append ({'role': 'user', 'content': query}) response = [] for response in bot. run (messages=messages): print('bot response:', response) messages. extend (response) The framework also provides more atomic components for developers to combine. For additional showcases, please refer to `examples <https://github. com /QwenLM/Qwen-Agent/tree/main/examples>`__. <article> #vllm. md # vLLM We recommend you trying [vLLM](https://github. com /vllm-project/vllm) for your deployment of Qwen. It is simple to use, and it is fast with state-of-the-art serving throughput, efficient management of attention key value memory with PagedAttention, continuous batching of input requests, optimized CUDA kernels, etc. To learn more about vLLM, please refer to the [paper](https://arxiv. org /abs/2309.06180 ) and [documentation](https://vllm. readthedocs. io/). ## Installation By default, you can install `vllm` by pip in a clean environment: ```bash pip install vllm ``` Please note that the prebuilt `vllm` has strict dependencies on `torch` and its CUDA versions. Check the note in the official document for installation ([link](https://docs. vllm. ai/en/latest/getting_started/installation. html )) for some help. We also advise you to install ray by `pip install ray` for distributed serving. ## Offline Batched Inference Models supported by Qwen2.5 codes are supported by vLLM. The simplest usage of vLLM is offline batched inference as demonstrated below. ```python from transformers import AutoTokenizer from vllm import LLM, SamplingParams # Initialize the tokenizer tokenizer = AutoTokenizer. from_pretrained ("Qwen/Qwen2.5 -7B-Instruct") # Pass the default decoding hyperparameters of Qwen2.5 -7B-Instruct # max_tokens is for the maximum length for generation. sampling_params = SamplingParams(temperature=0.7 , top_p=0.8 , repetition_penalty=1.05 , max_tokens=512) # Input the model name or path. Can be GPTQ or AWQ models. llm = LLM(model="Qwen/Qwen2.5 -7B-Instruct") # Prepare your prompts prompt = "Tell me something about large language models." messages = [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt} ] text = tokenizer. apply_chat_template ( messages, tokenize=False, add_generation_prompt=True ) # generate outputs outputs = llm. generate ([text], sampling_params) # Print the outputs. for output in outputs: prompt = output. prompt generated_text = output. outputs [0].text print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") ``` ## OpenAI-Compatible API Service It is easy to build an OpenAI-compatible API service with vLLM, which can be deployed as a server that implements OpenAI API protocol. By default, it starts the server at `http://localhost:8000`. You can specify the address with `--host` and `--port` arguments. Run the command as shown below: ```bash vllm serve Qwen/Qwen2.5 -7B-Instruct ``` You don't need to worry about chat template as it by default uses the chat template provided by the tokenizer. Then, you can use the [create chat interface](https://platform. openai. com/docs/api-reference/chat/completions/create) to communicate with Qwen: ::::{tab-set} :::{tab-item} curl ```bash curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen2.5 -7B-Instruct", "messages": [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."} ], "temperature": 0.7 , "top_p": 0.8 , "repetition_penalty": 1.05 , "max_tokens": 512 }' ``` ::: :::{tab-item} Python You can use the API client with the `openai` Python package as shown below: ```python from openai import OpenAI # Set OpenAI's API key and API base to use vLLM's API server. openai_api_key = "EMPTY" openai_api_base = "http://localhost:8000/v1" client = OpenAI( api_key=openai_api_key, base_url=openai_api_base, ) chat_response = client. chat. completions. create ( model="Qwen/Qwen2.5 -7B-Instruct", messages=[ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": "Tell me something about large language models."}, ], temperature=0.7 , top_p=0.8 , max_tokens=512, extra_body={ "repetition_penalty": 1.05 , }, ) print("Chat response:", chat_response) ``` :::: :::{tip} The OpenAI-compatible server in `vllm` comes with [a default set of sampling parameters](https://github. com /vllm-project/vllm/blob/v0.5.2/vllm/entrypoints/openai/protocol. py #L130), which are not suitable for Qwen2.5 models and prone to repetition. We advise you to always pass sampling parameters to the API. ::: ### Tool Use The OpenAI-compatible API could be configured to support tool call of Qwen2.5. For information, please refer to [our guide on Function Calling](../framework/function_call. md #vllm). ### Structured/JSON Output Qwen 2.5 , when used with vLLM, supports structured/JSON output. Please refer to [vllm's documentation](https://docs. vllm. ai/en/stable/serving/openai_compatible_server. html #extra-parameters-for-chat-api) for the `guided_json` parameters. Besides, it is also recommended to instruct the model to generate the specific format in the system message or in your prompt. ## Multi-GPU Distributed Serving To scale up your serving throughput, distributed serving helps you by leveraging more GPU devices. Besides, for large models like `Qwen2.5 -72B-Instruct`, it is impossible to serve it on a single GPU. Here, we demonstrate how to run `Qwen2.5 -72B-Instruct` with tensor parallelism just by passing in the argument `tensor_parallel_size`: ::::{tab-set} :::{tab-item} Offline ```python from vllm import LLM, SamplingParams llm = LLM(model="Qwen/Qwen2.5 -72B-Instruct", tensor_parallel_size=4) ``` ::: :::{tab-item} API ```bash vllm serve Qwen/Qwen2.5 -72B-Instruct --tensor-parallel-size 4 ``` ::: :::: ## Extended Context Support By default, the context length for Qwen2.5 models are set to 32,768 tokens. To handle extensive inputs exceeding 32,768 tokens, we utilize [YaRN](https://arxiv. org /abs/2309.00071 ), a technique for enhancing model length extrapolation, ensuring optimal performance on lengthy texts. vLLM supports YARN and it can be enabled by add a `rope_scaling` field to the `config. json ` file of the model. For example, ```json { ..., "rope_scaling": { "factor": 4.0 , "original_max_position_embeddings": 32768, "type": "yarn" } } ``` However, vLLM only supports _static_ YARN at present, which means the scaling factor remains constant regardless of input length, potentially impacting performance on shorter texts. We advise adding the `rope_scaling` configuration only when processing long contexts is required. ## Serving Quantized Models vLLM supports different types of quantized models, including AWQ, GPTQ, SqueezeLLM, etc. Here we show how to deploy AWQ and GPTQ models. The usage is almost the same as above except for an additional argument for quantization. For example, to run an AWQ model. e.g., `Qwen2.5 -7B-Instruct-AWQ`: ::::{tab-set} :::{tab-item} Offline ```python from vllm import LLM, SamplingParams llm = LLM(model="Qwen/Qwen2.5 -7B-Instruct-AWQ", quantization="awq") ``` ::: :::{tab-item} API ```bash vllm serve Qwen/Qwen2.5 -7B-Instruct-AWQ --quantization awq ``` ::: :::: or GPTQ models like `Qwen2.5 -7B-Instruct-GPTQ-Int4`: ::::{tab-set} :::{tab-item} Offline ```python from vllm import LLM, SamplingParams llm = LLM(model="Qwen/Qwen2.5 -7B-Instruct-GPTQ-Int4", quantization="gptq") ``` ::: :::{tab-item} API ```bash vllm serve Qwen/Qwen2.5 -7B-Instruct-GPTQ-Int4 --quantization gptq ``` ::: :::: Additionally, vLLM supports the combination of AWQ or GPTQ models with KV cache quantization, namely FP8 E5M2 KV Cache. For example, ::::{tab-set} :::{tab-item} Offline ```python from vllm import LLM, SamplingParams llm = LLM(model="Qwen/Qwen2.5 -7B-Instruct-GPTQ-Int4", quantization="gptq", kv_cache_dtype="fp8_e5m2") ``` ::: :::{tab-item} API ```bash vllm serve Qwen/Qwen2.5 -7B-Instruct-GPTQ-Int4 --quantization gptq --kv-cache-dtype fp8_e5m2 ``` ::: :::: ## Troubleshooting You may encounter OOM issues that are pretty annoying. We recommend two arguments for you to make some fix. - The first one is `--max-model-len`. Our provided default `max_position_embedding` is `32768` and thus the maximum length for the serving is also this value, leading to higher requirements of memory. Reducing it to a proper length for yourself often helps with the OOM issue. - Another argument you can pay attention to is `--gpu-memory-utilization`. vLLM will pre-allocate this much GPU memory. By default, it is `0.9 `. This is also why you find a vLLM service always takes so much memory. If you are in eager mode (by default it is not), you can level it up to tackle the OOM problem. Otherwise, CUDA Graphs are used, which will use GPU memory not controlled by vLLM, and you should try lowering it. If it doesn't work, you should try `--enforce-eager`, which may slow down infernece, or reduce the `--max-model-len`. <article> #langchain. rst Langchain ========================== This guide helps you build a question-answering application based on a local knowledge base using ``Qwen2.5 -7B-Instruct`` with ``langchain``. The goal is to establish a knowledge base Q&A solution. Basic Usage ----------- The implementation process of this project includes loading files -> reading text -> segmenting text -> vectorizing text -> vectorizing questions -> matching the top k most similar text vectors with the question vectors -> incorporating the matched text as context along with the question into the prompt -> submitting to the Qwen2.5 -7B-Instruct to generate an answer. Below is an example:.. code:: bash pip install langchain==0.0.174 pip install faiss-gpu.. code:: python from transformers import AutoModelForCausalLM, AutoTokenizer from abc import ABC from langchain. llms. base import LLM from typing import Any, List, Mapping, Optional from langchain. callbacks. manager import CallbackManagerForLLMRun model_name = "Qwen/Qwen2.5 -7B-Instruct" model = AutoModelForCausalLM. from_pretrained ( model_name, torch_dtype="auto", device_map="auto" ) tokenizer = AutoTokenizer. from_pretrained (model_name) class Qwen(LLM, ABC): max_token: int = 10000 temperature: float = 0.01 top_p = 0.9 history_len: int = 3 def __init__(self): super().__init__() @property def _llm_type(self) -> str: return "Qwen" @property def _history_len(self) -> int: return self. history_len def set_history_len(self, history_len: int = 10) -> None: self. history_len = history_len def _call( self, prompt: str, stop: Optional[List[str]] = None, run_manager: Optional[CallbackManagerForLLMRun] = None, ) -> str: messages = [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}, {"role": "user", "content": prompt} ] text = tokenizer. apply_chat_template ( messages, tokenize=False, add_generation_prompt=True ) model_inputs = tokenizer([text], return_tensors="pt").to(model. device ) generated_ids = model. generate ( **model_inputs, max_new_tokens=512 ) generated_ids = [ output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs. input_ids , generated_ids) ] response = tokenizer. batch_decode (generated_ids, skip_special_tokens=True)[0] return response @property def _identifying_params(self) -> Mapping[str, Any]: """Get the identifying parameters.""" return {"max_token": self. max_token , "temperature": self. temperature , "top_p": self. top_p , "history_len": self. history_len } After loading the Qwen2.5 -7B-Instruct model, you should specify the txt file for retrieval... code:: python import os import re import torch import argparse from langchain. vectorstores import FAISS from langchain. embeddings. huggingface import HuggingFaceEmbeddings from typing import List, Tuple import numpy as np from langchain. document_loaders import TextLoader from langchain. text_splitter import CharacterTextSplitter from langchain. docstore. document import Document from langchain. prompts. prompt import PromptTemplate from langchain. chains import RetrievalQA class ChineseTextSplitter(CharacterTextSplitter): def __init__(self, pdf: bool = False, **kwargs): super().__init__(**kwargs) self. pdf = pdf def split_text(self, text: str) -> List[str]: if self. pdf : text = re. sub (r"\n{3,}", "\n", text) text = re. sub ('\s', ' ', text) text = text. replace (" ", "") sent_sep_pattern = re. compile ( '([﹒﹔﹖﹗．。！？]["’”」』]{0,2}|(?=["‘“「『]{1,2}|$))') sent_list = [] for ele in sent_sep_pattern. split (text): if sent_sep_pattern. match (ele) and sent_list: sent_list[-1] += ele elif ele: sent_list. append (ele) return sent_list def load_file(filepath): loader = TextLoader(filepath, autodetect_encoding=True) textsplitter = ChineseTextSplitter(pdf=False) docs = loader. load_and_split (textsplitter) write_check_file(filepath, docs) return docs def write_check_file(filepath, docs): folder_path = os. path. join(os. path. dirname(filepath), "tmp_files") if not os. path. exists(folder_path): os. makedirs (folder_path) fp = os. path. join(folder_path, 'load_file. txt ') with open(fp, 'a+', encoding='utf-8') as fout: fout. write ("filepath=%s,len=%s" % (filepath, len(docs))) fout. write ('\n') for i in docs: fout. write (str(i)) fout. write ('\n') fout. close () def separate_list(ls: List[int]) -> List[List[int]]: lists = [] ls1 = [ls[0]] for i in range(1, len(ls)): if ls[i - 1] + 1 == ls[i]: ls1.append (ls[i]) else: lists. append (ls1) ls1 = [ls[i]] lists. append (ls1) return lists class FAISSWrapper(FAISS): chunk_size = 250 chunk_conent = True score_threshold = 0 def similarity_search_with_score_by_vector( self, embedding: List[float], k: int = 4 ) -> List[Tuple[Document, float]]: scores, indices = self. index. search(np. array ([embedding], dtype=np. float32 ), k) docs = [] id_set = set() store_len = len(self. index_to_docstore_id ) for j, i in enumerate(indices[0]): if i == -1 or 0 < self. score_threshold < scores[0][j]: # This happens when not enough docs are returned. continue _id = self. index_to_docstore_id [i] doc = self. docstore. search(_id) if not self. chunk_conent : if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {_id}, got {doc}") doc. metadata ["score"] = int(scores[0][j]) docs. append (doc) continue id_set. add (i) docs_len = len(doc. page_content ) for k in range(1, max(i, store_len - i)): break_flag = False for l in [i + k, i - k]: if 0 <= l < len(self. index_to_docstore_id ): _id0 = self. index_to_docstore_id [l] doc0 = self. docstore. search(_id0) if docs_len + len(doc0.page_content ) > self. chunk_size : break_flag = True break elif doc0.metadata ["source"] == doc. metadata ["source"]: docs_len += len(doc0.page_content ) id_set. add (l) if break_flag: break if not self. chunk_conent : return docs if len(id_set) == 0 and self. score_threshold > 0: return [] id_list = sorted(list(id_set)) id_lists = separate_list(id_list) for id_seq in id_lists: for id in id_seq: if id == id_seq[0]: _id = self. index_to_docstore_id [id] doc = self. docstore. search(_id) else: _id0 = self. index_to_docstore_id [id] doc0 = self. docstore. search(_id0) doc. page_content += " " + doc0.page_content if not isinstance(doc, Document): raise ValueError(f"Could not find document for id {_id}, got {doc}") doc_score = min([scores[0][id] for id in [indices[0].tolist().index(i) for i in id_seq if i in indices[0]]]) doc. metadata ["score"] = int(doc_score) docs. append ((doc, doc_score)) return docs if __name__ == '__main__': # load docs (pdf file or txt file) filepath = 'your file path' # Embedding model name EMBEDDING_MODEL = 'text2vec' PROMPT_TEMPLATE = """Known information: {context_str} Based on the above known information, respond to the user's question concisely and professionally. If an answer cannot be derived from it, say 'The question cannot be answered with the given information' or 'Not enough relevant information has been provided,' and do not include fabricated details in the answer. Please respond in English. The question is {question}""" # Embedding running device EMBEDDING_DEVICE = "cuda" # return top-k text chunk from vector store VECTOR_SEARCH_TOP_K = 3 CHAIN_TYPE = 'stuff' embedding_model_dict = { "text2vec": "your text2vec model path", } llm = Qwen() embeddings = HuggingFaceEmbeddings(model_name=embedding_model_dict[EMBEDDING_MODEL],model_kwargs={'device': EMBEDDING_DEVICE}) docs = load_file(filepath) docsearch = FAISSWrapper. from_documents (docs, embeddings) prompt = PromptTemplate( template=PROMPT_TEMPLATE, input_variables=["context_str", "question"] ) chain_type_kwargs = {"prompt": prompt, "document_variable_name": "context_str"} qa = RetrievalQA. from_chain_type ( llm=llm, chain_type=CHAIN_TYPE, retriever=docsearch. as_retriever (search_kwargs={"k": VECTOR_SEARCH_TOP_K}), chain_type_kwargs=chain_type_kwargs) query = "Give me a short introduction to large language model." print(qa. run (query)) Next Step --------- Now you can chat with Qwen2.5 use your own document. Continue to read the documentation and try to figure out more advanced usages of model retrieval! <article> #llamaindex. rst LlamaIndex ========== To connect Qwen2.5 with external data, such as documents, web pages, etc., we offer a tutorial on `LlamaIndex <https://www. llamaindex. ai/>`__. This guide helps you quickly implement retrieval-augmented generation (RAG) using LlamaIndex with Qwen2.5. Preparation -------------------------------------- To implement RAG, we advise you to install the LlamaIndex-related packages first. The following is a simple code snippet showing how to do this:.. code:: bash pip install llama-index pip install llama-index-llms-huggingface pip install llama-index-readers-web Set Parameters -------------------------------------- Now we can set up LLM, embedding model, and the related configurations. Qwen2.5 -Instruct supports conversations in multiple languages, including English and Chinese. You can use the ``bge-base-en-v1.5 `` model to retrieve from English documents, and you can download the ``bge-base-zh-v1.5 `` model to retrieve from Chinese documents. You can also choose ``bge-large`` or ``bge-small`` as the embedding model or modify the context window size or text chunk size depending on your computing resources. Qwen2.5 model families support a maximum of 32K context window size (up to 128K for 7B, 14B, 32B, and 72B, requiring extra configuration).. code:: python import torch from llama_index. core import Settings from llama_index. core. node_parser import SentenceSplitter from llama_index. llms. huggingface import HuggingFaceLLM from llama_index. embeddings. huggingface import HuggingFaceEmbedding # Set prompt template for generation (optional) from llama_index. core import PromptTemplate def completion_to_prompt(completion): return f"<|im_start|>system\n<|im_end|>\n<|im_start|>user\n{completion}<|im_end|>\n<|im_start|>assistant\n" def messages_to_prompt(messages): prompt = "" for message in messages: if message. role == "system": prompt += f"<|im_start|>system\n{message. content }<|im_end|>\n" elif message. role == "user": prompt += f"<|im_start|>user\n{message. content }<|im_end|>\n" elif message. role == "assistant": prompt += f"<|im_start|>assistant\n{message. content }<|im_end|>\n" if not prompt. startswith ("<|im_start|>system"): prompt = "<|im_start|>system\n" + prompt prompt = prompt + "<|im_start|>assistant\n" return prompt # Set Qwen2.5 as the language model and set generation config Settings. llm = HuggingFaceLLM( model_name="Qwen/Qwen2.5 -7B-Instruct", tokenizer_name="Qwen/Qwen2.5 -7B-Instruct", context_window=30000, max_new_tokens=2000, generate_kwargs={"temperature": 0.7 , "top_k": 50, "top_p": 0.95 }, messages_to_prompt=messages_to_prompt, completion_to_prompt=completion_to_prompt, device_map="auto", ) # Set embedding model Settings. embed_model = HuggingFaceEmbedding( model_name = "BAAI/bge-base-en-v1.5 " ) # Set the size of the text chunk for retrieval Settings. transformations = [SentenceSplitter(chunk_size=1024)] Build Index -------------------------------------- Now we can build index from documents or websites. The following code snippet demonstrates how to build an index for files (regardless of whether they are in PDF or TXT format) in a local folder named 'document'... code:: python from llama_index. core import VectorStoreIndex, SimpleDirectoryReader documents = SimpleDirectoryReader("./document").load_data() index = VectorStoreIndex. from_documents ( documents, embed_model=Settings. embed_model , transformations=Settings. transformations ) The following code snippet demonstrates how to build an index for the content in a list of websites... code:: python from llama_index. readers. web import SimpleWebPageReader from llama_index. core import VectorStoreIndex, SimpleDirectoryReader documents = SimpleWebPageReader(html_to_text=True).load_data( ["web_address_1","web_address_2",...] ) index = VectorStoreIndex. from_documents ( documents, embed_model=Settings. embed_model , transformations=Settings. transformations ) To save and load the index, you can use the following code snippet... code:: python from llama_index. core import StorageContext, load_index_from_storage # save index storage_context = StorageContext. from_defaults (persist_dir="save") # load index index = load_index_from_storage(storage_context) RAG ------------------- Now you can perform queries, and Qwen2.5 will answer based on the content of the indexed documents... code:: python query_engine = index. as_query_engine () your_query = "<your query here>" print(query_engine. query (your_query).response) <article> #qwen2.py # coding=utf-8 # Copyright 2024 The Qwen team, Alibaba Group and the HuggingFace Inc. team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www. apache. org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. """Qwen2 model configuration""" from...configuration_utils import PretrainedConfig from...modeling_rope_utils import rope_config_validation from...utils import logging logger = logging. get_logger (__name__) class Qwen2Config(PretrainedConfig): r""" This is the configuration class to store the configuration of a [`Qwen2Model`]. It is used to instantiate a Qwen2 model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of Qwen2-7B-beta [Qwen/Qwen2-7B-beta](https://huggingface. co /Qwen/Qwen2-7B-beta). Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 151936): Vocabulary size of the Qwen2 model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`Qwen2Model`] hidden_size (`int`, *optional*, defaults to 4096): Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 22016): Dimension of the MLP representations. num_hidden_layers (`int`, *optional*, defaults to 32): Number of hidden layers in the Transformer encoder. num_attention_heads (`int`, *optional*, defaults to 32): Number of attention heads for each attention layer in the Transformer encoder. num_key_value_heads (`int`, *optional*, defaults to 32): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details checkout [this paper](https://arxiv. org /pdf/2305.13245.pdf). If it is not specified, will default to `32`. hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the decoder. max_position_embeddings (`int`, *optional*, defaults to 32768): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02 ): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. rms_norm_eps (`float`, *optional*, defaults to 1e-06): The epsilon used by the rms normalization layers. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config. is_decoder =True`. tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether the model's input and output word embeddings should be tied. rope_theta (`float`, *optional*, defaults to 10000.0 ): The base period of the RoPE embeddings. rope_scaling (`Dict`, *optional*): Dictionary containing the scaling configuration for the RoPE embeddings. NOTE: if you apply new rope type and you expect the model to work on longer `max_position_embeddings`, we recommend you to update this value accordingly. Expected contents: `rope_type` (`str`): The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope', 'llama3'], with 'default' being the original RoPE implementation. `factor` (`float`, *optional*): Used with all rope types except 'default'. The scaling factor to apply to the RoPE embeddings. In most scaling types, a `factor` of x will enable the model to handle sequences of length x * original maximum pre-trained length. `original_max_position_embeddings` (`int`, *optional*): Used with 'dynamic', 'longrope' and 'llama3'. The original max position embeddings used during pretraining. `attention_factor` (`float`, *optional*): Used with 'yarn' and 'longrope'. The scaling factor to be applied on the attention computation. If unspecified, it defaults to value recommended by the implementation, using the `factor` field to infer the suggested value. `beta_fast` (`float`, *optional*): Only used with 'yarn'. Parameter to set the boundary for extrapolation (only) in the linear ramp function. If unspecified, it defaults to 32. `beta_slow` (`float`, *optional*): Only used with 'yarn'. Parameter to set the boundary for interpolation (only) in the linear ramp function. If unspecified, it defaults to 1. `short_factor` (`List[float]`, *optional*): Only used with 'longrope'. The scaling factor to be applied to short contexts (< `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden size divided by the number of attention heads divided by 2 `long_factor` (`List[float]`, *optional*): Only used with 'longrope'. The scaling factor to be applied to long contexts (< `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden size divided by the number of attention heads divided by 2 `low_freq_factor` (`float`, *optional*): Only used with 'llama3'. Scaling factor applied to low frequency components of the RoPE `high_freq_factor` (`float`, *optional*): Only used with 'llama3'. Scaling factor applied to high frequency components of the RoPE use_sliding_window (`bool`, *optional*, defaults to `False`): Whether to use sliding window attention. sliding_window (`int`, *optional*, defaults to 4096): Sliding window attention (SWA) window size. If not specified, will default to `4096`. max_window_layers (`int`, *optional*, defaults to 28): The number of layers that use SWA (Sliding Window Attention). The bottom layers use SWA while the top use full attention. attention_dropout (`float`, *optional*, defaults to 0.0 ): The dropout ratio for the attention probabilities. ```python >>> from transformers import Qwen2Model, Qwen2Config >>> # Initializing a Qwen2 style configuration >>> configuration = Qwen2Config() >>> # Initializing a model from the Qwen2-7B style configuration >>> model = Qwen2Model(configuration) >>> # Accessing the model configuration >>> configuration = model. config ```""" model_type = "qwen2" keys_to_ignore_at_inference = ["past_key_values"] # Default tensor parallel plan for base model `Qwen2` base_model_tp_plan = { "layers.*.self_attn. q_proj ": "colwise", "layers.*.self_attn. k_proj ": "colwise", "layers.*.self_attn. v_proj ": "colwise", "layers.*.self_attn. o_proj ": "rowwise", "layers.*.mlp. gate_proj ": "colwise", "layers.*.mlp. up_proj ": "colwise", "layers.*.mlp. down_proj ": "rowwise", } def __init__( self, vocab_size=151936, hidden_size=4096, intermediate_size=22016, num_hidden_layers=32, num_attention_heads=32, num_key_value_heads=32, hidden_act="silu", max_position_embeddings=32768, initializer_range=0.02 , rms_norm_eps=1e-6, use_cache=True, tie_word_embeddings=False, rope_theta=10000.0 , rope_scaling=None, use_sliding_window=False, sliding_window=4096, max_window_layers=28, attention_dropout=0.0 , **kwargs, ): self. vocab_size = vocab_size self. max_position_embeddings = max_position_embeddings self. hidden_size = hidden_size self. intermediate_size = intermediate_size self. num_hidden_layers = num_hidden_layers self. num_attention_heads = num_attention_heads self. use_sliding_window = use_sliding_window self. sliding_window = sliding_window if use_sliding_window else None self. max_window_layers = max_window_layers # for backward compatibility if num_key_value_heads is None: num_key_value_heads = num_attention_heads self. num_key_value_heads = num_key_value_heads self. hidden_act = hidden_act self. initializer_range = initializer_range self. rms_norm_eps = rms_norm_eps self. use_cache = use_cache self. rope_theta = rope_theta self. rope_scaling = rope_scaling self. attention_dropout = attention_dropout # Validate the correctness of rotary position embeddings parameters # BC: if there is a 'type' field, move it to 'rope_type'. if self. rope_scaling is not None and "type" in self. rope_scaling : self. rope_scaling ["rope_type"] = self. rope_scaling ["type"] rope_config_validation(self) super().__init__( tie_word_embeddings=tie_word_embeddings, **kwargs, )