Chatbots
Chatbots
Chatbots are conversational agents with world knowledge derived from pretraining. These are the easiest to evaluate because we only deal with the bot and its answer, but also they generally know the least and are prone to making up answers.
Datasets
To evaluate a chatbot, you'll need to build a dataset. You can create this in a spreadsheet (Google Sheets, Excel) or using Pandas.
Your dataset should include the following columns:
| Column | Type | Description |
|---|---|---|
input |
Required | The user message or question sent to the chatbot. |
ground_truth |
Required | The ideal or correct answer you expect from the chatbot. |
system_prompt |
Optional | The instructions given to the model (e.g., "You are a helpful assistant"). |
output |
Optional | The actual response from the model. If not provided, you can select a model in the platform to generate answers for you. |