Main Page

The main page of the Annotations Module provides access to 2 tabs - Datasets and AI Agents.

Datasets Tab

The Datasets Tab within the Annotations Module provides tools for managing your datasets for Human and AI annotations.

At the Datasats Tab you can:

• Create New Dataset: Easily create new datasets tailored to specific annotation requirements.
• View Existing Datasets: Access and manage previously created datasets. Clicking on a dataset name redirects the user to the dedicated dataset page for further management.

Once datasets are created you will see them on the list:

Schema

The Schema tab enables users to define and modify the structure of the dataset. Key functionalities include:

• Field Management: Create and edit fields within the dataset, such as prompt, response 1, response 2, etc.
• Question Creation: Define questions to collect feedback from annotators. Questions can have various types, including rating, ranking, labeling, multi-labeling, and text.
• Unification Strategy: Specify the unification strategy for each question to ensure consistency in annotator responses.

Add fields to your Schema:

Each field has to correspond with the field names of the dataset your're going to work with. For exapmple, if you want annotators to rate the answer provided by a model, your Fields will be Prompt and Answer.

Add Questions to your Schema:

Choose Unification Stratagy when creating a Question:

Label: Assigns a single category or class to each item.

Multi-label: Allows assigning multiple categories or classes to each item.

Rating: Assigns a numerical value to each item, indicating a score or level.

Ranking: Orders items relative to each other based on importance or relevance.

Text: Provides descriptive text annotations or comments for each item.

Records

The Records tab allows users to visualize dataset records and perform annotation tasks:

Features include:

• Populate from dataframe: Add data you want to annotate from dataframes uploaded in the Datasets Module. Make sure the Field parameters defined in Schema match the data.
• Record Visualization: View individual records within the dataset.
• Annotation Interface: Annotate records and view annotations made by other users. Users can access detailed information about each record, including annotations.

Annotators

The Annotators tab facilitates the assignment and management of annotators for the dataset.

Functions include:

• Annotator Assignment: View annotators assigned to the dataset and assign new annotators. Users can specify the number of annotators per record for multiple annotations.
• AI Agent Annotation: Initiate annotation tasks performed by AI agents for eautomatic annotations.

Formatting

The Formatting tab allows users to script formatting code to convert the dataset to a format required for specific NLP tasks.

Features include:

• Formatting Script: Script formatting code to convert the dataset to the desired format for tasks such as question answering, classification, named entity recognition, etc.
• Template Usage: Utilize platform-provided templates for different NLP tasks and customize them based on dataset requirements. Available templates are DPO (Direct Preference Optimization) and Question Answering.
• Formatted Data Storage: Once the formatting code is executed, the formatted data is stored as a dataframe, which users can download or use for other tasks.

Dataframes

The Dataframe tab offers functionalities related to previewing the dataframe created during the annotation.

Settings

Use Settings to manage or delete the datasets

AI Agents Tab

The AI Agents tab allows users to manage AI agents that can assist in automatic annotations.

Add New Agent

To add a new agent click Add New Agent and setup the parameters. You can choose our OneinnovationAI agent or Open AI models as providers:. Make sure you put the correct API key for the desired AI annotator:

View Existing AI Agents:

Access and manage previously created AI agents, including editing agent configurations or deleting them.