Reliability-Weighted Multi-Agent Annotation Workflow for Quality-Controlled LLM Labeling

Large Language Models (LLMs) are widely adopted for automatically labeling data because they work quickly and can handle large amounts of information. But there are problems like inconsistent results, made-up information, and different levels of reasoning ability across models, which can make the labeling less reliable. To fix these issues, this paper introduces a Multi-Agent Reliability-Weighted Annotation Workflow, a system designed to improve the trust in the labels created by LLMs. The system uses 3 to 5 different types of LLM agents, each with different structures and ways of generating responses. Each agent labels data on its own. The system then gives each agent a reliability score based on how well it has performed before, how confident it is in its answers, and how much the agents agree with each other. The final labels are created by combining the agents' results with more weight given to the more reliable ones, rather than just taking the most common answer. Cases where the model is unsure or there is a big disagreement are automatically marked for further checking by a person or for re-labeling. Tests on datasets like AG News, CoNLL-2003, and SST-20 show that this method improves accuracy by up to 3.4% and increases label agreement by 0.06 compared to simpler methods that don't consider reliability.