Multi-LLM Evaluation & Synthesis Pipeline

Overview

Individual language models can vary in the quality and grounding of their responses. I designed and built a multi-LLM pipeline to explore how evaluation and synthesis can support more reliable generated content.

The work began in an educational-content context, but the approach is task-agnostic. Its goal is to assess candidate responses and consolidate their strongest elements while checking factual grounding.

My role

Lead engineer — design, implementation, and evaluation.

My contributions span pipeline design and prototype implementation, prompt engineering, rubric-based evaluation, and hallucination detection.

Public approach

The pipeline coordinates outputs from multiple LLMs. An aggregator evaluates candidate responses against predefined rubrics and synthesizes a consolidated output. Hallucination checks support quality assurance.

The current aggregator does not send feedback for additional generation rounds.

Evaluation focus

A central research question is whether synthesis can match or improve on the strongest individual model’s response. I approached this through evaluation design and analysis of generated outputs, with attention to factual grounding and consistency.

Skills applied

• Applied NLP and LLM orchestration
• Prompt engineering and rubric-based assessment
• Evaluation design and hallucination detection
• Pipeline design and prototype implementation

Project status

Partial implementation/prototype developed in a client context under NDA.