Members-Only
Recent Talks & Demos are for members only
You must be an AI Tinkerers active member to view these talks and demos.
LLM Fingerprinting: Model Classification
This talk demonstrates a system to identify and classify large language models by analyzing their responses to benchmark prompts, using live API classification and code walkthroughs.
The presentation walks through a LLM classification system to identify and classify Large Language Models (LLMs) based on their ability to respond to various prompts from diverse disciplines. This project involves evaluating performance on specific benchmarks (relating to math, logic, self-identification, etc) and scoring the LLMs at various temperatures to then use that data to build a classifier.
The implementation combines benchmarking and classification to classify LLMs from different families, such as GPT, LLaMA, Claude, and Gemini.
The demo will include:
- A live classification run to determine which LLM is accessed through an API key.
- A code walkthrough of the frontend, evaluation process and classification model.
Compares Deepinfra LLM first-token responses, analyzing distribution across models/temperatures.