LM-SYS

Large Model Systems Organization (LMSYS Org) is an open research organization co-founded by students and teachers from the University of California, Berkeley, in collaboration with the University of California, San Diego, and Carnegie Mellon University. The team was established in March 2023. Its current job is to build a large model system and is the release team of the chat robot Vicuna.

The goal of LM-SYS is to make large models accessible to everyone through the joint development of open models, datasets, systems and evaluation tools. Their work covers machine learning and systems research. At the same time, the organization also trains large language models and makes them widely available, while also developing distributed systems to accelerate their training and inference processes.

The most famous results are the Vicuna series of open source large models and the large model arena ranking (Chatbot Arena) based on real human voting feedback.

With the development and growth of LLM-SYS, the organization has already achieved many achievements and projects. Mainly include:

Vicuna series large language model: The Vicuna series large language model is a large language model obtained by fine-tuning based on the LLaMA series basic model. By collecting the conversation history of large models shared by users, Vicuna has achieved very good performance improvements and achieved very good results in various evaluations. Reference: https://www.datalearner.com/blog/1051691043258851

Chatbot Arena: The famous large model arena, in order to better evaluate the effect of large language models. Also to avoid biased results obtained by large models on the evaluation data set. LM-SYS releases large model of Anonymous Arena. That is, users can get responses from one or more models by sending a question. Users can vote based on the response results to gain their preferences for different model answers to different questions. This is used to test the quality of the models. This is a crowdsourced scoring system that includes many popular and classic models and is very influential in large model reviews. The system also went to HuggingFace from the LM-SYS official website.

MT-Bench: MT-Bench is a well-known large model evaluation benchmark tool developed by LM-SYS. By using the strongest models such as GPT-4 to evaluate the automatic scoring of different models on different challenging problems, the results of large models on challenging problems such as multiple rounds are obtained. This scoring result also has very good visibility because it is very close to human preferences.

SGLang: A very efficient large model inference framework recently launched by LM-SYS, which can be used to accelerate the inference process of large models.

FastChat: A comprehensive large model framework and platform for training, deploying, and evaluating the effectiveness of large language model-based chatbots.

LMSYS-Chat-1M: LM-SYS open source large-scale data set, including more than 1 million large model dialogue data sets, from the historical records of conversations between users and 25 different large models in the real world. Reference: https://www.datalearner.com/blog/1051695352221980

LM-SYS official website: https://lmsys.org/

Chatbot Arena website: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard