SeaEval

A Comprehensive Multilingual and Cross-Lingual Evaluation of LLMs

-- English & 中文 & Bahasa Indonesia & Español & Tiếng Việt & Bahasa Melayu & Wikang Filipino --

Public Test

Cross-MMLU
Cross-MMLU - Accuracy per Language
Cross-LogiQA
Cross-LogiQA - Accuracy per Language

Hidden Test

Cross-MMLU
Cross-MMLU - Accuracy per Language
Cross-LogiQA
Cross-LogiQA - Accuracy per Language

Public Test

Release Date	Model	Model Size	Cross-MMLU
-	-	-	Accuracy	Consistency	AC3
11/2023	chatgpt (gpt-3.5-turbo-1106)	Unknown	68.5 🥇	56.0 🥈	61.6 🥇
06/2023	chatgpt (gpt-3.5-turbo-0613)	Unknown	67.1 🥈	51.7 🥉	58.4 🥈
10/2022	mt0-xxl*	13B	51.8	57.1 🥇	54.6 🥉
10/2023	chatglm3-6b	6B	36.0	16.8	23.4
09/2023	colossal-llama-2-7b-base	7B	36.7	14.1	20.6
09/2023	baichuan-2-7b	7B	41.4	25.1	31.4
09/2023	baichuan-2-7b-chat	7B	45.7	20.1	27.9
09/2023	baichuan-2-13b	13B	49.6	27.5	35.6
09/2023	baichuan-2-13b-chat	13B	53.6 🥉	25.5	34.3
07/2023	vicuna-7b-v1.5	7B	40.3	38.2	39.1
07/2023	vicuna-13b-v1.5	13B	47.8	37.8	42.3
07/2023	llama-2-7b	7B	32.5	51.2	38.1
07/2023	llama-2-7b-chat	7B	33.5	26.0	30.1
07/2023	llama-2-13b	13B	46.8	32.7	38.5
07/2023	llama-2-13b-chat	13B	38.2	32.2	34.9
07/2023	llama-2-70b	70B	53.2	26.5	35.4
07/2023	llama-2-70b-chat	70B	46.7	33.6	38.5
03/2023	alpaca-7b	7B	26.4	28.1	27.2
02/2023	llama-65b	65B	43.9	26.7	33.2
12/2022	flan-t5-xl*	3B	36.8	33.1	34.6
09/2022	bloomz-7b1*	7.1B	39.6	42.1	40.6

*: The model (might) has been exposed to supervisions.

Release Date	Model	Cross-MMLU (7 Languages)
-	-	English	Chinese	Indonesian	Spanish	Vietnamese	Malay	Filipino
11/2023	chatgpt (gpt-3.5-turbo-1106)	72.5	62.5	77.5	67.5	62.5	70.0	67.5
10/2023	chatglm3-6b	52.5	40.0	32.5	42.5	32.5	27.5	22.5
09/2023	colossal-llama-2-7b-base	55.0	37.5	30.0	32.5	37.5	30.0	25.0
09/2023	baichuan-2-7b	55.0	50.0	37.5	47.5	32.5	32.5	25.0
09/2023	baichuan-2-7b-chat	67.5	62.5	45.0	52.5	35.0	35.0	35.0
09/2023	baichuan-2-13b	75.0	67.5	47.0	55.0	37.5	35.0	37.5
09/2023	baichuan-2-13b-chat	77.5	70.0	50.0	67.5	47.5	30.0	35.0
07/2023	vicuna-7b-v1.5	55.0	32.5	42.5	52.5	42.5	32.5	25.0
07/2023	vicuna-13b-v1.5	55.0	50.0	47.5	50.0	52.5	40.0	37.5
07/2023	llama-2-7b	42.5	30.0	30.0	37.5	30.0	32.5	27.5
07/2023	llama-2-7b-chat	50.0	30.0	30.0	42.5	30.0	27.5	25.0
07/2023	llama-2-13b	52.5	37.5	45.0	50.0	50.0	40.0	42.5
07/2023	llama-2-13b-chat	50.0	35.0	35.0	45.0	45.0	32.5	32.5
07/2023	llama-2-70b	62.5	55.0	60.0	50.0	57.5	55.0	57.5
07/2023	llama-2-70b-chat	62.5	37.5	42.5	65.0	45.0	37.5	40.0
06/2023	chatgpt (gpt-3.5-turbo-0613)	75.0	67.5	60.0	50.0	75.0	72.5	70.0
03/2023	alpaca-7b	42.5	30.0	30.0	37.5	30.0	32.5	27.5
02/2023	llama-65b	60.0	40.0	37.5	67.5	32.5	35.0	47.5
12/2022	flan-t5-xl*	65.0	30.0	45.0	40.0	25.0	30.0	20.0
10/2022	mt0-xxl*	55.0	45.0	47.5	52.5	55.0	55.0	47.5
09/2022	bloomz-7b1*	45.0	45.0	42.5	50.0	37.5	40.0	17.5

*: The model (might) has been exposed to supervisions.

Release Date	Model	Cross-LogiQA
-	-	Accuracy	Consistency	AC3
11/2023	chatgpt (gpt-3.5-turbo-1106)	50.4	39.6	44.4
10/2023	chatglm3-6b	41.4	25.6	32.4
09/2023	colossal-llama-2-7b-base	36.4	14.4	20.7
09/2023	baichuan-2-7b	28.9	18.9	22.7
09/2023	baichuan-2-7b-chat	35.0	17.9	23.8
09/2023	baichuan-2-13b	41.4	18.2	25.1
09/2023	baichuan-2-13b-chat	42.1	23.2	29.4
07/2023	vicuna-7b-v1.5	40.4	37.2	38.7
07/2023	vicuna-13b-v1.5	45.4	32.7	38.0
07/2023	llama-2-7b	30.4	24.6	28.1
07/2023	llama-2-7b-chat	37.8	39.1	37.9
07/2023	llama-2-13b	34.3	30.5	33.4
07/2023	llama-2-13b-chat	36.4	36.4	36.2
07/2023	llama-2-70b	47.8	45.6	46.8
07/2023	llama-2-70b-chat	43.2	33.1	37.3
06/2023	chatgpt (gpt-3.5-turbo-0613)	44.3	29.7	35.6
03/2023	alpaca-7b	28.9	21.6	25.0
02/2023	llama-65b	42.8	29.4	35.6
12/2022	flan-t5-xl*	30.0	18.6	23.2
10/2022	mt0-xxl*	43.2	60.3	50.6
09/2022	bloomz-7b1*	39.3	35.0	37.0

*: The model (might) has been exposed to supervisions.

Release Date	Model	Cross-LogiQA (7 Languages)
-	-	English	Chinese	Indonesian	Spanish	Vietnamese	Malay	Filipino
11/2023	chatgpt (gpt-3.5-turbo-1106)	57.5	55.0	52.5	37.5	52.5	50.0	47.5
10/2023	chatglm3-6b	50.0	57.5	40.0	47.5	27.5	35.0	30.0
09/2023	colossal-llama-2-7b-base	52.5	40.0	35.0	37.5	30.0	37.5	25.0
09/2023	baichuan-2-7b	35.0	37.5	30.0	32.5	20.0	32.5	22.5
09/2023	baichuan-2-7b-chat	47.5	50.0	20.0	37.5	32.5	35.0	25.0
09/2023	baichuan-2-13b	40.0	35.0	37.5	35.0	40.0	32.5	27.5
09/2023	baichuan-2-13b-chat	47.5	40.0	30.0	42.5	27.5	35.0	30.0
07/2023	vicuna-7b-v1.5	52.5	37.5	40.0	43.75	32.5	40.0	35.0
07/2023	vicuna-13b-v1.5	55.0	45.0	45.0	55.0	45.0	42.5	27.5
07/2023	llama-2-7b	37.5	27.5	35.0	35.0	27.5	30.0	30.0
07/2023	llama-2-7b-chat	40.0	37.5	40.0	35.0	35.0	40.0	32.5
07/2023	llama-2-13b	40.0	35.0	37.5	35.0	40.0	32.5	27.5
07/2023	llama-2-13b-chat	47.5	40.0	30.0	42.5	27.5	35.0	30.0
07/2023	llama-2-70b	55.0	47.5	50.0	47.5	52.5	37.5	45.0
07/2023	llama-2-70b-chat	55.0	47.5	45.0	42.5	35.0	40.0	37.5
06/2023	chatgpt (gpt-3.5-turbo-0613)	50.0	42.5	42.5	45.0	50.0	45.0	35.0
03/2023	alpaca-7b	45.0	30.0	22.5	30.0	22.5	35.0	25.0
02/2023	llama-65b	52.5	45.0	37.5	50.0	37.5	32.5	42.5
12/2022	flan-t5-xl*	40.0	12.5	32.5	47.5	20.0	25.0	32.5
10/2022	mt0-xxl*	50.0	40.0	35.0	37.5	30.0	37.5	25.0
09/2022	bloomz-7b1*	45.0	45.0	40.0	42.5	37.5	30.0	30.0

*: The model (might) has been exposed to supervisions.

Hidden Test

Release Date	Model	Cross-MMLU
-	-	Accuracy	Consistency	AC3
11/2023	GPT-4	75.7	75.9	75.8
11/2023	chatgpt (gpt-3.5-turbo-1106)	62.5	51.7	56.6
10/2023	chatglm3-6b	42.5	27.1	33.1
09/2023	colossal-llama-2-7b-base	37.0	12.1	18.4
09/2023	baichuan-2-7b	43.6	23.2	30.2
09/2023	baichuan-2-7b-chat	44.0	26.0	33.0
09/2023	baichuan-2-13b	45.5	24.6	31.8
09/2023	baichuan-2-13b-chat	47.8	29.1	36.4
07/2023	vicuna-7b-v1.5	46.1	40.4	42.6
07/2023	vicuna-13b-v1.5	52.3	43.6	47.6
07/2023	llama-2-7b	26.5	46.2	32.9
07/2023	llama-2-7b-chat	38.7	27.5	32.6
07/2023	llama-2-13b	42.6	36.8	40.1
07/2023	llama-2-13b-chat	43.6	30.9	36.2
07/2023	llama-2-70b	51.9	32.4	39.5
07/2023	llama-2-70b-chat	50.6	36.8	42.6
03/2023	alpaca-7b	37.6	30.2	33.5
02/2023	llama-65b	38.8	25.4	30.7
12/2022	flan-t5-xl*	35.8	35.6	35.7
10/2022	mt0-xxl*	41.3	57.8	47.8
09/2022	bloomz-7b1*	41.3	47.9	44.3

*: The model (might) has been exposed to supervisions.

Release Date	Model	Cross-MMLU (7 Languages)
-	-	English	Chinese	Indonesian	Spanish	Vietnamese	Malay	Filipino
11/2023	GPT-4	81.8	78.2	72.7	80.0	71.8	70.0	74.5
11/2023	chatgpt (gpt-3.5-turbo-1106)	76.3	60.9	59.1	66.3	55.4	59.1	60.0
10/2023	chatglm3-6b	62.7	50.0	36.4	46.4	31.8	34.5	35.4
09/2023	colossal-llama-2-7b-base	51.8	40.9	34.5	36.4	33.6	32.7	31.8
09/2023	baichuan-2-7b	56.4	53.6	40.0	48.1	38.1	35.5	38.2
09/2023	baichuan-2-7b-chat	55.4	51.8	39.1	48.1	40.0	33.6	38.1
09/2023	baichuan-2-13b	55.5	57.2	41.8	53.6	34.5	39.9	40.9
09/2023	baichuan-2-13b-chat	63.6	61.8	35.5	57.3	37.3	35.5	42.7
07/2023	vicuna-7b-v1.5	53.6	44.5	49.1	50.0	45.5	41.8	37.3
07/2023	vicuna-13b-v1.5	70.0	53.6	50.9	54.5	44.5	45.5	48.2
07/2023	llama-2-7b	32.7	22.7	26.4	28.2	28.2	24.5	27.3
07/2023	llama-2-7b-chat	49.1	36.4	38.2	38.2	36.3	36.3	37.2
07/2023	llama-2-13b	50.9	41.8	45.5	46.4	36.4	36.4	37.3
07/2023	llama-2-13b-chat	50.0	43.6	41.8	47.3	40.9	36.4	40.0
07/2023	llama-2-70b	59.1	55.5	50.0	60.9	55.5	50.0	50.9
07/2023	llama-2-70b-chat	60.0	49.1	48.2	56.4	47.3	48.2	46.4
03/2023	alpaca-7b	51.8	36.4	31.8	45.5	30.0	33.6	33.6
02/2023	llama-65b	50.0	39.1	37.3	54.5	30.0	33.6	35.5
12/2022	flan-t5-xl*	48.2	25.5	36.4	46.4	25.5	37.3	33.6
10/2022	mt0-xxl*	42.7	45.5	40.0	33.6	41.8	45.5	40.9
09/2022	bloomz-7b1*	50.9	46.4	39.1	48.1	39.1	37.2	31.8

*: The model (might) has been exposed to supervisions.

Release Date	Model	Cross-LogiQA
-	-	Accuracy	Consistency	AC3
11/2023	GPT-4	69.1	66.6	67.8
11/2023	chatgpt (gpt-3.5-turbo-1106)	49.5	41.5	45.1
10/2023	chatglm3-6b	38.5	27.4	33.1
09/2023	colossal-llama-2-7b-base	36.1	17.2	18.5
09/2023	baichuan-2-7b	35.8	26.6	30.3
09/2023	baichuan-2-7b-chat	37.6	27.2	33.0
09/2023	baichuan-2-13b	38.7	21.4	31.8
09/2023	baichuan-2-13b-chat	43.2	28.6	36.3
07/2023	vicuna-7b-v1.5	41.7	43.5	42.6
07/2023	vicuna-13b-v1.5	41.1	37.1	47.6
07/2023	llama-2-7b	30.2	26.7	32.9
07/2023	llama-2-7b-chat	36.7	33.4	32.6
07/2023	llama-2-13b	36.5	33.5	40.1
07/2023	llama-2-13b-chat	38.4	39.8	36.2
07/2023	llama-2-70b	52.6	43.5	39.4
07/2023	llama-2-70b-chat	47.8	41.5	44.4
03/2023	alpaca-7b	30.0	24.2	33.5
02/2023	llama-65b	38.6	30.2	30.7
12/2022	flan-t5-xl*	37.1	24.6	29.4
10/2022	mt0-xxl*	51.2	57.1	47.8
09/2022	bloomz-7b1*	41.5	33.1	44.3

*: The model (might) has been exposed to supervisions.

Release Date	Model	Cross-LogiQA (7 Languages)
-	-	English	Chinese	Indonesian	Spanish	Vietnamese	Malay	Filipino
11/2023	GPT-4	75.7	72.1	69.8	70.6	65.4	64.7	65.4
11/2023	chatgpt (gpt-3.5-turbo-1106)	60.2	56.6	45.5	50.0	48.5	45.6	39.7
10/2023	chatglm3-6b	50.7	53.7	30.8	39.7	30.2	28.6	36.7
09/2023	colossal-llama-2-7b-base	46.3	40.4	35.3	38.9	31.6	33.1	33.8
09/2023	baichuan-2-7b	43.4	41.9	30.2	38.9	35.3	32.3	28.6
09/2023	baichuan-2-7b-chat	46.3	50.0	35.3	40.4	30.8	33.1	30.8
09/2023	baichuan-2-13b	41.2	52.2	33.1	38.2	36.0	34.5	31.6
09/2023	baichuan-2-13b-chat	59.6	55.8	35.3	47.8	38.2	37.5	31.6
07/2023	vicuna-7b-v1.5	50.7	42.6	41.9	43.4	43.4	38.2	31.6
07/2023	vicuna-13b-v1.5	51.5	46.3	41.2	43.4	38.2	33.8	33.8
07/2023	llama-2-7b	35.3	30.1	33.1	33.1	27.9	25.0	26.5
07/2023	llama-2-7b-chat	44.1	36.0	35.3	41.9	34.5	31.6	30.8
07/2023	llama-2-13b	38.2	33.8	37.5	37.5	43.4	35.3	34.6
07/2023	llama-2-13b-chat	47.1	38.2	38.2	37.5	39.7	36.0	33.8
07/2023	llama-2-70b	54.4	54.4	59.6	55.1	52.9	49.3	46.3
07/2023	llama-2-70b-chat	57.4	50.0	45.6	52.9	45.6	43.4	40.4
03/2023	alpaca-7b	33.1	28.7	30.2	33.1	25.7	26.5	30.2
02/2023	llama-65b	47.1	33.8	33.8	49.3	30.8	37.5	33.8
12/2022	flan-t5-xl*	55.8	27.9	25.0	31.6	27.2	22.8	26.5
10/2022	mt0-xxl*	53.7	52.2	50.7	50.0	48.5	53.7	50.0
09/2022	bloomz-7b1*	47.8	43.4	42.6	41.2	45.6	41.9	28.7

*: The model (might) has been exposed to supervisions.