From 244845c13df408b60277f1d536c87a234a129040 Mon Sep 17 00:00:00 2001 From: Daya Guo <40300434+guoday@users.noreply.github.com> Date: Sun, 29 Oct 2023 14:41:42 +0800 Subject: [PATCH] Update README.md --- README.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 5d6292f..83666f3 100644 --- a/README.md +++ b/README.md @@ -208,7 +208,7 @@ The reproducible code for the following evaluation results can be found in the E #### 1) [HumanEval](https://github.com/deepseek-ai/deepseek-coder/tree/main/Evaluation/HumanEval) -##### Multilingual Base Models +Multilingual Base Models | Model | Size | Python | C++ | Java | PHP | TS | C# | Bash | JS | Avg | | ------------------- | ---- | ------ | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | @@ -216,15 +216,15 @@ The reproducible code for the following evaluation results can be found in the E | CodeShell | 7B | 35.4% | 32.9% | 34.2% | 31.7% | 30.2% | 38.0% | 7.0% | 33.5% | 30.4% | | CodeGeeX2 | 6B | 36.0% | 29.2% | 25.9% | 23.6% | 20.8% | 29.7% | 6.3% | 24.8% | 24.5% | | StarCoderBase | 16B | 31.7% | 31.1% | 28.5% | 25.4% | 34.0% | 34.8% | 8.9% | 29.8% | 28.0% | -| CodeLLama (7B) | 7B | 31.7% | 29.8% | 34.2% | 23.6% | 36.5% | 36.7% | 12.0% | 29.2% | 29.2% | -| CodeLLama (13B) | 13B | 36.0% | 37.9% | 38.0% | 34.2% | 45.2% | 43.0% | 16.5% | 32.3% | 35.4% | -| CodeLLama (34B) | 34B | 48.2% | 44.7% | 44.9% | 41.0% | 42.1% | 48.7% | 15.8% | 42.2% | 41.0% | +| CodeLLama | 7B | 31.7% | 29.8% | 34.2% | 23.6% | 36.5% | 36.7% | 12.0% | 29.2% | 29.2% | +| CodeLLama | 13B | 36.0% | 37.9% | 38.0% | 34.2% | 45.2% | 43.0% | 16.5% | 32.3% | 35.4% | +| CodeLLama | 34B | 48.2% | 44.7% | 44.9% | 41.0% | 42.1% | 48.7% | 15.8% | 42.2% | 41.0% | | | | | | | | | | | | | -| OraCoder-Base (1B) | 1B | 34.8% | 31.1% | 32.3% | 24.2% | 28.9% | 36.7% | 10.1% | 28.6% | 28.3% | -| OraCoder-Base (7B) | 7B | 49.4% | 50.3% | 43.0% | 38.5% | 49.7% | 50.0% | 28.5% | 48.4% | 44.7% | -| OraCoder-Base (33B) | 33B | - | - | - | - | - | - | - | - | - | +| OraCoder-Base | 1B | 34.8% | 31.1% | 32.3% | 24.2% | 28.9% | 36.7% | 10.1% | 28.6% | 28.3% | +| OraCoder-Base | 7B | 49.4% | 50.3% | 43.0% | 38.5% | 49.7% | 50.0% | 28.5% | 48.4% | 44.7% | +| OraCoder-Base | 33B | - | - | - | - | - | - | - | - | - | -##### Instruction-Tuned Models +Instruction-Tuned Models | Model | Size | Python | C++ | Java | PHP | TS | C# | Bash | JS | Avg | | ------------------- | ---- | ------ | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | | ChatGPT | - | 70.7% | 50.3% | 54.5% | 52.2% | 62.3% | 64.6% | 34.8% | 60.9% | 52.2% | @@ -232,15 +232,15 @@ The reproducible code for the following evaluation results can be found in the E | WizardCoder | 16B | 51.8% | 41.6% | 41.1% | 42.2% | 44.7% | 46.8% | 12.7% | 42.8% | 40.5% | | Phind-CodeLlama | 34B | - | - | - | - | - | - | - | - | - | | | | | | | | | | | | | -| OraCoder-Chat (1B) | 1B | - | - | - | - | - | - | - | - | - | -| OraCoder-Chat (7B) | 7B | - | - | - | - | - | - | - | - | - | -| OraCoder-Chat (33B) | 33B | - | - | - | - | - | - | - | - | - | +| OraCoder-Chat | 1B | - | - | - | - | - | - | - | - | - | +| OraCoder-Chat | 7B | - | - | - | - | - | - | - | - | - | +| OraCoder-Chat | 33B | - | - | - | - | - | - | - | - | - | #### 2) [Math Reasoning](https://github.com/deepseek-ai/deepseek-coder/tree/main/Evaluation/PAL-Math) -##### Multilingual Base Models +Multilingual Base Models | Model | Size | GSM8k | MATH | GSM-Hard | SVAMP | TabMWP | ASDiv | MAWPS | Avg | | -------------- | ---- | ----- | ----- | -------- | ----- | ------ | ----- | ----- | ----- | @@ -256,7 +256,7 @@ The reproducible code for the following evaluation results can be found in the E | OraCoder-Base | 33B | - | - | - | - | - | - | - | - | -##### Instruction-Tuned Models +Instruction-Tuned Models | Model | Size | GSM8k | MATH | GSM-Hard | SVAMP | TabMWP | ASDiv | MAWPS | Avg | | ------------- | ---- | ----- | ----- | -------- | ----- | ------ | ----- | ----- | ----- | | ChatGPT | - | 78.6% | 38.7% | 67.6% | 77.8% | 79.9% | 81.0% | 89.4% | 73.3% |