deepseekmirror/DeepSeek-V3
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepSeek-V3.git synced 2025-02-23 06:08:58 -05:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
b4e06d883e
DeepSeek-V3/README_WEIGHTS.md

93 lines
3.6 KiB
Markdown
Raw Normal View History

Release DeepSeek-V3
2024-12-26 06:01:57 -05:00
# DeepSeek-V3 Weight File Documentation
## New Fields in `config.json`
Improve DeepSeek-V3 Weight File Documentation for Clarity and Readability - Enhanced sentence structure for better clarity and smoother flow. - Adjusted wording and phrasing to improve accuracy and professionalism. - Optimized the organization of information for better readability, especially in the sections related to parameters and technical details. - Refined formatting and sectioning of the documentation for easier navigation and comprehension.
2025-01-30 05:35:58 -05:00
- **model_type**: Specifies the model type, which is now set to `deepseek_v3` in this release.
- **num_nextn_predict_layers**: Defines the number of Multi-Token Prediction (MTP) Modules. The open-sourced V3 weights contain **1 MTP Module**.
- **quantization_config**: Details the configuration for FP8 quantization.
Release DeepSeek-V3
2024-12-26 06:01:57 -05:00
---
Improve DeepSeek-V3 Weight File Documentation for Clarity and Readability - Enhanced sentence structure for better clarity and smoother flow. - Adjusted wording and phrasing to improve accuracy and professionalism. - Optimized the organization of information for better readability, especially in the sections related to parameters and technical details. - Refined formatting and sectioning of the documentation for easier navigation and comprehension.
2025-01-30 05:35:58 -05:00
## Weight File Structure Overview
Release DeepSeek-V3
2024-12-26 06:01:57 -05:00
Improve DeepSeek-V3 Weight File Documentation for Clarity and Readability - Enhanced sentence structure for better clarity and smoother flow. - Adjusted wording and phrasing to improve accuracy and professionalism. - Optimized the organization of information for better readability, especially in the sections related to parameters and technical details. - Refined formatting and sectioning of the documentation for easier navigation and comprehension.
2025-01-30 05:35:58 -05:00
The DeepSeek-V3 weight file is divided into two primary components: **Main Model Weights** and **MTP Modules**.
Release DeepSeek-V3
2024-12-26 06:01:57 -05:00
### 1. Main Model Weights
- **Composition**:
Improve DeepSeek-V3 Weight File Documentation for Clarity and Readability - Enhanced sentence structure for better clarity and smoother flow. - Adjusted wording and phrasing to improve accuracy and professionalism. - Optimized the organization of information for better readability, especially in the sections related to parameters and technical details. - Refined formatting and sectioning of the documentation for easier navigation and comprehension.
2025-01-30 05:35:58 -05:00
- Includes input/output embedding layers and a full set of 61 Transformer hidden layers.
Release DeepSeek-V3
2024-12-26 06:01:57 -05:00
- **Parameter Count**:
- Total parameters: **671B**
Improve DeepSeek-V3 Weight File Documentation for Clarity and Readability - Enhanced sentence structure for better clarity and smoother flow. - Adjusted wording and phrasing to improve accuracy and professionalism. - Optimized the organization of information for better readability, especially in the sections related to parameters and technical details. - Refined formatting and sectioning of the documentation for easier navigation and comprehension.
2025-01-30 05:35:58 -05:00
- Activation parameters: **36.7B** (which includes 0.9B for Embedding and 0.9B for the Output Head).
Release DeepSeek-V3
2024-12-26 06:01:57 -05:00
#### Structural Details
- **Embedding Layer**:
- `model.embed_tokens.weight`
- **Transformer Hidden Layers**:
Improve DeepSeek-V3 Weight File Documentation for Clarity and Readability - Enhanced sentence structure for better clarity and smoother flow. - Adjusted wording and phrasing to improve accuracy and professionalism. - Optimized the organization of information for better readability, especially in the sections related to parameters and technical details. - Refined formatting and sectioning of the documentation for easier navigation and comprehension.
2025-01-30 05:35:58 -05:00
- From `model.layers.0` to `model.layers.60`, which correspond to `num_hidden_layers` layers.
Release DeepSeek-V3
2024-12-26 06:01:57 -05:00
- **Output Layer**:
- `model.norm.weight`
- `lm_head.weight`
### 2. Multi-Token Prediction (MTP) Modules
- **Composition**:
Improve DeepSeek-V3 Weight File Documentation for Clarity and Readability - Enhanced sentence structure for better clarity and smoother flow. - Adjusted wording and phrasing to improve accuracy and professionalism. - Optimized the organization of information for better readability, especially in the sections related to parameters and technical details. - Refined formatting and sectioning of the documentation for easier navigation and comprehension.
2025-01-30 05:35:58 -05:00
- These modules are determined by the `num_nextn_predict_layers` parameter. In this model, the value is set to 1.
Release DeepSeek-V3
2024-12-26 06:01:57 -05:00
- **Parameter Count**:
Improve DeepSeek-V3 Weight File Documentation for Clarity and Readability - Enhanced sentence structure for better clarity and smoother flow. - Adjusted wording and phrasing to improve accuracy and professionalism. - Optimized the organization of information for better readability, especially in the sections related to parameters and technical details. - Refined formatting and sectioning of the documentation for easier navigation and comprehension.
2025-01-30 05:35:58 -05:00
- Parameters: **11.5B unique parameters** (excluding the shared 0.9B Embedding and 0.9B Output Head).
- Activation parameters: **2.4B** (including the shared 0.9B Embedding and 0.9B Output Head).
Release DeepSeek-V3
2024-12-26 06:01:57 -05:00
#### Structural Details
Improve DeepSeek-V3 Weight File Documentation for Clarity and Readability - Enhanced sentence structure for better clarity and smoother flow. - Adjusted wording and phrasing to improve accuracy and professionalism. - Optimized the organization of information for better readability, especially in the sections related to parameters and technical details. - Refined formatting and sectioning of the documentation for easier navigation and comprehension.
2025-01-30 05:35:58 -05:00
- **embed_tokens**: **Shares parameters** with the Main Model’s Embedding layer.
- **enorm & hnorm**: RMSNorm parameters used for speculative decoding.
- **eh_proj**: Parameters used for dimensionality reduction of the normalized outputs.
Release DeepSeek-V3
2024-12-26 06:01:57 -05:00
- **Additional Transformer Hidden Layer**:
Improve DeepSeek-V3 Weight File Documentation for Clarity and Readability - Enhanced sentence structure for better clarity and smoother flow. - Adjusted wording and phrasing to improve accuracy and professionalism. - Optimized the organization of information for better readability, especially in the sections related to parameters and technical details. - Refined formatting and sectioning of the documentation for easier navigation and comprehension.
2025-01-30 05:35:58 -05:00
- `model.layers.61.self_attn & mlp` (these are structured the same as the Main Model hidden layers).
- **shared_head**: **Shares parameters** with the Output Head of the Main Model.
Release DeepSeek-V3
2024-12-26 06:01:57 -05:00
---
Improve DeepSeek-V3 Weight File Documentation for Clarity and Readability - Enhanced sentence structure for better clarity and smoother flow. - Adjusted wording and phrasing to improve accuracy and professionalism. - Optimized the organization of information for better readability, especially in the sections related to parameters and technical details. - Refined formatting and sectioning of the documentation for easier navigation and comprehension.
2025-01-30 05:35:58 -05:00
### Layer Loading Rules
Release DeepSeek-V3
2024-12-26 06:01:57 -05:00
Improve DeepSeek-V3 Weight File Documentation for Clarity and Readability - Enhanced sentence structure for better clarity and smoother flow. - Adjusted wording and phrasing to improve accuracy and professionalism. - Optimized the organization of information for better readability, especially in the sections related to parameters and technical details. - Refined formatting and sectioning of the documentation for easier navigation and comprehension.
2025-01-30 05:35:58 -05:00
- **Main Model Weights**: These are loaded according to the `num_hidden_layers` field in `config.json`.
- **MTP Modules**: These are loaded using the `num_nextn_predict_layers` field, with MTP layer IDs appended directly after the Main Model’s hidden layers. For example:
- With `num_hidden_layers = 61` and `num_nextn_predict_layers = 1`, the MTP Module layer ID will be `61`.
Release DeepSeek-V3
2024-12-26 06:01:57 -05:00
---
## FP8 Weight Documentation
Improve DeepSeek-V3 Weight File Documentation for Clarity and Readability - Enhanced sentence structure for better clarity and smoother flow. - Adjusted wording and phrasing to improve accuracy and professionalism. - Optimized the organization of information for better readability, especially in the sections related to parameters and technical details. - Refined formatting and sectioning of the documentation for easier navigation and comprehension.
2025-01-30 05:35:58 -05:00
DeepSeek-V3 natively supports the FP8 weight format with 128x128 block scaling.
Release DeepSeek-V3
2024-12-26 06:01:57 -05:00
### FP8 Configuration
Improve DeepSeek-V3 Weight File Documentation for Clarity and Readability - Enhanced sentence structure for better clarity and smoother flow. - Adjusted wording and phrasing to improve accuracy and professionalism. - Optimized the organization of information for better readability, especially in the sections related to parameters and technical details. - Refined formatting and sectioning of the documentation for easier navigation and comprehension.
2025-01-30 05:35:58 -05:00
The FP8 weight file introduces a `quantization_config` field, which defines the quantization method. Below is an example of the configuration:
Release DeepSeek-V3
2024-12-26 06:01:57 -05:00
```json
"quantization_config": {
"activation_scheme": "dynamic",
"fmt": "e4m3",
"quant_method": "fp8",
"weight_block_size": [128, 128]
}
```
- **Quantization Format**:
Improve DeepSeek-V3 Weight File Documentation for Clarity and Readability - Enhanced sentence structure for better clarity and smoother flow. - Adjusted wording and phrasing to improve accuracy and professionalism. - Optimized the organization of information for better readability, especially in the sections related to parameters and technical details. - Refined formatting and sectioning of the documentation for easier navigation and comprehension.
2025-01-30 05:35:58 -05:00
- Format type: `fp8` and `e4m3` (aligned with `torch.float8_e4m3fn`).
Release DeepSeek-V3
2024-12-26 06:01:57 -05:00
- Weight block size: `128x128`.
- **Activation Quantization Scheme**:
Improve DeepSeek-V3 Weight File Documentation for Clarity and Readability - Enhanced sentence structure for better clarity and smoother flow. - Adjusted wording and phrasing to improve accuracy and professionalism. - Optimized the organization of information for better readability, especially in the sections related to parameters and technical details. - Refined formatting and sectioning of the documentation for easier navigation and comprehension.
2025-01-30 05:35:58 -05:00
- Uses dynamic activation quantization (`dynamic`).
Release DeepSeek-V3
2024-12-26 06:01:57 -05:00
### Dequantization Method
The FP8 weight file includes a `weight_scale_inv` field, which stores the dequantization scale for each weight block.
Improve DeepSeek-V3 Weight File Documentation for Clarity and Readability - Enhanced sentence structure for better clarity and smoother flow. - Adjusted wording and phrasing to improve accuracy and professionalism. - Optimized the organization of information for better readability, especially in the sections related to parameters and technical details. - Refined formatting and sectioning of the documentation for easier navigation and comprehension.
2025-01-30 05:35:58 -05:00
- **Storage Format**: Stored as a `float32 Tensor`, alongside the weight data.
Release DeepSeek-V3
2024-12-26 06:01:57 -05:00
- **Dequantization Formula**:
Improve DeepSeek-V3 Weight File Documentation for Clarity and Readability - Enhanced sentence structure for better clarity and smoother flow. - Adjusted wording and phrasing to improve accuracy and professionalism. - Optimized the organization of information for better readability, especially in the sections related to parameters and technical details. - Refined formatting and sectioning of the documentation for easier navigation and comprehension.
2025-01-30 05:35:58 -05:00
- If a weight block is not aligned to 128, it is zero-padded to 128 before calculating the scale. The padded portion is discarded after quantization.
- Dequantization is performed using the formula: `(128x128 weight block) * weight_scale_inv`.
Release DeepSeek-V3
2024-12-26 06:01:57 -05:00
Improve DeepSeek-V3 Weight File Documentation for Clarity and Readability - Enhanced sentence structure for better clarity and smoother flow. - Adjusted wording and phrasing to improve accuracy and professionalism. - Optimized the organization of information for better readability, especially in the sections related to parameters and technical details. - Refined formatting and sectioning of the documentation for easier navigation and comprehension.
2025-01-30 05:35:58 -05:00
This dequantization process enables runtime operations to apply online quantization on a per-token, per-128-channel basis.
Reference in New Issue Copy Permalink