Appended Mixed Precision Training (FP16/BF16)
Generated Low-Rank Factorization (SVD) Functionality
Generated Attention Efficiency using Linformer
Reducing Memory & Computational Complexity using FlashAttention
Attached Functionality for Spare Matrices using Butterfly Matrices (Structured Linear Layers)
Generated Function for Low-Rank Approximations
Changes to the Transformer Class:
Efficient Initialization
Uses list comprehension for self.layers instead of a loop.
Consolidated distributed initialization logic.
Memory and Performance Enhancements
Avoids unnecessary operations on tensors.
Uses .shape instead of .size() for clarity.
Code Clarity and Maintainability
Removed redundant variables.
Used in-place operations where applicable.
Changes to the Gate Class:
Replaced linear(x, self.weight) with torch.matmul(x, self.weight.T):
More efficient for linear transformations.
Reduced Redundant Computations:
Avoided unnecessary reassignments.
Merged bias addition into a single step.
Optimized Group-Based Routing:
Used amax instead of unnecessary top-k and sum operations.
Applied in-place scatter operation for memory efficiency.
Simplified Expert Selection:
Directly applied topk for selecting top experts.
* handle missing scale_inv_name
Fixed an issue where `weight` and `weight_scale_inv` (e.g. `model.layers.39.mlp.experts.92.gate_proj.weight` and `model.layers.39.mlp.experts.92.gate_proj.weight_scale_inv`) were not in the same SafeTensor, causing an assertion error due to scale_inv_name not being in the state_dict.
* sort filename to reduce memory costs
* Add CUDA cache clearing in memory management
Added torch.cuda.empty_cache() to free up unused memory on the GPU,
