DeepSeek-V3/experimental

DeepZig V3 Implementation 🚀

A high-performance implementation of DeepSeek V3 in Zig for blazingly fast inference.

⚠️ Status: Experimental Foundation

This project provides an experimental foundation for DeepZig V3 with working draft implementation:

• ✅ HTTP server with OpenAI-compatible API
• ✅ BLAS-accelerated tensor operations (Apple Accelerate working)
• ✅ Cross-platform build system (Zig 0.15.0-dev)
• ✅ Memory management and backend architecture
• ✅ Apple Silicon detection and optimization
• ✅ Functional matrix operations (significant performance improvement)

Recent Progress: Matrix operations now use BLAS acceleration
Performance Status: 1000+ GFLOPS with Apple Accelerate backend working

See Performance Results for detailed benchmarks.

Overview

This experimental implementation aims to leverage Zig's unique advantages for systems programming to create a high-performance LLM inference engine:

• Zero-cost abstractions with compile-time optimization
• Direct hardware access for SIMD and platform-specific optimizations
• Manual memory management without garbage collection pauses
• Single binary deployment with no runtime dependencies
• Cross-platform compilation for multiple architectures

🚀 BLAS Acceleration Achieved! We've successfully integrated Apple Accelerate backend delivering 1000+ GFLOPS performance - a 3000x speedup over the initial naive implementation.

🔗 Related: See the main project README for architecture overview and vision.

Project Structure

experimental/
├── build.zig              # Build system configuration
├── build.zig.zon          # Package dependencies  
├── src/
│   ├── main.zig           # HTTP server entry point
│   ├── core/              # Core ML components
│   │   ├── root.zig       # Module exports
│   │   ├── tensor.zig     # SIMD-optimized tensors
│   │   ├── model.zig      # DeepSeek V3 model
│   │   ├── attention.zig  # MLA attention mechanism
│   │   ├── moe.zig        # Mixture of Experts
│   │   ├── tokenizer.zig  # Text tokenization
│   │   ├── backend.zig    # Backend abstraction
│   │   ├── memory.zig     # Memory management
│   │   └── math/          # Math utilities
│   │       ├── root.zig   # Math module exports
│   │       ├── simd.zig   # SIMD operations
│   │       ├── activation.zig  # Activation functions
│   │       └── rms_norm.zig    # RMS normalization
│   ├── web/               # HTTP API layer
│   │   ├── root.zig       # Web module exports
│   │   ├── server.zig     # HTTP server (std.http)
│   │   ├── handlers.zig   # Request handlers
│   │   ├── middleware.zig # CORS, auth, rate limiting
│   │   ├── websocket.zig  # WebSocket support
│   │   ├── openai.zig     # OpenAI API compatibility
│   │   ├── request.zig    # Request wrapper
│   │   └── response.zig   # Response wrapper
│   ├── backends/          # Compute backends
│   │   ├── cpu/           # CPU with SIMD
│   │   ├── metal/         # Apple Silicon
│   │   └── cuda/          # NVIDIA GPUs
│   └── wasm/
│       └── main.zig       # WebAssembly entry point
├── bench/
│   └── main.zig           # Performance benchmarks
└── README.md               # This file

Requirements

• Zig 0.15.0-dev
• Platform-specific requirements:
  • macOS: Xcode Command Line Tools (for Metal backend)
  • Linux: CUDA Toolkit (for CUDA backend, optional)
  • Windows: CUDA Toolkit (for CUDA backend, optional)

Quick Start

Building

# Clone and navigate to experimental directory
cd experimental/

# Build the project
zig build

# Run the server
zig build run

# Run tests
zig build test

# Run benchmarks
zig build bench

# Build WebAssembly
zig build wasm

Running the Server

# Start server on default port (8080)
./zig-out/bin/deepseek-v3-zig

# Custom configuration
./zig-out/bin/deepseek-v3-zig --port 3000 --backend metal --model ./path/to/model

API Usage

The server exposes OpenAI-compatible endpoints:

# Chat completion
curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-v3",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 100
  }'

# Health check
curl http://localhost:8080/health

# Model info
curl http://localhost:8080/v1/models

Performance Features

SIMD Optimizations

• x86_64: AVX2/AVX-512 vectorization for matrix operations
• ARM64: NEON SIMD for Apple Silicon optimization
• Auto-vectorization: Compiler-optimized loops with @Vector types

Backend Support

Backend	Status	Features
CPU	✅ Implemented	Multi-threaded, SIMD, cache-optimized
Metal	🚧 In Progress	Apple Silicon GPU, unified memory
CUDA	🚧 Planned	NVIDIA GPU, Tensor Cores
WebGPU	📋 Future	Browser GPU acceleration

Memory Management

• Arena allocators for request-scoped memory
• Memory pools for tensor allocations
• Zero-copy operations where possible
• Cache-friendly data layouts

Development Status

✅ Drafted

• Project structure and build system
• Core tensor operations with SIMD
• HTTP server with OpenAI API compatibility
• CPU backend with optimizations
• Memory management utilities
• Benchmark suite

🚧 In Progress

• DeepSeek V3 model architecture
• Multi-Head Latent Attention (MLA)
• Mixture of Experts (MoE) implementation
• Metal backend for Apple Silicon
• Model loading and weight management

📋 Planned

• CUDA backend for NVIDIA GPUs
• WebSocket streaming
• Model quantization (INT8, FP16)
• Flash Attention optimization
• Distributed inference
• Advanced sampling strategies

Architecture Decisions

Why Zig?

1. Performance: Zero-cost abstractions without runtime overhead
2. Memory Safety: Compile-time memory management without GC
3. Simplicity: Single binary deployment, cross-compilation
4. Control: Direct hardware access for optimization

Design Principles

• Modularity: Clean separation between core, web, and backend layers
• Performance: SIMD-first design with cache-friendly algorithms
• Compatibility: OpenAI API compatibility for easy adoption
• Extensibility: Plugin architecture for new backends

Contributing

This is an experimental project! Contributions are welcome:

1. Core ML: Implement transformer layers, attention mechanisms
2. Backends: Optimize CUDA/Metal compute kernels
3. Performance: Profile and optimize bottlenecks
4. Testing: Add comprehensive test coverage
5. Documentation: Improve setup and usage guides

Development Setup

# Install Zig 0.15.0-dev
# https://ziglang.org/download/

# Clone repository
git clone [repository-url]
cd experimental/

# Run tests during development
zig build test --watch

# Format code
zig fmt src/

Benchmarks

Run benchmarks to measure performance:

zig build bench

Hardware Context: Benchmarks run on Apple M1 MacBook Pro (MacBookPro17,1) with 16GB unified memory, Zig 0.15.0-dev.703+597dd328e, debug build.

Example output:

🚀 DeepZig V3 Performance Benchmarks
==========================================

Backend: CPU (BLAS accelerated)
Architecture: aarch64  
Thread count: 8
Hardware: Apple M1 MacBook Pro, 16GB unified memory

Operation                      | Iterations |  Avg Time | Operations/s | Memory
-------------------------------|------------|-----------|--------------|-------
Tensor Creation (1024x1024)    |   1000 iter |     2.03 ms |        493 ops/s |   4.0 MB
Tensor Addition (SIMD)         |    100 iter |     1.49 ms | 2806962690 ops/s |  48.0 MB  
Matrix Multiplication (BLAS)   |     10 iter |     2.1 ms |      1004 GFLOPS |  12.0 MB
SwiGLU Activation              |   1000 iter |     4.44 ms |  236002478 ops/s |   12.0 MB
RMS Normalization (SIMD)       |   1000 iter |     0.00 ms |    1077586 ops/s |    0.0 MB
Memory Bandwidth               |    100 iter |     4.92 ms |         13 ops/s |  128.0 MB

Known Issues

• Model Loading: Currently creates dummy models - real weight loading not implemented
• Tokenizer: Placeholder implementation - needs proper BPE tokenizer
• WebSocket: Basic structure only - streaming not implemented
• Metal/CUDA: Backend stubs only - GPU kernels not implemented

License

This experimental implementation follows the same license as the original DeepSeek V3 project.

Resources

• Original DeepSeek V3 Paper
• Zig Language Documentation
• Zig Performance Guide
• SIMD in Zig

Is This Ready for Production?

No - this is a research/development foundation. But it's theoretical and compiles:

• What works now: ✅ Compiles and runs with Zig 0.15.0-dev, HTTP server, tensor operations, SIMD math, benchmarks execute successfully
• What's missing: Optimized matrix operations, actual DeepSeek V3 model implementation
• Timeline: Foundation is compiling, model implementation is the next major milestone

Comparison to Other Projects

Project	Language	Status	Focus
This	Zig	Foundation + API	Web-first inference
llama.cpp	C++	Production	CLI/library
Candle	Rust	Production	ML framework
ZML	Zig	Research	Low-level ML ops

Unique advantages: Built-in web server, Zig's zero-cost abstractions, single binary deployment.

---

⚡ Built with Zig for blazing fast LLM inference!

Performance Notes

Current Status: ✅ BLAS integration working - Apple Accelerate backend now functional in draft implementation.

Performance Results (Apple M1, Accelerate backend):

• Matrix 256×256: 0.1ms/iter, 561 GFLOPS (21.6% efficiency)
• Matrix 512×512: 0.2ms/iter, 1129 GFLOPS (43.4% efficiency)
• Matrix 1024×1024: 2.1ms/iter, 1004 GFLOPS (38.6% efficiency)
• Matrix 2048×2048: 21.5ms/iter, 799 GFLOPS (30.7% efficiency)

Performance Improvement: From 6418ms naive → 2.1ms BLAS = significant speedup for matrix operations

System Status:

• ✅ BLAS Backend: Apple Accelerate integration working
• ✅ Efficiency: 20-44% of theoretical maximum (good for draft implementation)
• ✅ Memory Bandwidth: 23.5 GB/s copying, basic optimization
• ✅ Hardware Detection: M-series Apple Silicon detection functional

Next Steps: Focus on transformer architecture, attention mechanisms, and model-specific optimizations for the draft DeepSeek V3 implementation.