Language Model
Small Language Models (SLMs), Large Language Models (LLMs) and their application
Local SLM Optimization Studio & Hardware Performance Benchmarker
Architected and engineered an intelligent, 100% data-sovereign AI profiling application designed to benchmark and compare local Small Language Models (SLMs) running on resource-constrained consumer hardware. Built a high-utility monitoring pipeline that hooks into active background system daemons using psutil to isolate model footprint metrics from baseline operational overhead. Incorporating native Ollama streaming wrappers, the system captures precise performance metrics—including Peak RAM footprint, hardware swap-file invocation, true first-token load times, and native tokens-per-second generation speeds—compiling the multi-model data into beautiful interactive UI elements and multi-page technical PDF reports.
Metrics at a Glance:
Key Architectural Achievements:
- Process-Isolated Memory Tracking: Engineered targeted resource isolation logic utilizing psutil to query the background Ollama daemon process tree, cleanly tracking Resident Set Size (RSS) and Virtual Memory Size (VMS) to reveal true model footprints instead of skewed system-wide deltas.
- Native Token Throughput Analysis: Bypassed crude whitespace splitting fallbacks by extracting real-time performance attributes natively from Ollama's final chunk response metadata, calculating true generation speed using precise nanosecond-level execution durations.
- Dynamic Model Weight Mapping: Developed a dynamic model tag routing engine supporting strict quantization levels (such as Q4_K_M vs Q8_0 setups) across diverse model families (Llama 3.2, Phi-3, and Qwen 2.5), allowing developers to analyze speed-vs-quality trade-offs accurately.
- Asynchronous UI Stream Pipelines: Built a highly performant data streaming layer inside Streamlit that cleanly decouples background inference generation from foreground chart paint routines, eliminating UI locks, memory leaks, and interface stuttering during multi-token execution.
- In-Memory Chart & Report Export: Integrated headless server image generation with Plotly and Kaleido to convert live UI charts into static image buffers, feeding the streams directly into an FPDF document layout generator to build downloadable multi-page PDF performance records completely in memory.