Community Learning

An agent harness is out of the box token efficient because it heavily relies on prompt caching.

But, these best practices are important so that you can actually maintain those savings turn after turn without accidentally breaking the cache.

When building complex multi-agent systems, the harness holds all the heavy lifting: tool definitions, system instructions, and loaded skills.

Without caching, you pay for this massive background context on every single step.

To ensure your setup stays token efficient, structure your context so the cache actually hits:

1) Keep the prefix static:
Caching matches the start of your request. Put core tool definitions and base information at the very top of the context window.

2) Put dynamic data at the end: …

LangChain OpenWiki 0.2 Realizes Karpathy's LLM Wiki Through Google's OKF

The evolution of AI codebase understanding has rapidly advanced through three major milestones: Andrej Karpathy's conceptual LLM Wiki, Google's standardization of the format, and finally, LangChain's practical implementation in OpenWiki 0.2.

1. The Spark: Andrej Karpathy's LLM Wiki

In April 2026, Andrej Karpathy proposed a brilliant concept known as the "LLM Wiki". Instead of relying on scattered search processes on the fly, he suggested pointing an AI agent at a folder of documents and letting the agent autonomously compile and maintain a structured knowledge base.

The AI reads the source material once and writes comprehensive, Wikipedia-style markdown files. Over time, these files form an interlinked knowledge graph. When you ask …

Yesterday, PrismML announced 27 Billion parameter model, the first multi-modal model of its capability class to run on a phone.

Bonsai 27B comes with great agentic capabilities: multi-step reasoning, structured tool calls, vision tasks, and computer-use, and agentic loops that stay coherent across many steps.

Until today, deploying that tier locally has been impractical for a concrete reason: a 27B model occupies roughly 54GB in 16-bit precision, and even a good 4-bit build, at 18GB, is too large for a phone and for most laptops.

Bonsai 27B changes that. It comes in two variants:

Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight. At 5.9 GB, it is …

There is a reason why Deep Tech companies spent so much money to train larger models: they knew something important.

In 2023, Ilya Sutskever, one of the OpenAI Co-Founders, said:

“If you train a large and deep neural network on a big enough data set, then you will succeed necessarily.”

This wasn't just a bold prediction. Ilya knew it was a mathematical certainty.

His confidence stemmed from a landmark 2020 research paper called the “Scaling Laws for Neural Language Models.” The researchers studied what actually makes an AI smarter, and proved that performance is predictably driven by 3 main factors:

a) Compute
b) Dataset Size
c) Parameter Count

In short, the most notable observation was that LLM performance improves …

I've been digging into MeshLLM and it's worth a look for anyone here running models locally.

The core idea: it pools the GPUs and memory across the machines you already own and exposes the whole pool as one OpenAI-compatible API at localhost:9337. Point any OpenAI client at it and stop caring which machine does the work. A request runs one of three ways: locally on your GPU, routed to a peer that already has the model loaded, or split across several machines for a model too big for any single box.

It runs on top of llama.cpp, which provides functionalities such as the inference itself and even splitting a model across machines. That distribution works through llama.cpp's RPC support: you …

How is this even possible, how can someone run a 744 billion parameter frontier model on a local machine ? This was my exact reaction when I stumbled upon this repository https://github.com/JustVugg/colibri

It runs GLM-5.2 on a laptop with just 25 GB of RAM. No GPU. Pure C. Zero dependencies.

My first instinct was, no way. You simply cannot run a model this huge on consumer hardware. That's the rule, right?

Turns out the rule had a hidden assumption: that the whole model must sit in RAM. But GLM-5.2 is a Mixture of Experts model. Out of 744B parameters, only about 40B are active per token. So colibri keeps just the always-needed dense part in RAM (about 10 GB) and …

Standard RAG

- The query is converted into an embedding and matched against a vector database.
- The top-K closest chunks are pulled out and passed to the LLM as context.
- The LLM writes a grounded answer using only what was retrieved.

Graph RAG

- The query is classified: specific questions route to local search, broad questions route to global search.
- Local search: query embedded → vector DB finds matching entities → pipeline traverses across the knowledge graph collecting linked context → LLM synthesis final answer.
- Global search: no vector search, no graph traversal → community reports loaded in batches → LLM scores each for relevance → top-ranked context → LLM synthesizes final response.

Agentic RAG

- …

LMCache is a KV cache management layer for LLM inference. It turns KV cache from a temporary state into reusable AI-native knowledge that can be stored persistently, reused across multiple serving engines, monitored with an observability stack, and transformed for better generation quality. As a result, LMCache reduces TTFT (time-to-first-token) and improves throughput, especially for long-context agentic, multi-turn conversation, and knowledge-augmented workloads (e.g., RAG).

LMCache is vendor-neutral. It can be used as a KV cache layer for a range of mainstream open-source serving engines, inference frameworks, hardware vendors, storage systems, and infrastructure providers. The vendor neutrality allows users to freely switch between serving engines and storage vendors, while reusing the stored KV caches.

Inside the GPU

(https://www.hubbleflow.ai/resources/blog/gpu-architecture)

In this post, we explore what actually happens on the silicon when you spin up a Large Language Model. We break down an NVIDIA RTX 3090 running Qwen3 locally via Ollama.

Instead of just reading about it, you can interact with a scroll-driven 3D teardown. Strip away the heatsinks to explore:

- The GDDR6X memory layout and VRAM limits.

* The Graphics Processing Clusters that handle matrix multiplications.

* The physical architecture powering your local AI workflows.

DiffusionGemma is Google DeepMind’s experimental model that generates text in a fundamentally different way from typical LLMs.

Instead of generating text one token at a time like normal LLMs, it generates entire blocks of text simultaneously, delivering up to 4x faster generation on GPUs.

Think of it like this: a normal LLM writes a sentence word by word, left to right. DiffusionGemma instead starts with a “noisy” scrambled block of 256 tokens (a “canvas”) and repeatedly refines it, like watching a blurry image sharpen, until the whole block is clean text. It generates text by iteratively denoising blocks of tokens in parallel, then moves block-by-block (“block-autoregressive multi-canvas sampling”).

Its built on the Gemma 4 26B Mixture-of-Experts architecture with only ~4B …

Open Weight Alternatives to Frontier Models and their cost comparisions

1. Reasoning

Claude Opus 4.8 → DeepSeek V4 Pro
$1.80 → $0.04 per task

~45x cheaper, and it's tied with Opus on SWE-bench (80.6 vs 80.8)

2. Code generation

GPT-5.5 → GLM 5.2
$1.03 → $0.48 per task

~2x cheaper AND it beats GPT-5.5 on long-horizon coding

3. Agents + tool calling

Claude Sonnet 5 → Kimi K2.7
$2.29 → ~$0.31 per task

~7x cheaper, best agentic stability of the open models.

4. Bulk / high volume

Gemini 3.5 Flash → DeepSeek V4 Flash
$0.59 → $0.02 per task

~30x cheaper, plenty for extraction, classification, drafts

The tradeoff:

coding + agents.. near parity or better

hardest reasoning + …

Entire.io : A git compatible version control system for Agentic development

Just spare 2 mins to read this. My hunch is that all repository keepers will provide this as first class feature itself.

Sarvam’s open weight 105B wins on average 90% across all benchmarked dimensions and on average 84% on STEM. math, and coding compared to other open weight models of similar size

Headroom compresses everything your AI agent reads i.e tool outputs, logs, RAG chunks, files, and conversation history, before it reaches the LLM.

You get full accurate answers on fraction of the tokens.

Must try

https://github.com/headroomlabs-ai/headroom

Comparing different local open-weight LLMs across different harnesses (Qwen-Code, Codex, Claude Code) 🧪

Turns out 30B Mixture-of-Expert models land really well. They’ve got enough muscle to crack genuinely hard problems, and they push around 40 tok/sec on a Mac or DGX Spark. That’s roughly what you’d get from GPT 5.5 on a Pro plan, so it’s plenty fast for daily work ⚡️

The part that was a bit surprising was how much the harness matters 🤔 Claude Code seems to take 2x the tokens that Codex does.

Cool new open-weight model by Cohere: a new lightweight 30B open-weight model for agentic coding tasks.

This one builds on Command A+ using the parallel transformer design. Interestingly, even though it's almost half as big, it almost doubles the number of layers.

Also, they say that it's been specifically developed for agentic coding, not just coding. I.e., the evaluation is inside a workflow, not just on a single prompt-to-code-answer task.

For Terminal-Bench, the model has to use a terminal, inspect the environment, run commands, read outputs, etc.

For SWE-Bench the model works on real GitHub-style software issues where it has to understand the repository, find relevant files, make a patch, pass tests, etc.

SciCode and LiveCodeBench are more traditional because …

LPU i.e Language Processing Unit is a type of computer chip designed by Groq specifically to run AI language models (LLMs) quickly and efficiently. The LPU genuinely is a distinct architecture, not a rebranded GPU.

LPU stores weights directly in on-chip static RAM, placing memory next to compute and eliminating the energy-intensive data movement across the memory bus.

It also uses deterministic scheduling, where the compiler pre-plans the exact timing of every instruction, which pushes hardware utilization very high.

For LLM inference, the speed advantage is real and large, third-party figures put Groq at roughly 300 tokens/sec on Llama 2 70B versus 30–40 on an NVIDIA H100.

Note : I took comparision from here

https://introl.com/blog/groq-lpu-infrastructure-ultra-low-latency-inference-guide-2025

Memory poisoning happens when incorrect, adversarial, or irrelevant information gets stored in an agent’s long‑term memory and later influences its behaviour.

Production AI agents do not just process prompts; they accumulate memory. Over time, that memory becomes more important than any single input. If it drifts, the agent drifts. If it is poisoned, the agent becomes untrustworthy.

Poisoning can happen because of "n" number of reasons, for instance

a) A single adversarial or noisy session can introduce long-lived bad memories.
b) Rare but high-impact events stay forever, while better data later does not override them.
c) The agent cannot differentiate a user joke from a serious preference.
d) A model hallucination written back to memory becomes a self-reinforcing "fact".

The …

GLM 5.2, the open weight model competing directly with Opus 4.8 and GPT-5.5, has added something called “IndexShare mechanism.”

You already know that in standard self-attention, every token attends to every other token, so a query compares itself against all keys. That is what makes long context expensive: at 1M tokens, each token doing full attention means roughly a trillion pairwise comparisons per layer. Sparse attention fixes this by having each token attend to only a small subset of earlier tokens instead of all of them. The trick is choosing which subset. DeepSeek Sparse Attention does this with a lightweight scoring step called an indexer: before doing the real attention, it cheaply scores all earlier tokens for each query and …

VibeThinker-3B

According to benchmarks, a new small 3B-parameter model achieves Opus-4.5-/frontier-level coding performance.

It comes with a very good technical report with lots to learn. What's fascinating is, the whole model builds on the old Qwen2.5-Coder-3B stack (yes, Qwen2.5, not Qwen3.5).

So, that's a pretty clear example that highlights how much of the performance gains come from good data curation and post-training pipelines.

Based on the tech report, here are some of the important pieces of their post-training stack:

1. High-signal synthetic data (math problems with credible solutions, code with tests)

2. Multiple reasoning paths for each answer

3. Filtering, filtering, filtering

4. 2-stage SFT (start with broad training, then train on hard long-reasoning samples)

5. Use target (pass@k) accuracy …

Under the Hoods : Mem0 AI Agent’s Memory Decay

An agent’s memory keeps filling up, and most of what lands there goes stale fast. Some facts stay useful forever, like a user always wanting their code in Java, or which timezone your teammate works in. Others are useful exactly once, like the stack trace you debugged last Tuesday or the ID of a batch job that finished three weeks ago. A memory store that treats both the same way slowly drowns in things nobody needs.

So you want recent and frequently used memories to rank higher, and stale ones to fade.

Recency and frequency. We have all seen these two signals before. They are the exact knobs behind cache eviction. …

Note: GLM 5.2, MiniMax M3 and Kimi K2.6 are all open weight models (analogous to Open Source Tech) 🤯

SWE-Bench Pro is a benchmark designed to provide a rigorous and realistic evaluation of AI agents for software engineering. It was developed to address several limitations in existing benchmarks by tackling four key challenges:

1) Data Contamination: Models have likely seen the evaluation code during training, making it hard to know if they are problem-solving or recalling a memorized solution.
2) Limited Task Diversity: Many benchmarks fail to capture the full spectrum of real-world software challenges and instead focus on simple utility libraries.
3) Oversimplified Problems: Ambiguous or underspecified issues are often removed from benchmarks, which doesn't reflect a real developer's …

What actually is cross-entropy loss in an LLM? And what does the term "entropy" even mean when it comes to language? 🧠

Let's break it down. 👇

Hmmm… we all know that the primary objective of an LLM is basically to predict the next token for a given input sequence. So, if I give this sentence to an LLM:

"Sachin Tendulkar is a great _____"

The LLM will give me a list of tokens. In this case:

Batsman (0.60)
Cricketer (0.25)
Sportsman (0.10)
and so on.

Associated with each token is a probability value that tells us how relevant it is to the context. Just to state the obvious, the sum of the probabilities of all output tokens is exactly …

Why RL(Reinforcement Learning) is quite important

Most of the agentic gains we have seen recently came from RL in the post-training phase. Models that think longer before answering, stay coherent on long-horizon tasks, and keep coding without losing the thread. Cursor's Composer is the clearest case. It was specialized for software engineering through RL and picked up behaviors on its own like running tests and fixing linter errors. Cursor now runs RL online, shipping a better Composer checkpoint several times a day using real user interactions as the reward. Anthropic's latest Claude Opus 4.8 and Google's Gemini 3 lean on the same approach, post-trained with RL for reasoning and long-horizon agentic work.

A quick primer for anyone newer to this. …