How a 28.9-Million-Parameter Language Model Runs on an ESP32-S3
Running a transformer language model directly on a microcontroller is normally constrained not only by computational throughput, but more fundamentally by the amount of fast memory available to hold model parameters, activations, temporary buffers and the attention state. A project published in July 2026 demonstrated a different approach: a 28.9-million-parameter language model running entirely on an ESP32-S3, generating approximately 9.5 tokens/s end-to-end. The result is not based on fitting the complete model into SRAM. Instead, it relies on a deliberate decomposition of the model across the device's memory hierarchy, with a large Per-Layer Embedding (PLE) table retained in flash and only the computationally active portion of the model kept in fast memory.
This result deserves attention because it demonstrates a broader principle in efficient neural-network deployment: the effective size of a model is determined not only by parameter count, but also by the access pattern associated with those parameters. Parameters that participate in dense computation at every inference step have fundamentally different memory requirements from parameters that can be accessed through sparse lookup.
The discussion below analyzes the architecture quantitatively, derives the relevant memory requirements, explains the role of 4-bit quantization and Per-Layer Embeddings, and examines the limitations of the demonstrated system. The distinction between the underlying PLE architecture and its application to the ESP32-S3 is particularly important: PLE is an architectural technique developed and used in Google's Gemma-family models; the contribution of this implementation is its use as a mechanism for fitting a substantially larger stored model into the memory hierarchy of the ESP32-S3. [1] [2]
1. The Memory Problem
The fundamental difficulty is straightforward to state. The demonstrated model contains approximately 28.9 × 106 stored parameters. If those parameters were represented using 32-bit floating-point values, the storage requirement would be
Even FP16 would require approximately
Neither representation is remotely compatible with the fast on-chip memory budget of the ESP32-S3 configuration used in the demonstration. The system has approximately 512 KB of internal SRAM, 8 MB of PSRAM and 16 MB of flash. The problem therefore cannot be solved by conventional loading of the entire model into SRAM or by treating all parameters identically. [3]
This immediately suggests a more useful formulation of the deployment problem:
2. Quantization Reduces Storage, but Does Not Solve the Problem by Itself
The first reduction is achieved through 4-bit post-training quantization. If each weight is represented by four bits, the theoretical storage cost of the raw weight values becomes
In practice, a quantized representation also requires scaling information. The project therefore reports a final exported model size of approximately 14,912,332 bytes, rather than the idealized 14.45 MB. [4]
A simplified symmetric quantization model can be expressed as
where w is the original floating-point parameter, q is the quantized integer representation and s is a scale factor. The resulting quantization error is
Group-wise quantization reduces the overhead associated with scale factors while retaining substantially better fidelity than applying a single global scale to an entire tensor. The reported experiments used group-wise symmetric INT4 post-training quantization, with the final flash representation using a tighter format to satisfy the available flash partition. [4]
Quantization is therefore necessary, but it is not sufficient. Even an approximately 14.9 MB quantized model remains much larger than the internal SRAM budget. The central problem becomes one of parameter placement.
3. The Critical Observation: Parameter Count Is Not Equivalent to Compute Cost
A conventional view of a language model treats all parameters as part of a single monolithic weight set. Such a view is particularly inconvenient for a resource-constrained processor because it implicitly assumes that all parameters must be equally accessible during inference.
The deployed architecture instead separates the stored model into three major functional regions:
| Component | Approximate parameters | Primary location | Functional role |
|---|---|---|---|
| Dense computational core | ≈ 559 K | SRAM / flash-mapped in the final firmware | Repeated transformer computation |
| Output head | ≈ 3.1 M | PSRAM | Generation of vocabulary logits |
| PLE lookup table | ≈ 25 M | Flash | Sparse per-token, per-layer lookup |
Consequently, the statement that the device runs a “28.9-million-parameter model” must be interpreted precisely. The number represents the total number of parameters stored by the deployed model. It does not mean that a conventional 28.9-million- parameter dense transformer is repeatedly evaluated entirely from SRAM. [4]
This distinction is central to understanding the result.
4. Per-Layer Embeddings
Per-Layer Embeddings are designed to provide an auxiliary representation to individual decoder layers rather than relying exclusively on one shared token embedding at the beginning of the network.
In a conventional embedding layer, a vocabulary of size V and embedding dimension d gives an embedding matrix
Given a token identifier t, the embedding operation is simply a row selection:
The computational character of this operation is therefore fundamentally different from a dense matrix multiplication. For a dense layer,
the processor must perform a large number of multiply-accumulate operations involving the elements of W. An embedding lookup, by contrast, requires retrieving a comparatively small vector associated with an identifier.
Google describes PLE in Gemma as a mechanism in which per-layer embedding parameters can be kept outside the principal model memory and made available to each layer during inference. In the Gemma architecture, each decoder layer receives an auxiliary per-layer signal derived from the PLE mechanism. [1] [2]
4.1 Mathematical representation of PLE injection
A simplified representation of the token-identity component is
where Ei denotes the embedding table associated with layer i and t is the current token identifier.
The full modern PLE implementation also incorporates a context-dependent projection. In simplified form, one can represent the combined per-layer signal as
where pi is the token-identity contribution and ci is the context-dependent component. The exact normalization and scaling operations depend on the model implementation. [5]
The important systems property is that these parameters can be accessed through lookup rather than dense streaming.
5. Why the PLE Table Can Reside in Flash
The demonstrated implementation stores approximately 25 million PLE parameters in flash. At 4-bit precision, this represents roughly 12 MB of lookup data. The project reports that, for each generated token, only approximately six rows of the PLE table are accessed, corresponding to roughly 450 bytes per token. [4]
This is the critical memory-bandwidth observation.
If the complete 14.9 MB model had to be streamed for every generated token, the required parameter bandwidth at 10 tokens/s would be approximately
That is not the access pattern of the deployed PLE system. For the flash-resident table, the measured access is on the order of
The two quantities differ by several orders of magnitude. The entire lookup table is therefore not continuously transferred through the memory interface. Instead, its size is large while its per-token traffic is small.
This is precisely why flash becomes viable as a repository for the large parameter component.
6. The Memory Hierarchy Is Part of the Model Architecture
The implementation can be understood as a three-tier data-placement strategy:
| Memory tier | Required property | Data stored |
|---|---|---|
| Internal SRAM | Very low latency; tightly constrained capacity | Dense model state and critical computation |
| PSRAM | Larger capacity; lower performance than SRAM | Working memory, KV state and output head |
| Flash | Large persistent storage; comparatively high access latency | Large PLE lookup table |
The general optimization criterion can be expressed as an access-frequency principle:
Conversely:
This reframes model deployment as a joint optimization problem involving neural-network structure and memory architecture rather than as a simple exercise in parameter compression.
7. Quantitative Deployment Configuration
The published deployment configuration is unusually specific. The reported values include:
| Parameter | Reported value |
|---|---|
| Vocabulary | 32,768 tokens |
| Dense core | ≈ 559 K parameters |
| PLE table | ≈ 25 M parameters |
| Total stored parameters | ≈ 28.9 M |
| Model dimension | 96 |
| Number of layers | 6 |
| PLE dimension | 128 |
| Quantization | 4-bit |
| Exported model size | 14,912,332 bytes |
| End-to-end generation rate | ≈ 9.5 tokens/s |
| Pure model-step rate | ≈ 9.72 tokens/s |
The published results report a model step of approximately 102.9 ms for the pure computation measurement and approximately 9.5 tokens/s for the end-to-end system, including serial output. [4]
The reciprocal relationship is
At 9.5 tokens/s:
Thus, the device produces a new token roughly every 105 ms in the reported end-to-end configuration.
8. Model Quality: What the Additional Parameters Actually Achieve
The project evaluates the PLE architecture against a parameter-matched dense baseline. The relevant comparison is not between two models with equal total parameter counts. Rather, the dense computational core is matched at approximately 559 K parameters, while the PLE configuration adds the large flash-resident table.
| Configuration | Dense core | Total stored parameters | Validation perplexity |
|---|---|---|---|
| Baseline | ≈ 559 K | ≈ 3.7 M | 12.58 |
| PLE | ≈ 558 K | ≈ 28.9 M | 11.41 |
The reported reduction is approximately 0.098 nats in validation loss, corresponding to about a 9.3% reduction in perplexity. Importantly, the experiment was repeated with two random seeds, and the reported PLE advantage was substantially larger than the variation attributed to seed noise. [4]
8.1 Perplexity
For a sequence of N tokens, perplexity is commonly written as
Lower perplexity indicates that the model assigns greater average probability to the observed test sequence. Perplexity is therefore useful for comparing the language-model objective, although it does not by itself establish general reasoning or instruction-following capability.
9. The Result Survives 4-Bit Quantization
A particularly relevant question is whether the architectural gain disappears when the model is aggressively quantized.
The reported experiments indicate that it does not. The PLE advantage relative to the dense baseline remained after 4-bit post-training quantization. The project reports separate experiments using group-wise 4-bit quantization as well as the exact tighter representation used by the final flash artifact. [4]
This is technically significant because it indicates that the observed improvement is not merely an artifact of an impractically large full-precision implementation. The PLE structure remains useful under the numerical constraints imposed by embedded deployment.
10. Verification of the On-Device Implementation
The project does not rely only on generated text as evidence of correctness. The published results report a comparison between the portable C implementation and the PyTorch reference: the exported runtime matched all 32,768 output logits with a maximum absolute difference of approximately 10−5 before deployment to the target device. [4]
This verification step is important in embedded neural-network deployment because quantization, integer staging, tensor layout transformations and custom kernels can each introduce implementation errors that are not obvious from qualitative output alone.
A useful correctness criterion is therefore
For the published comparison, the reported maximum absolute deviation is approximately 10−5 over the complete output vector.
11. Why the Demonstration Should Not Be Described as “ChatGPT on an ESP32”
The architectural result should not be confused with general-purpose language-model capability.
The demonstrated network is trained in the TinyStories setting. TinyStories was introduced specifically to study whether very small language models can produce coherent language when trained on a constrained and synthetic corpus. The original work showed that models substantially smaller than conventional language models can produce coherent stories under the appropriate training conditions. [6]
Consequently, the demonstrated system should be interpreted as a small-domain generative language model, not as a reduced implementation of a modern general-purpose conversational model.
This distinction is especially important when comparing models using parameter counts. Two models with similar numbers of stored parameters can have radically different computational structure, training distributions and capabilities.
12. Why the Result Is More Interesting as a Systems Problem Than as a Parameter-Count Record
The most important contribution of the demonstration is the interaction between neural architecture, numerical representation and memory hierarchy.
The deployed model can be represented conceptually as
The dense component is constrained by fast-memory capacity. The large parameter component is constrained mainly by non-volatile storage capacity and lookup latency.
This changes the optimization problem from
to a more useful constrained problem:
Here, Cinference is computational cost, Mfast is the amount of data that must reside in fast memory, and Bslow is the volume of data transferred from the slower storage tier.
PLE is attractive precisely because it changes the relationship between these quantities.
13. Arithmetic Intensity and Data Movement
The performance of an embedded neural-network implementation cannot be characterized exclusively by arithmetic throughput. Data movement is often equally important.
A convenient abstraction is arithmetic intensity:
If a large weight tensor must be repeatedly streamed from a slower memory tier, then performance can become memory-bandwidth limited before the arithmetic units are saturated.
The PLE architecture changes this balance because the largest parameter block is accessed through sparse lookup rather than dense streaming. The system therefore exchanges capacity for access locality.
The approach is not universally applicable. It works because the large parameter set has a lookup-oriented access pattern. A conventional dense matrix containing millions of parameters cannot generally be treated in the same way without changing the model architecture.
14. Flash Latency Is Still Real
An important technical qualification is that flash is not “free”.
The published measurements explicitly note that random-read latency remains measurable. The implementation succeeds because the PLE dimensionality and number of layers keep the number of accesses sufficiently small. The project estimates that the observed approximately 20 μs random-read latency remains negligible at the selected configuration, but would become increasingly relevant as the width of the lookup table grows. [4]
This leads to an important scalability condition:
Increasing the number of per-token lookups or the number of independent flash accesses will eventually turn the memory system into the dominant bottleneck.
Therefore, PLE does not eliminate the memory-bandwidth problem. It changes its scaling law.
15. The Output Head Becomes Another Bottleneck
The published implementation also exposes a second important issue: the output head. The vocabulary contains 32,768 tokens, and the head is sufficiently large that its memory traffic contributes materially to the per-token latency.
The project reports that a substantial fraction of the measured head execution time is associated with reading data from PSRAM, implying that the limiting factor is not necessarily raw arithmetic throughput. [4]
This observation is useful because it demonstrates that optimization must be performed end-to-end. Improving the transformer core alone does not guarantee a proportional increase in token generation rate.
16. A Useful Energy Perspective
For an embedded inference system, token throughput is only one performance metric. Energy consumption per generated token is arguably more useful when the system operates from a constrained power source.
If the average system power is P and the generation rate is R tokens/s, then
Equivalently, since the token period is 1/R,
The published ESP32 project emphasizes model fitting and throughput rather than presenting a complete energy-per-token characterization. Therefore, any numerical energy claim would require additional experimental measurements and should not be inferred from the reported token rate alone.
17. What the Experiment Actually Demonstrates
The technically defensible interpretation of the result can be summarized in five points.
- A 28.9-million-parameter language model can be deployed on the ESP32-S3 when the parameter set is distributed across multiple memory tiers rather than loaded uniformly into SRAM.
- Approximately 25 million parameters can be represented as a large PLE lookup table resident in flash.
- The repeatedly computed dense component can be constrained to approximately 559 K parameters, compatible with the fast-memory budget.
- 4-bit quantization reduces the total stored representation to approximately 14.9 MB while retaining the measured PLE advantage.
- The complete system generates approximately 9.5 tokens/s end-to-end according to the published on-device measurement.
These are substantially more precise claims than simply stating that “a 28.9M LLM fits on an ESP32”.
18. What the Experiment Does Not Demonstrate
Several conclusions should explicitly not be drawn from the experiment.
- It does not demonstrate that a conventional dense 28.9-million-parameter transformer can be loaded entirely into 512 KB of SRAM.
- It does not demonstrate general-purpose language understanding equivalent to a modern instruction-tuned model.
- It does not imply that model capability scales proportionally with the number of flash-resident PLE parameters.
- It does not imply that arbitrary large language models can be placed in flash without a corresponding change in architecture and access pattern.
19. Relationship to Google's Gemma Architecture
The terminology and underlying mechanism should also be placed in the correct historical context. Google's Gemma documentation describes Per-Layer Embeddings as a mechanism that permits large embedding parameters to be kept outside the main operating-memory footprint, with the resulting per-layer information injected during inference. [1]
The ESP32 implementation takes advantage of precisely the property that makes PLE attractive for resource-constrained inference: the embedding parameters are large in aggregate but relatively inexpensive to access on a per-token basis.
Therefore, the important engineering step is not the invention of PLE itself, but the recognition that this architecture maps unusually well onto a memory hierarchy in which flash capacity is several orders of magnitude larger than the fast computational memory budget.
20. Broader Significance for Small Language Models
The result is consistent with a broader research direction in which model architecture and deployment hardware are optimized jointly rather than independently.
Work such as Deeploy has already demonstrated that efficient small-language-model inference on MCU-class processors requires explicit consideration of memory placement, computation scheduling and specialized execution paths. [7]
The ESP32 demonstration highlights a complementary strategy: rather than attempting to reduce every parameter to a form suitable for fast memory, the architecture distinguishes between parameters that must be computed repeatedly and parameters that can be accessed selectively.
In that sense, the result is best viewed as a hardware-aware neural-network architecture experiment.
21. A General Mathematical View of the Deployment Problem
The deployment can be abstracted as the following constrained optimization problem:
subject to
The PLE strategy changes the feasible region of this optimization problem. Parameters that would otherwise contribute directly to the fast-memory constraint can instead contribute primarily to the flash-capacity and lookup-bandwidth constraints.
That is the fundamental reason why the architecture can support a substantially larger stored model than the fast-memory capacity would otherwise suggest.
22. Conclusion
The successful deployment of a 28.9-million-parameter language model on an ESP32-S3 should not be reduced to the headline that “an LLM now runs on a microcontroller”. The technically significant result is more specific.
A model whose total parameter storage is approximately 14.9 MB after quantization can operate within a system whose fast internal memory is only a small fraction of that size when its parameters are partitioned according to their computational access patterns. Approximately 25 million parameters are represented by a flash-resident PLE lookup table, while the dense computational core remains close to the SRAM capacity constraint.
The resulting system reaches approximately 9.5 tokens/s end-to-end, and the published experiments show that the PLE configuration improves validation perplexity relative to a same-core baseline while retaining this advantage after 4-bit quantization. [4]
More importantly, the demonstration establishes a design principle:
PLE provides a mechanism through which a large parameter store can be separated from the high-frequency computational state. Quantization reduces its storage cost, and selective lookup prevents the complete parameter store from becoming a prohibitive bandwidth requirement.
The ESP32-S3 experiment therefore represents a particularly clear example of memory–architecture co-design for edge inference. Its importance is not that a small device has suddenly acquired the capabilities of a general-purpose large language model. Rather, it demonstrates that carefully structured neural networks can exploit heterogeneous memory resources in ways that conventional dense-model deployment cannot.
For embedded artificial intelligence, this distinction is fundamental: increasing the computational capacity of a model and increasing its stored parameter capacity are not necessarily the same problem. Per-Layer Embeddings exploit that difference explicitly.
References
- Google AI for Developers, Gemma 3n Model Overview — Per-Layer Embedding (PLE). https://ai.google.dev/gemma/docs/gemma-3n
- Google AI for Developers, Gemma 4 Model Card. https://ai.google.dev/gemma/docs/core/model_card_4
- slvDev, esp32-ai: Running a 28.9M parameter LLM on an ESP32-S3. https://github.com/slvDev/esp32-ai
- slvDev, esp32-ai — Results and Ablation Study. https://github.com/slvDev/esp32-ai/blob/main/RESULTS.md
- Hugging Face Transformers, Gemma 4 documentation — Per-Layer Embeddings. https://github.com/huggingface/transformers/blob/main/docs/source/en/model_doc/gemma4.md
- Eldan, R. and Li, Y., TinyStories: How Small Can Language Models Be and Still Speak Coherent English?, arXiv:2305.07759, 2023. https://arxiv.org/abs/2305.07759
- Scherer, M. et al., Deeploy: Enabling Energy-Efficient Deployment of Small Language Models on Heterogeneous Microcontrollers, arXiv:2408.04413, 2024. https://arxiv.org/abs/2408.04413
Comments
Post a Comment