Beyond Code: DLL Deep Engineering Reference

If code is execution instructions, then this document is the execution contract. It defines the "unwritten rules" of the underlying core in physical timing and memory topology.

Invocation State Machine

stateDiagram-v2 [*] --> EngineCreated: LiteRtLm_CreateEngine EngineCreated --> ConvCreated: LiteRtLm_CreateConversation ConvCreated --> Ready: AppendUserMessage Ready --> Inferencing: LiteRtLm_RunInference Inferencing --> Driving: LiteRtLm_WaitUntilDone Driving --> Ready: bIsDone = 1 Inferencing --> [*]: LiteRtLm_StopMessage EngineCreated --> [*]: LiteRtLm_DestroyEngine

Core Insight: Non-Thread Queuing

The LiteRT-LM DLL does not have an automatic task queue internally. If RunInference is forcibly called on the same Conversation from another thread before the WaitUntilDone blocking loop has finished, it will lead to memory context overwrite or illegal access. Developers must ensure task serialization for the same session at the application layer.

Memory Contract & Ownership

VRAM: Persistent Ownership

KV Cache blocks allocated by Engine are persistent. As long as DestroyConversation is not called, the corresponding VRAM block will be retained forever. This is the physical foundation for achieving "zero-copy Agent switching."

Pointers: Temporary Ownership

The text_chunk in callbacks is the .c_str() of the internal std::string in the DLL.

“When the callback returns, the pointer is destroyed.”

graph TD subgraph DLL[DLL Core Memory] Buffer[Buffer: Hello World] end subgraph App[External App] String[std::string MyResponse] end Buffer -- Pointer --> App Note right of Buffer: Lifecycle limited to Callback stack frame App -- Deep Copy --> String Note over String: Data persistence

Driving Heartbeat Essence

WaitUntilDone is more than just waiting.

Under the WebGPU backend, instructions submitted by RunInference are asynchronous. If no one calls WaitUntilDone, the underlying event loop will remain stagnant.

It is the "crankshaft" of the inference engine: every call drives the advancement of GPU tasks, the production of Tokens, and the triggering of callback functions. This is why we recommend starting while(!done) { WaitUntilDone(...) } in a background thread.

CPU Load
LOW
Event-driven only
Response Latency
< 1ms
Sub-millisecond callback trigger

Full API Contract Dictionary

LiteRtLm_CreateEngine O(Initial Load) 
DLL_EXPORT void* LiteRtLm_CreateEngine(LiteRtLm_Config config);
Physical Behavior:
  • Loads .bin or .gguf weight files.
  • Initializes WebGPU adapter and compiles pipeline.
  • VRAM Allocation: Pre-allocates base memory based on max_num_tokens.
Engineering Trap: If backend specifies GPU but driver version is incompatible, this function may lead to silent process crash or return a null pointer.
LiteRtLm_RunInference Non-Blocking 
DLL_EXPORT void LiteRtLm_RunInference(void* conv, LiteRtLm_SamplingParams p, LiteRtLmCallback cb, void* user);

Incremental Logic: This function detects the current state of conv. If there are new messages appended since the last inference, it will perform Prefill calculation before starting Decode.