What LLM Systems Teach Healthcare IT About Architecture

Introduction

LLMs are not just larger models.

They force architectural decisions that healthcare IT has historically deferred.

Healthcare systems optimized for:

• Batch workloads
• Predictable traffic
• Latency tolerance

LLMs demand:

• Dynamic scheduling
• State-aware routing
• Memory optimization
• Real-time guarantees

That tension is revealing.

Main Content

Batch vs Continuous Work

Healthcare analytics is largely batch-oriented.

Claims arrive.
Models run overnight.
Reports are generated.

LLMs operate continuously.

flowchart LR
A[User Prompt] —> B[Scheduler]
B —> C[GPU Decode]
C —> B
C —> D[Stream Output]

This resembles streaming analytics more than traditional model serving.

If we embed LLMs into documentation workflows or appeals drafting, batch infrastructure will not suffice.

Memory Is the Bottleneck

In decode, performance is often memory-bandwidth-bound.

Healthcare IT often assumes compute is the constraint.

LLMs invert that assumption.

This mirrors large-scale attribution pipelines where I/O and memory dominate runtime.

Design for memory bandwidth and state persistence, not just FLOPs.

State Becomes a First-Class Primitive

Traditional inference is stateless.

LLMs are stateful per conversation.

That mirrors healthcare’s shift from episodic encounters to longitudinal care management.

State must be:

• Located
• Routed consistently
• Persisted intentionally
• Evicted deliberately

Stateless microservices are not enough.

Architectural Implications for Healthcare IT

LLMs are being embedded into:

• Documentation workflows
• Appeals drafting
• Prior authorization summarization
• Quality measure explanation
• Risk adjustment review

If infrastructure is naïve:

• Latency spikes
• Costs increase
• Engineers blame the model

The real issue is architecture.

Conclusion

LLM inference is not interesting because it is expensive.

It is interesting because it forces systems thinking.

Healthcare IT is entering an era where:

• Scheduling matters
• Routing matters
• Memory matters
• State matters

Not just modeling.
Not just analytics.
Architecture.