Conduit
AI Infrastructure Framework
Open-source framework for tuning, deploying, and building applications with open-source models. Compute-provider agnostic. Type-safe by default.
Overview
Conduit abstracts AI infrastructure into composable, type-safe building blocks. Define your models, schemas, and compute requirements in Python. Conduit handles provisioning, deployment, health management, and orchestration.
from conduit import ModelConfig, ComputeProvider
from conduit.runtime import LMLiteBlock
block = LMLiteBlock(
models=[
ModelConfig(
"Qwen/Qwen3-4B",
max_model_len=1400,
max_model_concurrency=50,
model_batch_execute_timeout_ms=1000,
)
],
compute_provider=ComputeProvider.RUNPOD,
gpu=GPUS.NVIDIA_L4,
replicas=2,
)
Architecture
Runtime Blocks
Conduit uses a block-based architecture for composable AI pipelines:
| Block | Function |
|---|---|
| LMLiteBlock | Inference runtime with streaming + batching |
| HttpGetBlock | HTTP request handling |
| FileSystemWriteBlock | File system operations |
| Sqlite3Block | Database operations |
Blocks are chainable. Each conforms to Conduit's communication spec.
Resource Management
Conduit automatically calculates infrastructure requirements:
Developer defines: models, replicas, constraints
Conduit calculates: VRAM requirements, GPU selection, cost
Conduit validates: resource compatibility, fit constraints
Conduit provisions: infrastructure on target provider
Add as many models as needed—Conduit throws an error if resources don't fit.
State Management
- Deployment state: What's running, where, with what configuration
- Health monitoring: Automatic failure detection
- Garbage collection:
LMLiteBlock.gc()cleans stale instances after config updates - Readiness gates:
.readyflag prevents traffic to unready nodes
Key Innovations
LM Lite Runtime
Full specification →Multi-model batching engine integrated into Conduit.
Capabilities:
- Efficient batch processing with configurable timeouts
- Round-robin load balancing across replicas
- Health-check readiness gates
- GPU-aware execution routing
- Multi-model deployments on single GPUs
Configuration:
ModelConfig(
model_id, # HuggingFace model ID
max_model_len=1400, # Max tokens per request
max_model_concurrency=50, # Max concurrent requests per batch
model_batch_execute_timeout_ms=1000, # Batch timeout
)
Model Data Language (MDL)
Full specification →Type-safe LLM communication integrated into Conduit.
Capabilities:
- Python dataclass schemas for input/output
- Automatic compilation to model-safe type hints
- Payload validation on request and response
- Guaranteed structured outputs
Usage:
from dataclasses import dataclass
from typing import List
@dataclass
class Input:
name: str
context: str
@dataclass
class Output:
response: str
confidence: float
tags: List[str]
result = block(
model_id="Qwen/Qwen3-4B",
input=Input(name="query", context="..."),
output=Output,
)
# result is guaranteed valid Output instance
Compute Abstraction
Conduit treats all compute providers as interchangeable container endpoints.
compute_provider=ComputeProvider.RUNPOD # or AWS, GCP, LOCAL, etc.
Provider agnostic design:
- Same code runs on any supported provider
- Switch providers by changing config, not application code
- Mix providers in the same pipeline
Optimized for Covenant Secure Compute Cloud (2026):
- Native MEP integration for encrypted inference
- End-to-end sovereignty guarantees
- Cryptographic attestation of execution
Lifecycle Management
Initialization
Garbage Collection
Readiness Check
while not block.ready:
time.sleep(5)
# Safe to send traffic
Cleanup
API Compatibility
Conduit supports OpenAI-style message format:
result = block(
model_id="Qwen/Qwen3-4B",
messages=[{\"role\": \"user\", \"content\": \"Hello\"}],
system_message="You are a helpful assistant",
)
Or typed MDL format:
result = block(
model_id="Qwen/Qwen3-4B",
input=TypedInput(...),
output=TypedOutput,
)
Installation
Requirements: Python 3.12.6+
Roadmap
- [ ]vLLM runtime support (
VllmBlock) - [ ]Local compute provider for on-device execution
- [ ]AWS (EKS / ECS / EC2)
- [ ]GCP (GKE / Vertex / Compute Engine)
- [ ]Azure (AKS / Azure ML)
- [ ]Lambda Labs, Modal, CoreWeave
- [ ]Covenant Secure Compute Cloud integration