Deployment Patterns: Cloud, On-Premise and Hybrid AI

What this means

Cloud deployment means using managed AI infrastructure provided by a hyperscaler or model provider: AWS Bedrock, Azure OpenAI Service, Google Vertex AI, or direct provider APIs (Anthropic, OpenAI). The provider manages hardware, model hosting, scaling, and availability. Your organisation sends API requests; the provider processes them and returns responses.

On-premise deployment means running model inference on hardware you own or lease — either in your physical data centre or in a dedicated private cloud environment. You run the model, manage the infrastructure, and control where data is processed. This is enabled by open-weight models (Llama 3, Mistral, Qwen) and purpose-built inference hardware (NVIDIA H100, A100 GPUs) or inference optimised hardware.

Hybrid deployment combines both: cloud APIs for workloads where data classification permits and latency is acceptable; on-premise or private cloud for workloads with stricter requirements. An orchestration or middleware layer routes requests to the appropriate environment based on data classification, latency targets, or cost rules.

Why it matters for business

Deployment pattern is not a purely technical decision — it has direct implications for regulatory compliance, cost predictability, and capability access. Australian organisations in healthcare, financial services, government, and other regulated sectors face obligations under the Privacy Act 1988 and Australian Privacy Principles that constrain where personal information can be processed.

At the same time, running large models on-premise requires significant capital investment in GPU infrastructure and ongoing engineering effort to maintain. For most organisations, the cost and operational burden of full on-premise AI cannot be justified — particularly when cloud providers now offer Australian region endpoints for the most commonly used services.

How it works technically

Cloud Deployment Architecture: Requests flow from applications through a middleware or API gateway layer to the cloud provider's model endpoint. Authentication uses provider-specific IAM mechanisms. Data is encrypted in transit using TLS. For organisations with data residency requirements, region selection in the API configuration determines where processing occurs. AWS Sydney (ap-southeast-2), Azure Australia East, and Google Cloud Sydney regions host AI services that process data within Australia.

On-Premise Deployment Architecture: An inference server (such as Ollama, vLLM, or NVIDIA NIM) hosts the model on your hardware. Applications connect to this server's API — typically exposing an OpenAI-compatible endpoint — in the same way they would call a cloud API. The middleware layer is identical; only the endpoint changes. Embedding generation, vector stores, and retrieval infrastructure also run on your own hardware.

Key considerations for on-premise deployment include: GPU memory requirements (Llama 3 70B requires approximately 140GB VRAM for full precision; quantised models reduce this significantly), inference throughput, cooling and power requirements, and model update cadence.

Hybrid Architecture: A model routing or middleware layer evaluates each request against classification rules. Requests containing personal information, commercially sensitive data, or data with contractual restrictions route to the on-premise or private cloud endpoint. General-purpose, non-sensitive requests route to cloud APIs. This approach captures the capability advantage of frontier cloud models while meeting data handling obligations for sensitive workloads.

Data classification must be implemented reliably for hybrid routing to work correctly. This requires either structured metadata from the application layer (labelling which requests contain sensitive data) or a classification model that evaluates requests before routing them.

Practical implementation considerations

Most Australian mid-market organisations do not have the engineering capacity or infrastructure investment to run large on-premise models for general-purpose AI. The practical starting point is cloud deployment, with on-premise infrastructure reserved for specific high-sensitivity use cases where regulatory or contractual requirements make cloud deployment untenable.

When using cloud AI services in Australia, verify — not assume — that the services you use process data in Australian regions by default. Some AI services default to US or EU regions even when Australian region options exist. Configuration of region selection and data residency should be explicitly documented and audited.

For organisations subject to APRA CPS 234 (financial services) or equivalent sector obligations, AI infrastructure and data processing arrangements should be included in information security risk assessments and third-party risk management frameworks.

Edison AI's AI implementation team works with Australian organisations to assess which workloads require on-premise processing and which can safely use cloud APIs — mapping deployment pattern decisions to specific data classification and regulatory requirements rather than applying a blanket policy.

Open-weight models deployed on-premise are evolving rapidly. Capability gaps between open-weight and frontier models have closed significantly in the past 18 months, making on-premise deployment more feasible for a wider range of use cases than it was previously.

Common mistakes

• Defaulting to full cloud without assessing data obligations: Many organisations assume cloud AI is compliant by default. It is not. Data processing location, encryption configuration, and provider data handling agreements must be explicitly verified.
• Full on-premise for cost reasons without adequate engineering investment: On-premise AI infrastructure requires ongoing engineering effort. Under-resourced on-premise deployments often run stale models on degraded hardware — delivering worse capability than cloud at higher cost.
• Not documenting data flows: Hybrid architectures where data flows between on-premise and cloud environments create complex data processing maps. These must be documented for privacy compliance and audit purposes.
• Ignoring latency implications: On-premise inference over a corporate network may have lower latency than cloud for some workloads but higher latency for others depending on hardware and network topology. Test before committing.
• Lock-in through tight cloud provider integration: Deep integration with a single cloud provider's proprietary AI services makes future migration expensive. Design your middleware layer to abstract provider specifics where feasible.

What leaders should do next

1. Classify your AI use cases by data sensitivity and map each to the most appropriate deployment pattern.
2. Verify region and data residency configuration for any cloud AI services currently in use — do not assume defaults are compliant.
3. For use cases involving sensitive personal or commercial data, assess whether Australian region cloud endpoints, private cloud, or on-premise deployment is required.
4. Document AI data flows as part of your privacy compliance obligations and include them in information security risk assessments.

Edison AI builds the AI implementation layer that connects your existing tools, data and agents into one operating system.