# Wen Feng Law — /llms-full.txt # Full plain-text article archive for AI ingestion # Generated: 2026-10-21T18:42:24Z ## Moving an Azure VM Between Virtual Networks: The Delete-and-Recreate Constraint URL: https://wenfeng.my/moving-azure-vm-between-virtual-networks-delete-recreate-constraint/ Author: admin Published: 2026-10-18 23:30:50 Modified: 2026-10-18 23:30:50 When the destination is a different virtual network, Azure VM relocation has one documented path. The Virtual Network overview states: to move a virtual machine from one virtual network to another, you must delete and recreate the virtual machine in the new virtual network. The virtual machine's disks can be retained for use in the new virtual machine (Azure Virtual Network overview). That sentence is the whole platform contract for VNet-to-VNet moves; everything else the operator needs to reason about is inference.What the documentation explicitly guaranteesThe Virtual Network overview describes Azure Virtual Network as the fundamental building block for your private network in Azure, delivering the scale, availability, and isolation benefits of Azure infrastructure while keeping the familiar networking concepts used in traditional datacenters (Azure Virtual Network overview). Resources listed as deployable inside a virtual network include virtual machines, App Service Environments, Azure Kubernetes Service, and Virtual Machine Scale Sets.The move constraint and the disk-retention exception are the only guarantees the documentation makes. The silence on every other detail is itself an operational signal: if something is not the disk, the documented path is to rebuild it.What the default VM set actually containsThe VM overview lists the default resources the platform provisions alongside a new VM: a virtual network, a NIC, a private IP address and possibly a public IP, an NSG, an OS disk, and in some cases a license for the OS. The same source notes that all new virtual machines have an operating system disk and a local disk (Azure VMs overview). A separate data disk is described as a best practice for separating data from the operating system, not as part of the default set.Managed disks are stored as page blobs, which the Storage introduction describes as a random IO storage object in Azure; Azure treats a managed disk as an abstraction over page blobs, blob containers, and Azure storage accounts (Azure Storage introduction). That abstraction is what lets the disk layer outlive the VM object: the disk is a storage primitive with its own identity.Inferences an engineer must reason about (design judgments)The items below are design judgments inferred from the move constraint and from how the default resources are described. They are not separately documented guarantees.Because the VM is deleted, identities attached to that VM object must be re-established on the new VM before dependent services will accept its tokens.Because the NIC does not survive, any private IP allocation, any public IP, and any NSG associations are part of the deleted object and must be rebuilt for the new VM.Because Azure Hybrid Benefit license cost is described as tied to the VM's cores, the new VM inherits a fresh license posture rather than carrying the old one forward.Because data disks are a separate best-practice artifact rather than part of the default set, they are the documented survival path for application state.Planning shape, not a vendor procedureThe following sequence is a hypothetical planning shape, not a vendor-validated procedure:Inventory state outside the VM object: NSG rules, route tables, private endpoints, custom DNS, certificate bindings, and any IP allow-lists tied to the current addresses.Decide whether retained data disks are sufficient, or whether a fresh image with a known baseline is the safer path.Decide whether the new VM reuses the same managed identity, a new system-assigned identity, or a different user-assigned identity.Validate the recreated VM's network reachability, identity, and extension state before decommissioning the source.Topology discipline that reduces the need to moveThe cleanest answer to a VNet-move problem is to avoid needing one. The VNet itself carries no charge: there's no charge for using Azure Virtual Network, and standard charges apply only to the resources placed inside it (Azure Virtual Network overview). That pricing structure makes it cheap to over-provision VNet boundaries correctly at the start: place a workload in the network where its governance, peering, and routing requirements will continue to fit for its lifetime.The platform gives you one documented escape hatch when the original choice was wrong: rebuild the VM in the right VNet, keeping the disks. Everything else in the move must be reasoned about and rebuilt by the operator. --- ## Terraform Refresh-Only Mode: Reconciling State With Out-of-Band Changes URL: https://wenfeng.my/terraform-refresh-only-mode-out-of-band-state-reconciliation/ Author: admin Published: 2026-10-18 06:30:14 Modified: 2026-10-18 06:30:14 The reader situationUse case: post-incident reconciliation. An incident forces a security group rule to be widened. An asset tag is added manually so the on-call team can locate a stray resource. An AMI is swapped on a live instance to clear a vulnerability. In each case the remote object no longer matches what Terraform's state records, and a normal terraform plan proposes to undo the change. Engineers facing this need a way to tell Terraform: accept the new reality, refresh the records, and propose nothing. That is refresh-only mode.What refresh-only mode actually doesRefresh-only mode creates a plan whose goal is only to update Terraform state and any root module output values to match changes made to remote objects outside of Terraform (terraform plan command). In practice Terraform reads the current remote attributes, writes them into the state file, and the resulting plan reports no resource actions.Activation, exclusivity, and the apply pathRefresh-only is activated with -refresh-only on either terraform plan or terraform apply; both commands accept the same planning modes when no saved plan is supplied (terraform plan command, terraform apply command). The non-default planning modes are mutually exclusive: activating refresh-only disables the normal planning mode, and you cannot use more than one alternative mode at the same time (terraform plan command). The option is available only in Terraform v0.15.4 and later (terraform plan command).When terraform apply receives a saved plan file, it honors the decisions in that plan and ignores additional planning modes (terraform apply command). For reconciliation work, the canonical pattern is terraform plan -refresh-only -out=tfplan, then terraform show tfplan to confirm which resource attributes are about to be absorbed into state, then terraform apply tfplan after review. This same two-phase discipline is covered in Speculative vs. Saved Terraform Plans: Making CI Approval Match Execution — for refresh-only, the inspection is a state-scope review rather than a configuration-diff review.The -refresh=false trapOne flag interaction is worth knowing before you reach for the command. You cannot combine -refresh=false with -refresh-only, because disabling the refresh step would effectively disable the entirety of the refresh-only planning operation (terraform plan command). The refresh is the whole job; skipping it leaves the mode with nothing to do, so the combination is rejected.State locking still appliesRefresh-only writes state, so it is covered by the same locking rules as any other state-writing operation: locking happens automatically, and if acquiring the lock fails Terraform does not continue (State Locking). Disabling the lock with -lock=false for a refresh-only run is the same anti-pattern as for any other write-state command, because concurrent writers can corrupt state, and the state-locking page discourages that flag explicitly (State Locking).Operator workflow, not CI gateDesign judgment, not source fact: refresh-only is an operator tool, not a CI step. A pipeline that runs refresh-only before every apply adds work without changing the outcome, because normal terraform plan already refreshes state by default. Reach for refresh-only specifically when a real out-of-band change has produced a divergence and you want to absorb it on purpose, not as routine pre-apply hygiene.Design judgment: refresh-only accepts whatever the remote says. If both the configuration and the remote have drifted (someone edited main.tf and someone edited the resource), a refresh-only run will not surface the configuration drift. A normal plan is still required to detect that. Use refresh-only only when the configuration is the trusted baseline and only the remote has moved.Design judgment: because the refresh walks every tracked resource, -refresh-only against a very large configuration with a slow provider API can be slow. Narrowing with -target reduces the scope, but -target is documented as for exceptional circumstances only, with the recommended direction being to split large configurations into smaller, independently applicable ones (terraform plan command).SummaryRefresh-only mode reconciles Terraform state with out-of-band remote changes without proposing any infrastructure changes of its own. Treat it as a narrow reconciliation tool for incident-driven divergences, respect the -refresh=false incompatibility, let state locking do its job, and use it only when the configuration itself is the trusted baseline. --- ## PostgreSQL MVCC vs DDL: When Migrations Silently Break Long-Running Snapshots URL: https://wenfeng.my/postgresql-mvcc-ddl-migration-snapshot-boundary/ Author: admin Published: 2026-10-16 23:30:54 Modified: 2026-10-16 23:30:54 You run a 90-minute analytics job inside a Serializable transaction. A teammate commits a TRUNCATE on a staging table during a scheduled schema migration. Both operations succeed. Your job's final report is silently wrong: the table appears empty because the snapshot you hold predates the DDL commit. The reason this can happen at all is that the transaction had not yet touched that table when the DDL started, so it never held the ACCESS SHARE lock that would normally block the migration.This is not a tool bug. It is an explicitly documented boundary in PostgreSQL's concurrency model that data engineers can miss.What the Documentation Calls OutThe PostgreSQL concurrency-control documentation flags this directly. In its MVCC Caveats section, it states that some DDL commands — currently only TRUNCATE and the table-rewriting forms of ALTER TABLE — are not MVCC-safe (caveats). After such a command commits, the table will appear empty to concurrent transactions using a snapshot taken before the DDL commit.The same caveat describes the protection mechanism: any transaction that previously accessed the table holds at least an ACCESS SHARE table lock, which blocks the DDL. The window opens only when a concurrent transaction has not yet touched the table when the DDL starts (caveats).Why Snapshots Make This WorseThe transaction-isolation page describes how a SELECT in Read Committed sees a fresh snapshot per statement, while Repeatable Read and Serializable fix the snapshot to the start of the first non-transaction-control statement (isolation). That long-lived snapshot is exactly what makes a long analytics query safe from concurrent writes — and exactly what makes it vulnerable to a DDL commit that happened after the snapshot.For read-only Serializable work, SERIALIZABLE READ ONLY DEFERRABLE will block until it can acquire a snapshot guaranteed to be free of serialization anomalies before reading any data — the only documented case where Serializable transactions block but Repeatable Read does not (isolation). For mixed read-write transactions, no such auto-blocking exists.Planning vs Applying the State ChangeThe hazard grows when migration tools separate generate plan from apply. A planner that checks for blockers, returns, and then commits gives a long-running Serializable job a window to begin work after the check but before the apply. Design judgment: treat the schema migration as a state change with snapshot semantics, and ensure any in-flight transaction holding an older snapshot of the target table is visible to the planner before the apply step runs.The pg_locks system view exposes the currently outstanding locks held by the database server — the documented entry point for inspecting lock-manager state across table, row, page, advisory, and predicate locks (locking). Snapshot visibility, by contrast, is governed by MVCC rules that operate independently of the lock manager. Migration tooling which only inspects pg_locks is checking one of the relevant signals — not the snapshot-visibility boundary described in the caveats.Coordination Without Locking the WorkflowThe locking page documents advisory locks as application-defined locks the system does not enforce — useful when you want something faster than a flag table, that avoids table bloat, and that is automatically cleaned up by the server at the end of the session (locking). The system will not enforce the lock; the application must coordinate correctly.Hypothetical example: a migration orchestrator acquires a session-level advisory lock at planning time, before any DDL, to signal “a migration is in progress for this table.” Long-running jobs that opt in to check the same advisory lock could wait or abort. This is a coordination pattern, not a workflow restriction — it cannot stop a job that does not opt in, and session-level advisory locks do not honour transaction semantics.Three Mechanisms, Not OneIt is easy to conflate three distinct mechanisms the documentation describes:DDL non-MVCC-safety — about snapshot visibility across DDL, not about row locks (caveats).Serializable predicate locking — uses SIReadLock entries in pg_locks to detect anomalies; these locks do not cause blocking and so cannot be part of a deadlock (isolation).Row-level FOR UPDATE locking — can cause a disk write when a row is marked locked (locking).Label one mechanism's behaviour as another's and you misdiagnose production incidents.What the Right Posture Looks LikeFor data engineering teams, the practical posture is to separate the planning question from the apply question. The docs do not promise that any of this is automatic — they describe the boundary so you can design around it. If a migration must run during analytic hours, build the coordination in explicitly rather than rely on the lock manager to surface a hazard it cannot see. --- ## Reading Prompt Caching and Latency Together Across Microsoft Foundry Deployment Types URL: https://wenfeng.my/reading-prompt-caching-latency-microsoft-foundry-deployment-types/ Author: admin Published: 2026-10-15 23:30:40 Modified: 2026-10-15 23:30:40 When TTLT drops on a Microsoft Foundry deployment, three explanations often arrive at once, and reading any one alone tells the wrong story. In many teams, dashboards pin one chart, one metric, or one deployment type — and attribute cache work to throughput savings (or vice versa), or load-shedding to a model upgrade. This article maps the signals three Microsoft documentation surfaces carry, and the correlation rules none of them call out.The three surfaces are deployment types, latency, and prompt caching.Axis one: deployment type changes what the other axes exposeFoundry offers three billing families — Standard, Provisioned, and Batch — whose sub-tiers change which metrics are worth plotting. Global Standard has the lowest price and broadest region coverage. Provisioned types reserve capacity for more consistent latency than Global Standard. Batch gives a 50% discount for a 24-hour async turnaround with no real-time SLA. Developer is fine-tune evaluation only, with a 24-hour lifetime and no SLA. Data Zone tiers restrict processing to a Microsoft-defined US/EU/APAC zone.Dashboards built on Global Standard are not directly comparable to one built on Provisioned. Build per account type.Axis two: the latency formula is half the answerThe Azure OpenAI latency article commits to a single formula — total response time equals time to first token plus time between tokens multiplied by tokens generated. TTLT scales with output tokens, so always pair a latency metric with a token-count metric before calling something a regression. The same five-second TTLT across a 2,000-token generation and a 50-token generation is two different signals.The metrics that matter are five of the six documented latency signals: Time to Last Byte (end-to-end non-streaming), Time to Response (streaming first-byte), Time Between Tokens, Processed Prompt Tokens, and Generated Completion Tokens. Skip Normalized Time to First Byte for absolute diagnosis — the source documents it as a cross-prompt comparison tool.Axis three: prompt caching only counts when the prefix qualifiesCache eligibility is narrow: a request must contain at least 1,024 tokens, and the first 1,024 tokens must be identical across requests. A single character difference produces a cached_tokens value of zero. Hits surface as cached_tokens inside prompt_tokens_details.On GPT-5.6 and later model families, Provisioned managed (PTU-M) deployments continue serving cached content, but neither breakpoints nor cache_write_tokens are reachable on this path. Standard pay-as-you-go exposes cache_write_tokens alongside cached_tokens. A dashboard that mixes both account types will misread the cost and pressure story.Three correlation rules — design judgment, not source factTTLT down, cached_tokens up. A real cache hit. Verify the saved TTLT is consistent with the cached prefix length and re-check recent prefixes against the 1,024-token identity rule before crediting the cache.TTLT down, cached_tokens flat, GeneratedTokens flat. Not caching — capacity pressure eased. On PTU accounts, watch Provisioned-managed Utilization V2. On pay-as-you-go, watch Azure OpenAI Requests for 429 throttling and concurrent volume.TTLT down, cached_tokens flat, GeneratedTokens rising under load. Cache-eligible traffic is missing the cache — prompts arrive but fail the 1,024-token identity rule, or sit below the minimum. The latency article warns that mixing workloads on the same endpoint hurts both latency and cache hit rate.What to instrument firstTime to Last Byte split by ModelDeploymentName, paired with Generated Completion Tokens on the same window.For streaming products: Time to Response plus Time Between Tokens and Processed Prompt Tokens.cached_tokens sampled from chat completions responses.On PTU accounts: Provisioned-managed Utilization V2. On Standard pay-as-you-go: cache_write_tokens alongside cached_tokens.Which deployment type earns the cacheCache mechanics are largely stable across Global Standard, Data Zone Standard, and Regional Provisioned; what varies is the discount on cache reads (Provisioned managed reads can be billed at up to 100% discount on input tokens versus a smaller Standard discount) and the breakpoint or write-token exposure on PTU-M. Data Zone tiers only become a real choice under residency constraints; the deployment types article points to the model availability matrix for region-by-region support.Rule 3 is the scenario worth acting on: TTLT falling, cached_tokens flat, GeneratedTokens rising under load signals cache-eligible traffic missing the cache — a workload worth migrating to a Provisioned tier, where cache reads carry a stronger discount and reserved capacity removes the capacity-easing explanation from Rule 2. No chart in any of the three surfaces names that combination. --- ## Speculative vs. Saved Terraform Plans: Making CI Approval Match Execution URL: https://wenfeng.my/terraform-speculative-vs-saved-plans-two-phase-cicd/ Author: admin Published: 2026-10-14 06:30:53 Modified: 2026-10-14 06:30:53 A green check on a CI terraform plan step is not a green light for terraform apply. A useful pipeline policy should distinguish whether planning succeeded, whether it found changes, and whether those particular changes were approved. HashiCorp exposes three options that, used together, draw a defensible boundary between review and execution: -detailed-exitcode, -out=FILE, and -lock-timeout. Speculative plans are not authorization to apply Running terraform plan without -out=FILE produces a speculative plan: a description of the plan's effect, with no intent to actually apply it terraform plan documentation. Speculative plans exist for code-review workflows where developers verify the effect of a change before submitting it. The same documentation warns, plainly, that intervening changes to the target system can make the final effect of a configuration change different from what an earlier speculative plan indicated, and that the final non-speculative plan should always be re-checked before applying terraform plan documentation. A speculative plan is a snapshot at one moment, not a contract for later execution. Saved plans shift the approval boundary Passing -out=FILE writes a non-speculative plan file that terraform apply will execute as written terraform plan documentation. When a saved plan is passed to apply, the command runs without prompting, and -auto-approve is ignored, because passing the plan file is itself the approval terraform apply documentation. The apply command also rejects any additional planning modes or options in that mode, because the plan file already encodes the final decisions terraform apply documentation. Creating the file and authorizing its execution are separate decisions for the pipeline owner; passing it to apply is the execution boundary. When terraform apply runs without a saved plan file, it automatically creates a new execution plan as if terraform plan had been run, then prompts for approval before applying terraform apply documentation. This is the mechanism that makes apply-time re-planning possible without an explicit intermediate plan step. (Design framing; the regeneration behavior is sourced.) What -detailed-exitcode changes for a CI gate -detailed-exitcode returns three values: 0 for empty diff, 1 for error, and 2 for non-empty diff terraform plan documentation. This is the flag that lets a pipeline answer that question without scraping human-readable output. Hypothetical example: a policy-enforced pipeline can branch on the exit code — 0 for no changes, 2 for changes present — and require an authorized reviewer to comment before the apply job can start. The approval is then decoupled from the plan job's runtime. (Design judgment, not a source claim.) State locking in a two-phase pipeline When the selected backend supports locking, Terraform automatically locks state for operations that could write it. Routine acquisition can be silent; a longer wait can produce a status message state locking documentation. The upstream guidance is explicit: -lock=false is documented as something the maintainers do not recommend for most commands, because concurrent writers can corrupt state state locking documentation. For contended CI environments where another job may briefly hold the lock, the right knob is -lock-timeout=DURATION, which retries acquisition for a specified window before erroring terraform plan documentation. The same flag is also supported on terraform apply terraform apply documentation. A hypothetical saved-plan approval workflow The following is a design proposal, not a claim about a particular team's deployment. It uses the saved-plan workflow HashiCorp describes for automation, while keeping the approval tied to the plan that will be executed. Create the plan — run terraform plan -out=tfplan -detailed-exitcode -lock-timeout=30s. The 30-second lock wait is an example policy choice, not a vendor recommendation. Handle exit code 1 as failure and 2 as a successful plan containing changes, rather than treating every nonzero exit code identically. Review and authorize — have an authorized reviewer examine the proposed changes and approve use of that saved plan. The plan documentation warns that saved plan files can contain sensitive values in cleartext; treat the file as a sensitive artifact rather than a public build log. Apply the approved file — pass that same file to terraform apply tfplan. The command does not ask again for approval when a saved plan is supplied, so the pipeline must enforce its approval policy before invoking it. Do not pass new planning options at this step; the saved plan already contains the planning decisions. These flag and saved-file behaviors are documented in the plan reference and the apply reference. If the pipeline instead runs apply without a saved file, Terraform generates another plan and normally asks for approval. Approval of an earlier speculative result should not be represented as approval of that newly generated result. A team choosing this alternative must design its approval interaction accordingly. The practical design question is which planning result is reviewed and how that result reaches execution. Saved plans and automatic planning offer different approval boundaries; neither removes the need to handle errors and backend capabilities. A green planning check alone does not answer those policy questions. --- ## Kubernetes Rollout Budgets Are Not a Capacity Plan URL: https://wenfeng.my/kubernetes-rollout-budgets-capacity-plan/ Author: admin Published: 2026-10-11 06:30:47 Modified: 2026-10-11 06:30:47 A rolling update can stay within its availability settings and still put pressure on a cluster. The useful planning question is therefore broader than how many replacement Pods Kubernetes may start: what else will consume resources while those replacements become ready? The distinction matters when a service has a slow startup, a long shutdown, or little spare capacity. The following is a design walkthrough, not a report of a measured production incident. Separate availability from resource consumption The Kubernetes Deployment documentation defines maxUnavailable as the allowed unavailability during a rolling update and maxSurge as additional Pods above the desired count. It also warns that terminating Pods can consume resources beyond replicas + maxSurge until their termination grace period expires. A surge setting is consequently insufficient as a physical resource budget. Consider a hypothetical service whose old workers drain long-running requests while replacements initialize. Sketch three groups on a planning sheet: workers serving traffic, replacements preparing to serve, and old workers shutting down. For each group, record the resource requests, observed consumption, and expected duration. The purpose is to expose overlap, not to derive a universal safe replica count. This gives an operator two separate questions to answer. Is enough usable serving capacity available? Can the infrastructure accommodate the temporary overlap? A favorable answer to either question does not establish the other. Test both under conditions representative of the application. Give each probe a specific job The probe concepts documentation distinguishes readiness, liveness, and startup checks. Failed readiness checks remove a Pod from matching Service endpoints; failed liveness checks can restart the container. A configured startup probe prevents liveness and readiness probes from running until it succeeds. Those outcomes are different operational decisions. For the hypothetical service, write down what each endpoint actually demonstrates. Can it accept useful work? Is the process stuck in a state that restarting can repair? Is initialization still legitimately in progress? Avoid using an undifferentiated dependency check to answer all three questions without examining its failure consequences. As a design exercise, trace a shared dependency slowdown. If every instance fails the same check simultaneously, what happens to traffic and restart activity? This exercise does not prescribe a universal probe implementation. It forces the team to examine whether the configured response helps the particular failure or adds more work during recovery. Measure the transition, not just the steady state The probe configuration guide describes allowing sufficient startup time with failureThreshold and periodSeconds. Use those documented controls with measurements of the actual initialization path; a copied value from another service is only a starting hypothesis. In a controlled test environment, observe a replacement from scheduling through initialization, readiness, useful work, and eventual shutdown. Repeat under a representative workload. Record the sequence alongside resource consumption and application errors. A green readiness check is evidence about the configured check; application measurements are still needed to assess whether the rollout meets the service's goals. Before a production change, agree on explicit pause criteria, who evaluates them, and how to inspect a stalled rollout. Separate configuration mistakes from capacity shortages and application regressions. Changing several controls at once makes that diagnosis harder. A practical review checklist Document availability settings and temporary resource overlap separately.Explain the intended outcome of each probe failure.Measure startup and shutdown under a representative load.Check whether a shared dependency failure amplifies restarts.Define pause criteria before starting the rollout. These checks turn rollout settings into testable assumptions. They do not promise a particular latency improvement or eliminate failure. Their value is making the application's transition behavior visible before availability controls are mistaken for a complete capacity plan. --- ## Why a WordPress Timeout Must Not Trigger a Second Publication URL: https://wenfeng.my/wordpress-timeout-no-second-publication/ Author: admin Published: 2026-10-10 02:28:09 Modified: 2026-10-10 02:28:09 The seam where duplicates are bornAutomated content pipelines look simple from the outside: a scheduler triggers, the script posts, the post appears. The interesting failures all live in the seam between the script and the publishing endpoint, where a network call can succeed silently, fail loudly, or do neither.When a small automated pipeline posts to WordPress over REST, the create call may time out. The publishing script then sees no response, no post ID, and no confirmation. Retrying POST /wp/v2/posts after a lost response can create a second post if the first request succeeded.The WordPress REST API reference exposes POST /wp/v2/posts to create a post and POST /wp/v2/posts/<id> to update a known post. The reference returns id, slug, status, date, and date_gmt, with date in site timezone and date_gmt in GMT. The API does not document a general idempotency-key parameter for creation, and matching slugs do not by themselves guarantee exactly-once publishing.What "safe retry" actually requiresThe Microsoft retry pattern guidance treats retries as appropriate only for transient failures, and warns that operations should be idempotent before retries are layered on. A retry strategy that fans out from multiple layers at once tends to multiply rather than absorb failures.The AWS Builders' Library on idempotent APIs goes further: a client request identifier can carry the same intent across retries, but identical parameters alone do not distinguish a retry from a new request. The server must associate the identifier with the effect. The cited WordPress posts reference does not promise this request-identifier contract; a custom plugin would need its own verified guarantees.For a publication pipeline, that means the safety has to be designed in the pipeline, not borrowed from the API.Separate draft creation from publicationThe concrete design is to keep two steps apart: build the content first, then publish by updating a known post.Persist an operation record locally before any write. The record holds the intended slot time, the content hash, the agent identity, and an initially empty remote_id. Then create the post as a draft to obtain the numeric ID. Only after the ID is confirmed should the script move to publication, using POST /wp/v2/posts/<id> with status set to publish.Reviews should bind to the exact visible content, not to a slot time alone. A scheduled slot owned by one writer removes the race where two agents each think they hold it.Failure walkthrough: a 07:30 MYT scheduled slot07:30 MYT (Malaysia Time, UTC+8) is 23:30 UTC the previous day; 14:30 MYT is 06:30 UTC the same day. The following timeline is an illustrative design example, not a reported incident.Before the 07:30 MYT publication slot, the pipeline records a draft-creation operation and calls POST /wp/v2/posts with status=draft. The HTTP request times out. The local record still has remote_id = null.Now inspect the remote state instead of creating again. Compare the intended slug, author and exact content against candidate drafts using an authenticated, fully paginated read. A verified unique match can establish the numeric ID for subsequent review. An empty or ambiguous lookup does not prove that the earlier request failed: processing may still be in flight, and matching rules can be incomplete. Keep the operation and its publication slot held for operator reconciliation. A client-side request identifier has no deduplication effect unless the server explicitly implements that contract.In this conservative design, the pipeline does not automatically issue a second POST /wp/v2/posts for the same content, and at no point does it attempt to publish by re-creating.Verification checklistOperation record is written before the first network call.Local remote_id is only filled from a confirmed create response or a verified re-read.Publication uses POST /wp/v2/posts/<id>, never a second create.Slot owner is recorded; concurrent agents for the same slot are rejected.Retry budget is bounded and coordinated across layers.A slot whose state cannot be inspected stays held for an operator.When you cannot prove safetyA local ledger is not an atomic transaction with WordPress and does not guarantee exactly-once publication. If the create response is lost and the re-read also fails, do not assume the post is absent. Hold the slot and surface it for manual reconciliation rather than letting a second create risk a duplicate. Fail closed when inspection is unavailable.Treat publication as an effect that once begun must not be repeated without proof it did not begin. The proof has to come from the remote state, not from a second create call.For architectural background, see the earlier multi-agent content pipeline walkthrough. The retry and reconciliation rules above are a separate design proposal, not a claim about how that earlier implementation behaves. --- ## Azure Cobalt 200 + Blackwell RTX: AI VM Cost-Perf Playbook (2026) URL: https://wenfeng.my/azure-cobalt-200-arm-vm-nc-rtx-pro-blackwell/ Author: admin Published: 2026-09-01 06:30:34 Modified: 2026-10-04 15:32:57 Azure Cobalt 200 Arm VMs and NC RTX PRO Blackwell VMs — New Compute Options for AI and Cost OptimizationAzure just gave us two very different but equally compelling reasons to rethink our VM choices. The Cobalt 200 Arm-based VMs are now in preview, delivering up to 50% better CPU performance than their Cobalt 100 predecessors while maintaining aggressive cost advantages. Meanwhile, the NCv6 series powered by NVIDIA RTX PRO 6000 Blackwell GPUs just hit general availability in West US 2 and Southeast Asia, bringing a new class of converged AI and visualization compute to Azure.If you're an Azure-first enterprise team in Southeast Asia — or anywhere — these two VM families represent a fork in the road: one path optimizes for cost and scale on CPU-bound workloads, the other unleashes GPU-accelerated AI inference and visual computing. Understanding when to use each, and how to migrate, will directly impact your cloud spend and your AI ambitions.Let me walk through both, with real numbers, real CLI commands, and practical guidance on making the switch.Azure Cobalt 200: The Next Generation of Arm at ScaleMicrosoft's custom Arm silicon has come a long way since the first Cobalt 100 instances shipped. The Cobalt 100 is now deployed across 32 Azure regions, and early adopters reported up to 45% better performance while using 35% fewer compute cores. Cobalt 200 builds on that momentum with a significant generational leap.What's Under the HoodThe Cobalt 200 SoC is built on Arm Neoverse V3 compute subsystems, fabricated on TSMC's 3nm (N3P) process with a modern chiplet architecture. The cache hierarchy is where it gets interesting:3 MB of L2 cache per core — this is substantial and directly benefits database and caching workloads192 MB of system-level L3 cache — a massive shared pool that reduces memory latency for scale-out applicationsMemory encryption enabled by default via a custom memory controller, with negligible performance impactThe result? Real-world performance gains that go well beyond synthetic benchmarks: Workload Type Performance Improvement vs Cobalt 100 Cloud databases Up to 135% better Caching workloads Up to 80% better Communication encryption Up to 45% better Web serving Up to 40% better Those numbers are not theoretical. Microsoft's own Azure SQL Database and Dataverse teams are validating Cobalt 200 right now, with the Dataverse team reporting up to 60% better performance for their base workload.The New VM FamiliesCobalt 200 expands the portfolio across multiple optimized SKUs: Family vCPUs Mem:vCPU Local NVMe Best Fit Dpsv7 / Dpdsv7 (General Purpose) 1–128 4:1 Up to 7 TiB Web/app servers, small-medium databases, caches Dplsv7 / Dpldsv7 (Light General Purpose) 1–128 2:1 Up to 7 TiB Microservices, small databases, gaming servers Epsv7 / Epdsv7 (Memory Optimized) 1–128 8:1 Up to 7 TiB Large relational/NoSQL DBs, Redis, analytics Mpsv4 / Mpdsv4 (High Memory Optimized) 1–84 16:1 Up to 4.4 TiB In-memory databases, ERP systems, large-scale caching Lpsv5 (Storage Optimized) 1–128 8:1 Up to 23 TiB Data pre-processing, big data analytics, search engines Networking and storage throughput are equally impressive, with most families supporting up to 85 Gbps network bandwidth and 70 Gbps remote storage throughput — all backed by Azure Boost for distributed application performance.Cost Comparison: Cobalt 200 vs x86This is where Arm VMs really shine for enterprise teams watching their cloud bills. Historically, Azure's Arm-based instances have been priced 20–40% lower than comparable Intel or AMD instances. While specific Cobalt 200 pricing will settle as general availability approaches, here's the illustrative historical pattern from Cobalt 100 — not current list rates — to give a sense of the relative savings band: Configuration Intel Dv5 (Illustrative) Cobalt 100 Dpsv6 (Illustrative) Savings 8 vCPUs / 32 GB ~$0.38/hr ~$0.28/hr ~26% 16 vCPUs / 64 GB ~$0.77/hr ~$0.56/hr ~27% 32 vCPUs / 128 GB ~$1.54/hr ~$1.12/hr ~27% The figures above are illustrative historical Cobalt 100 / Intel Dv5 numbers from prior comparison articles, not verified against the current Azure Pricing Calculator as of August 2026. Always confirm live pricing via the Azure Pricing Calculator before committing to a migration plan.For workloads that run 24/7 at scale, this compounds quickly. A fleet of 50 general-purpose web servers saving 25–30% on compute adds up to meaningful budget relief — money you can redirect toward GPU instances for AI workloads.Migration: Easier Than You ThinkIf you're already on Cobalt 100, migration to Cobalt 200 is a non-event. It's the same Arm64 architecture, same OS, same binaries. Microsoft describes it as "a change of hardware, not a change of platform."For teams migrating from x86 to Arm for the first time, the ecosystem has matured significantly:# Check if your current workload runs on Arm-compatible images az vm image list --publisher Canonical --offer 0001-com-ubuntu-server-jammy \ --sku 22_04-lts-arm64 --all --output table # Create a Cobalt 200 VM (once you have preview access) az vm create \ --resource-group myResourceGroup \ --name cobalt200-web-01 \ --image Canonical:ubuntu-24_04-lts:server:latest \ --size Standard_D16psv7 \ --admin-username azureuser \ --ssh-key-values ~/.ssh/id_rsa.pub \ --generate-ssh-keys # Verify Arm64 architecture az vm run-command invoke \ --resource-group myResourceGroup \ --name cobalt200-web-01 \ --command-id RunShellScript \ --scripts "uname -m && cat /proc/cpuinfo | head -20" Key migration considerations:Container workloads are the easiest win. If you're running Docker or Kubernetes, most base images (Ubuntu, Alpine, Debian) have mature Arm64 variants. Rebuild your images with --platform linux/arm64 and you're done.Language runtimes are well-supported. Python, Java, .NET, Node.js, Go, and Rust all have production-ready Arm64 builds. Over 95% of the Ubuntu archive is built and tested for Arm.Database engines work out of the box. PostgreSQL, MySQL, MongoDB, Redis — all have official Arm64 binaries. Check for any x86-specific extensions or native modules in your application dependencies.Test before you commit. Run your CI/CD pipeline against Arm64 targets. Tools like Docker's buildx make multi-architecture builds straightforward:# Build and push multi-arch image docker buildx build --platform linux/amd64,linux/arm64 \ -t myregistry.azurecr.io/myapp:latest \ --push . The Linux kernel supports transparent page size migration, and Ubuntu on Cobalt 200 includes Kernel Livepatch from Ubuntu 26.04 LTS onward, so you can patch without downtime.Azure NCv6: NVIDIA RTX PRO 6000 Blackwell for AI and VisualizationIf Cobalt 200 is about cost-efficient CPU scale, the NCv6 series is about raw GPU power. Microsoft positions these VMs as a "dual engine" — capable of handling both industrial visualization (digital twins, rendering) and AI inference on a single platform.The HardwareAt the heart of the NCv6 is the NVIDIA RTX PRO 6000 Blackwell Server Edition GPU: Specification Detail GPU Architecture NVIDIA Blackwell VRAM 96 GB GDDR7 Memory Bandwidth 1,597 GB/s CUDA Cores 24,064 Tensor Cores 5th Generation (FP4, FP8, INT8) AI Performance 4,000 AI TOPS (FP4 w/ sparsity) Single Precision 120 TFLOPS CPU Intel Xeon Granite Rapids (up to 4.2 GHz turbo) System Memory 132 GiB – 1,032 GiB DDR5 (GP); 72 GiB – 516 GiB (CO), 6,400 MT/s Networking Up to 180 Gbps Azure Accelerated Networking vCPUs 36 – 288 (GP); 24 – 288 (CO) The 96 GB of GDDR7 memory is the headline feature. That's enough VRAM to run 30B-class models in FP16 or 70B-class models in 4-bit/8-bit quantized format — without needing to split across multiple GPUs. For teams in Southeast Asia building RAG pipelines, deploying small-to-medium LLMs, or running agentic AI workloads, this is a very practical amount of memory.Note: 1,597 GB/s is the Server Edition spec; NVIDIA's Workstation Edition lists 1,792 GB/s. Azure NCv6 uses the Server Edition.Multi-Instance GPU (MIG) support means you can carve a single RTX PRO 6000 into smaller fractional GPUs. This is critical for cost optimization — you don't need to rent a full GPU for a small inference endpoint or a VDI session.When to Choose NCv6 over Cobalt 200This is the practical question. Here's a decision framework:Choose Cobalt 200 when:- Your workload is CPU-bound (web servers, APIs, microservices, databases)- You're running Linux-based scale-out services- Cost optimization is the primary driver- You need high memory density at low cost (Mpsv4 for in-memory workloads)- Your AI involvement is limited to data preprocessing or orchestration (not inference)Choose NCv6 with RTX PRO 6000 when:- You need GPU-accelerated AI inference (LLM serving, RAG, embeddings, fine-tuning)- You're running visualization workloads (digital twins, Omniverse, 3D rendering)- You need real-time video processing or encoding at scale- Your workload demands both GPU compute and high CPU throughput (the Granite Rapids CPU prevents bottlenecks)- You need MIG for multi-tenant GPU sharingThe hybrid architecture is where most enterprise teams will land:Client → Load Balancer ├── Cobalt 200 Dpsv7 (API Gateway, Web Tier, Auth) — cost-optimized ├── Cobalt 200 Epsv7 (Redis Cache, Session Store) — memory-optimized └── NCv6 Standard_NC144ds_xl_RTXPRO6000BSE_v6 (AI Inference, Embeddings, RAG) — GPU-accelerated This pattern gives you the best of both worlds: Arm-powered cost savings on the CPU-heavy tiers, and Blackwell GPU power where it matters for AI.Deploying an NCv6 Instance# List available NCv6 SKUs in Southeast Asia (now GA there as of mid-2026) az vm list-sizes --location southeastasia \ --query "[?contains(name, 'RTXPRO6000BSE')].{Name:name, vCPUs:numberOfCores, MemoryMB:memoryInMb}" \ --output table # Create an NCv6 GPU VM (Windows) az vm create \ --resource-group myResourceGroup \ --name ncv6-inference-01 \ --image MicrosoftWindowsServer:WindowsServer:2022-datacenter-gpu:latest \ --size Standard_NC144ds_xl_RTXPRO6000BSE_v6 \ --admin-username azureuser \ --generate-ssh-keys # For Linux GPU workloads az vm create \ --resource-group myResourceGroup \ --name ncv6-inference-linux-01 \ --image Canonical:ubuntu-24_04-lts:server:latest \ --size Standard_NC144ds_xl_RTXPRO6000BSE_v6 \ --admin-username azureuser \ --ssh-key-values ~/.ssh/id_rsa.pub \ --generate-ssh-keys # Install NVIDIA drivers (Linux) — NCv6 uses GRID/vGPU drivers, not the # standard CUDA driver path. Follow Microsoft's NCv6 driver guide: # https://learn.microsoft.com/azure/virtual-machines/linux/n-series-driver-setup # # NOTE: The apt-get command below is ILLUSTRATIVE ONLY. NCv6 officially ships # with NVIDIA GRID/vGPU driver v20.x (R595) as of Apr 23 2026, not the standard # CUDA driver. For production deployments, follow Microsoft's N-series driver # setup guide and use the GRID driver (vGPU) install path — not nvidia-driver-550. az vm run-command invoke \ --resource-group myResourceGroup \ --name ncv6-inference-linux-01 \ --command-id RunShellScript \ --scripts "sudo apt-get update && sudo apt-get install -y nvidia-driver-550 nvidia-utils-550" # Verify GPU availability az vm run-command invoke \ --resource-group myResourceGroup \ --name ncv6-inference-linux-01 \ --command-id RunShellScript \ --scripts "nvidia-smi" GPU Pricing ContextNCv6 instances will carry the premium you'd expect for Blackwell-class GPUs. As of August 2026, Azure lists the Standard_NC144ds_xl_RTXPRO6000BSE_v6 (single GPU, 144 vCPUs, 516 GiB) at $6.38/hr on-demand in West US 2 and $8.93/hr in Southeast Asia (Linux, on-demand). That puts the single-GPU effective rate higher than AWS g7e (~$3.36/hr/GPU on g7e.2xlarge) and GCP g4-standard-48 (~$4.50/hr/GPU), but Azure's regional availability in Southeast Asia and the included 96 GB of VRAM per GPU make the comparison more nuanced than the headline rate suggests. Azure Spot pricing on NCv6 is region-dependent and generally does not drop below ~$2/hr for this SKU.For teams comparing against older NCv3 or NVv5 series, the NCv6 offers a clear upgrade path. The MIG capability means you can start with fractional GPU slices (1/4 GPU on NC36ds or 1/2 GPU on NC72ds) for development and testing, then scale to full GPU allocation for production inference.Practical Migration Strategy: A Step-by-Step ApproachHere's how I'd approach adopting these new VM families in a real enterprise environment:Phase 1: Assess (Week 1–2)# Audit your current VM fleet az vm list --query "[].{Name:name, Size:hardwareProfile.vmSize, ResourceGroup:resourceGroup}" --output table # Identify Arm-eligible workloads (Linux, no x86-specific dependencies) az vm list --query "[?contains(storageProfile.imageReference.publisher, 'Canonical') || contains(storageProfile.imageReference.publisher, 'RedHat')].{Name:name, OS:storageProfile.imageReference.offer}" --output table Phase 2: Pilot (Week 3–4)- Deploy 2–3 non-critical workloads on Cobalt 200 Dpsv7 instances- Compare performance metrics (response time, throughput, CPU utilization) against current x86 VMs- For AI workloads: provision a single NCv6 instance and benchmark your inference pipelinePhase 3: Migrate (Month 2–3)- Move web/API tiers to Cobalt 200 (usually the easiest migration)- Migrate database tiers to Cobalt 200 memory-optimized instances (Epsv7/Mpsv4)- Stand up GPU inference endpoints on NCv6Phase 4: Optimize (Ongoing)- Right-size instances based on actual utilization data- Apply Reserved Instances or Savings Plans for predictable workloads- Use MIG on NCv6 to consolidate smaller inference workloads onto fewer GPUsCommon Pitfalls to Watch ForWindows workloads need x86 for now. Cobalt 200 is Arm64. If your application stack requires Windows, you'll need to stay on Intel/AMD VMs. The good news: most cloud-native Linux workloads migrate cleanly.Don't assume GPU is always needed for AI. Many inference workloads — especially smaller models, embeddings generation, and preprocessing — run perfectly well on CPU. Start with Cobalt 200 and upgrade to NCv6 only when GPU acceleration demonstrably improves latency or throughput.Network topology matters. Cobalt 200 supports up to 85 Gbps and NCv6 up to 180 Gbps. If your AI inference pipeline involves large data transfers between compute and storage, make sure your architecture accounts for this bandwidth differential.Memory encryption is free on Cobalt 200. It's enabled by default with negligible performance overhead. This is a security win — especially relevant for regulated industries in Southeast Asia handling sensitive data.Preview ≠ GA (Cobalt 200). Cobalt 200 is currently in preview in 8 regions (West US 3, East US 2, Central US, Sweden Central, East US, West US 2, Spain Central, and Indonesia Central). Availability is limited and pricing may shift. Plan your migration but don't commit production workloads until general availability is confirmed for your target region. (NCv6 is GA in West US 2 and Southeast Asia, so it is production-ready today.)ConclusionThe Azure compute landscape is evolving fast. Cobalt 200 gives CPU-bound workloads a compelling reason to move to Arm — better performance, lower cost, and a mature ecosystem that makes migration straightforward. NCv6 with RTX PRO 6000 Blackwell gives AI teams a practical, cost-effective GPU option for inference and visualization without the HPC-grade price tag of H100/B200 instances.For enterprise teams in Malaysia and across Southeast Asia, the playbook is clear: audit your fleet, pilot on Cobalt 200, benchmark AI workloads on NCv6, and build an architecture that uses both where they make sense. The cost savings from Arm can fund the GPU investment for AI. That's a trade worth making.Key TakeawaysCobalt 200 delivers 50% better CPU performance than Cobalt 100 with up to 135% improvement for database workloads — and Arm instances are typically 20–40% cheaper than comparable x86 VMs.NCv6 with RTX PRO 6000 Blackwell is a practical AI GPU with 96 GB GDDR7 memory, MIG support for fractional allocation, and Azure on-demand list pricing around $6.38/hr per single-GPU VM in West US 2 (Aug 2026).Migration from x86 to Cobalt 200 is a deployment decision, not an engineering project for most Linux workloads. Containerized applications with multi-arch images are the lowest-risk starting point.The optimal architecture uses both: Cobalt 200 for CPU-intensive tiers (web, API, cache, database) and NCv6 for GPU-accelerated AI inference — letting Arm cost savings fund GPU investment.Start with a pilot. Deploy non-critical workloads on Cobalt 200 to validate performance, then expand. For AI, benchmark your inference pipeline on NCv6 before committing to full production deployment. Related Reading LLM Deployment on Azure: Azure OpenAI vs. Self-Hosted Open-Source Models — A Cos Self-Hosting LLM Inference in 2026: vLLM vs SGLang vs llama.cpp Decision Guide & LLM Inference Optimization — Quantization, Speculative Decoding, and the 10x Cos --- ## Claude Fable 5 and the Export Control Ban — What It Means for Enterprise AI Strategy URL: https://wenfeng.my/claude-fable-5-export-control-ban-enterprise-ai-strategy/ Author: admin Published: 2026-08-31 23:30:22 Modified: 2026-10-04 15:30:53 On June 12, 2026, the US government issued an export control directive ordering Anthropic to suspend all access to Claude Fable 5 and Mythos 5 for any foreign national — whether inside or outside the United States. This was reportedly triggered by a jailbreak vulnerability discovered in the Mythos-class architecture. Anthropic complied immediately, disabling access worldwide within hours. Update: Export controls were lifted on June 30, and Anthropic began redeploying Claude Fable 5 on July 1, 2026 with updated cybersecurity safeguards and a new industry jailbreak framework. Mythos 5 remains restricted. The episode lasted roughly three weeks — but the precedent it set is permanent, and that is what enterprise architects need to plan around. This was the first time a frontier AI model has been pulled from global availability by government order. Forbes, BBC, and Snyk all covered the story extensively. If your enterprise strategy depends on Claude as your primary AI platform, this event demands immediate attention. What Actually Happened Claude Fable 5, part of Anthropic's Mythos architecture class, achieved state-of-the-art performance on nearly every major benchmark when it launched on June 9. It represented a significant capability leap: improved reasoning, better tool use, and stronger safety properties than previous generations. Then three days later, it was gone. The export control directive — reportedly issued under national security authorities — targeted the model specifically, not the company. This means: No model version is exempt. API access, on-premises deployments, and even cached weights fell under the directive. Geographic scope is absolute. Foreign nationals anywhere in the world were affected, including foreign employees of US companies. Compliance is immediate. Anthropic had no grace period for enterprise migration. For Malaysian enterprises, this is not a hypothetical scenario. It is a precedent that has already been set. Why This Matters for Enterprise AI Strategy The core lesson is not about Claude specifically. It is about model dependency risk — the concentration of your AI capabilities in a single provider's model, subject to a foreign government's policy decisions. The Dependency Chain Most enterprise AI deployments follow a dependency chain: Business Application → AI Framework → Model Provider → Model → Foreign Government Policy When you deploy Claude as your primary model, you are accepting the risk that any link in that chain can break without notice. The export control ban demonstrates that the final link — government policy — can override your cloud SLA, your vendor contract, and your technical architecture. Cost of the Fable 5 Interruption Organizations that had already adopted Fable 5 for production workloads faced: Immediate downtime for any workflow depending on Fable 5 APIs No migration path — the model was not deprecated, it was banned Forced re-evaluation of every prompt, every chain, every fine-tuned adaptation Potential compliance exposure for workflows that processed regulated data through a now-unavailable model The irony is that Fable 5's safety properties were among its selling points. A model specifically designed for safe deployment became unavailable due to safety concerns at the infrastructure level. The Multi-Model Resilience Framework The Fable 5 incident should accelerate every enterprise toward a multi-model architecture. Here is a practical framework. 1. Model Abstraction Layer Never hard-code your application to a specific model. Use an abstraction layer that allows model switching without application changes. # BAD: Hard-coded model dependency response = anthropic.messages.create( model="claude-fable-5", messages=[{"role": "user", "content": prompt}] ) # GOOD: Abstracted model routing class ModelRouter: def __init__(self): self.providers = { "primary": AzureOpenAIClient(model="gpt-5.5"), "fallback_1": AzureOpenAIClient(model="gpt-5-mini"), "fallback_2": SelfHostedClient(model="deepseek-v4"), } def route(self, task_type, prompt): provider = self.select_provider(task_type) return provider.complete(prompt) Azure AI Foundry's model catalog provides exactly this abstraction. With thousands of models from multiple providers, you can route workloads across providers without application changes. 2. Provider Diversification Spread your model dependencies across providers and jurisdictions: ProviderJurisdictionStrengthsRisk Profile Azure OpenAIUSGPT-5.x series, enterprise complianceSubject to US export controls Azure AI FoundryUS (multi-region hosting)Model catalog, governanceBroader model range reduces single-provider risk Self-hosted (DeepSeek, Qwen)China (model origin), your infra (deployment)No API dependency, PDPA data residencyModel origin jurisdiction risk Azure-hosted open-sourceYour Azure subscriptionData sovereignty, no API dependencyYou manage infrastructure The key insight: self-hosted models on your own Azure infrastructure eliminate API-level jurisdiction risk. A DeepSeek V4 model deployed on your Malaysia West AKS cluster has no foreign API dependency. 3. Automated Failover Testing Model failover should be tested, not assumed. Build automated tests that verify your primary model is available, confirm fallback models produce acceptable quality for your workload, and measure latency and cost differences between primary and fallback. Run these tests daily, not quarterly. # Example: Model health check in CI/CD model-health-check: runs-on: ubuntu-latest steps: - name: Check primary model availability run: | python3 tests/model_health.py \ --primary azure-openai/gpt-5.5 \ --fallbacks azure-openai/gpt-5-mini,self-hosted/deepseek-v4 \ --min-quality-threshold 0.85 4. PDPA Compliance Under Model Uncertainty Malaysia's Personal Data Protection Act requires data controllers to ensure adequate protection of personal data. When your AI model provider loses the ability to serve your region: Data already sent to the model API may be in transit or cached Fine-tuning data stored by the provider may be subject to new jurisdictional rules Embeddings and vectors generated by the model may need to be regenerated with an alternative Build PDPA compliance that accounts for model unavailability: Data classification — never send classified data to a single-provider API without a fallback plan Processing audit trail — log which model processed which data, for compliance and incident response Data deletion capability — ensure you can request and verify deletion from any provider Jurisdiction mapping — know where each model's inference endpoints are hosted Practical Migration Checklist If your enterprise currently depends on Claude as a primary model, here is a staged plan. Immediate (this week): Inventory all production workflows using Claude APIs Identify which workflows have no alternative model configured Set up at least one fallback model for each critical workflow Verify Azure AI Foundry catalog access for alternative models Short-term (30 days): Implement model abstraction layer in all Claude-dependent applications Create automated model health checks in CI/CD Document model dependency map for audit purposes Test failover for top-5 critical workflows Medium-term (90 days): Adopt Azure AI Foundry model catalog for model routing Evaluate self-hosted options for PDPA-sensitive workloads Build model performance benchmarks for your specific workloads Establish quarterly model risk review process Pitfalls to Avoid A few patterns I have seen in failed model diversification projects: Pretending a smaller model is a real fallback. If your primary is a 200B frontier model and your fallback is a 7B distill, you do not have resilience — you have a slower outage. Ignoring cost asymmetry. Failover routing is cheap insurance until the primary is down and you are paying 4x for a substitute model at peak. Self-hosting without an ops runway. Running DeepSeek V4 on your own AKS cluster means you own the patching, the model upgrade path, and the on-call. Budget for that, or do not pretend it is sovereign. Treating the ban as reversible. Even after Fable 5 came back, the terms changed. Plan for both: the loss and the conditional return. The Bigger Picture The Fable 5 ban is not an isolated incident. It is part of a pattern: Semiconductor export controls already restrict chip access to certain regions Data sovereignty regulations are tightening globally Model licensing terms are becoming more restrictive Geopolitical tensions are increasingly intersecting with technology availability For Malaysian enterprises, the implication is clear: model sovereignty matters as much as data sovereignty. Your AI strategy must account for the possibility that any model — no matter how capable — can become unavailable due to factors entirely outside your control. The enterprises that thrive will be those that treat models as interchangeable infrastructure, not as irreplaceable capabilities. Build your AI architecture on abstraction, diversification, and resilience — not on any single model's continued availability. Key Takeaways Model dependency is geopolitical risk. The Fable 5 ban proved that foreign government policy can override your cloud SLA without notice — even when the interruption is later reversed. Build a multi-model architecture now. Use Azure AI Foundry's model catalog or a similar abstraction layer to eliminate single-provider dependency. Self-hosting eliminates API jurisdiction risk. Open-source models (DeepSeek V4, Qwen 3.7) deployed on your own Azure infrastructure have no foreign API dependency. PDPA compliance must account for model unavailability. Data classification, audit trails, and deletion capabilities are essential when model availability is uncertain. Test failover regularly. Model availability testing should be part of your CI/CD pipeline, not a quarterly review exercise. --- ## Microsoft Fabric FabCon 2026 Recap — Data Agents, HorizonDB, and What It Means for Enterprise Data Platform URL: https://wenfeng.my/microsoft-fabric-fabcon-2026-recap-data-agents-horizondb/ Author: admin Published: 2026-08-31 06:30:22 Modified: 2026-10-04 15:24:54 Microsoft Fabric FabCon 2026 Recap — Data Agents, HorizonDB, and What It Means for Enterprise Data PlatformBy Law Wen Feng, Principal Solution Architect · wenfeng.my/I spent the last week catching up on everything Microsoft announced at FabCon Atlanta 2026 and Microsoft Build 2026, and frankly, there is a lot to unpack. If you are an Azure-first enterprise team in Southeast Asia trying to figure out what to do next with your data platform, this post is for you.Here is what stood out to me: Data Agents went GA, Azure HorizonDB hit public preview, Database Hub landed in Fabric, and GPU-accelerated Data Warehouse promises 7x faster queries. These are not just feature announcements — they signal a real shift in how Microsoft thinks about the enterprise data stack. Let me walk through each one and what it actually means for your team.Data Agents: Your Virtual Analysts, Now Production-ReadyData Agents hit general availability at FabCon 2026, and this is the announcement I am most excited about for practical reasons. Think of Data Agents as virtual analysts that sit on top of your governed Fabric data — they let users ask questions in natural language and get answers grounded in your actual enterprise data, not some generic model output.What makes them different from the chatbots we have been tolerating for the past two years? A few things:Grounded in governed data. Data Agents connect directly to your OneLake data, semantic models, and lakehouse tables. They respect your row-level and column-level security. When a user asks "What were our Southeast Asia Q1 revenue figures?", the agent queries your actual warehouse — it does not hallucinate numbers from training data.Domain-aligned. You can create agents scoped to specific business domains. Your finance team gets a finance agent, your supply chain team gets a supply chain agent. Each one is configured with the right data sources, the right semantic models, and the right business context.Cross-platform reach. Data Agents are not siloed inside Fabric. They can be used as knowledge sources in Microsoft Foundry, Copilot Studio, and Microsoft 365 Copilot. That means your finance team can access their finance agent directly from Teams without switching tools.Operations Agents. Beyond interactive Q&A, there are Operations Agents that continuously monitor live data and act on predefined business logic. Detect anomalies, trigger alerts, initiate workflows — all grounded in real-time data signals from Fabric.For enterprise teams, the production-readiness angle is key. This is not a research project. The improved NL2SQL engine is already in preview with better accuracy, transparency, and resilience when translating natural language to SQL. The agents handle query complexity behind the scenes — joins, aggregations, filters — while presenting results in natural language with supporting visualizations.The prerequisites are worth knowing before you plan a pilot: a paid F2 or higher Fabric capacity (or Power BI Premium P1+ capacity with Fabric enabled), and at least one supported data source — a warehouse, lakehouse, Power BI semantic model, KQL database, or mirrored database — with read access. If your tenant runs Microsoft Purview policies, agents operate within them: DLP policies can restrict access to sensitive warehouse assets, and access restriction policies can block results from classified sources. That is governance working as intended.If you are building a data culture where self-service analytics actually means something, Data Agents are the missing piece. The challenge has never been giving people access to data — it has been making that data accessible to people who do not speak SQL. This addresses that directly.Azure HorizonDB: PostgreSQL Meets Cloud-Native ScaleThis one is a bigger deal than most people realize. Azure HorizonDB is Microsoft's new fully managed, PostgreSQL-compatible database service, and it just entered public preview. Let me explain why this matters.HorizonDB is built on a cloud-native architecture with disaggregated compute and storage. That means you can scale them independently — something that traditional managed PostgreSQL services cannot do well. The numbers are impressive:Elastic storage scaling up to 128 TB — auto-scaling shared storage that grows with your dataCompute scale-out up to 3,072 vCores across primary and replica nodesDatabase-as-log architecture — transactions commit directly to shared, zone-resilient write-ahead log (WAL) storage, eliminating traditional checkpoint I/O bottlenecksUp to 3x faster performance compared with self-managed Postgres for transactional workloadsThe architecture is worth understanding because it explains the performance story. Compute and storage are fully disaggregated: one writable primary replica plus readable standby replicas share a single copy of zone-resilient data. The WAL service is optimized purely for transaction log semantics, and filtered WAL is sent only to the storage nodes that own the affected data. Storage nodes reconstruct page state by applying the WAL, which reduces write amplification and delivers consistent write latency regardless of database size. You get the full PostgreSQL ecosystem — compatibility, tooling, community — with cloud-scale behavior baked in.Provisioning is already covered by the Azure CLI, ARM templates, and the portal. Here is the CLI path from Microsoft's official quickstart — a zone-resilient cluster with two replicas:az extension add --name horizondb az horizondb create \ --resource-group rg-prod-data \ --name hz-orders-01 \ --location <region> \ --version 17 \ --administrator-login pgadmin \ --administrator-login-password <secure-password> \ --v-cores 2 \ --replica-count 2 \ --zone-placement-policy Strict Connection details follow the standard PostgreSQL wire protocol — any existing psql, driver, or ORM stack works:psql "host=hz-orders-01.<id>.<region>.horizondb.azure.com port=5432 \ dbname=postgres user=pgadmin sslmode=require" Vector Search and AI IntegrationHere is where HorizonDB gets really interesting for AI workloads. It ships with an advanced DiskANN vector index that introduces filtering capabilities — query predicate pushdowns directly into vector similarity search. This is a significant improvement over the pgvector HNSW approach that most managed Postgres services use today.Why does filtered vector search matter? Because in production, you almost never do a raw "find me the most similar items" query. You want "find me the most similar items where status = active AND region = APAC AND created_date > 2025-01-01". HorizonDB pushes those filters down into the index traversal, which means dramatically better performance on real-world queries.It also includes built-in AI model management. You can call generative, embedding, and reranking models directly from SQL, integrated with Microsoft Foundry. No separate vector database, no separate model hosting, no stitching together multiple services.As Mohsin Shafqat from NASDAQ put it: "Instead of stitching together multiple components, it brings transactional data, vector search, and AI capabilities into a single platform."Enterprise Security and ComplianceHorizonDB is enterprise-ready from day one:Native Microsoft Entra ID integrationPrivate Endpoints supportData encryption at rest and in transitAutomated backups with point-in-time recoveryFor teams in regulated industries — financial services, healthcare, government — these are not nice-to-haves. They are prerequisites.Database Hub: One Pane of Glass for EverythingThe Database Hub announcement is about operational simplicity, and honestly, it is overdue. If you are running a mix of Azure SQL, Cosmos DB, PostgreSQL, MySQL, and SQL Server on-premises via Azure Arc, managing them is a headache. Each has its own portal, its own monitoring, its own governance model.Database Hub in Fabric brings unified management for databases across edge, cloud, and Fabric into a single experience. It supports:Azure SQL DatabaseAzure Cosmos DBAzure Database for PostgreSQLSQL Server via Azure ArcAzure Database for MySQLFabric DatabasesThe key capabilities include agent-assisted management (Copilot-powered insights), built-in observability, and delegated governance. This is not just a dashboard — it is an intelligent management layer that can help you spot issues before they become incidents.For Southeast Asia teams running hybrid infrastructure — databases on-premises in your Singapore or KL datacenter alongside cloud databases — the Azure Arc integration means you can manage that on-prem SQL Server from the same interface as your cloud databases. One governance model, one monitoring surface, one place to see everything.GPU-Accelerated Data Warehouse: The 7x Performance LeapThis is the headline grabber, and for good reason. Microsoft announced GPU-accelerated Fabric Data Warehouse in early access preview at Build 2026, with early access starting in July 2026 across a limited set of regions — currently East US, East US 2, South Central US, Southeast Asia, and Germany West Central (sign-up required) — and no GA date committed yet.The numbers from Microsoft's benchmarks are striking: up to 7x faster than three comparable cloud data warehouse providers across reporting, application, and AI scenarios. At 64-user concurrency, the performance stays stable — which is the real story here, not just the peak speed.The underlying technology is called CoddSpeed, and it won the SIGMOD 2026 Best Industry Paper award. It uses NVIDIA accelerated computing, with a GPU-based execution engine derived from CoddSpeed's Tensor Query Processor (TQP) research. The optimizer intelligently offloads compute-intensive operations — large joins, aggregations — to GPUs while keeping other parts on CPUs.What This Means in PracticePer Microsoft's early access materials, UNC Health saw up to 5x improvement in query speeds, and WTW reported complex workloads running 3.4x faster at single concurrency. These are named enterprise customers — not synthetic benchmarks.And here is the part that should make you pay attention: you do not need to rewrite a single query. Enable it from workspace settings and it applies automatically to all SQL Analytics Endpoints and Data Warehouses in the workspace. Same T-SQL surface, same tools, same architecture — just faster execution.The scale of the gains matters more than any single benchmark run. Microsoft's own TPC-H testing at 100 GB on an F64 trial capacity found that the GPU-accelerated warehouse completes the full 22-query workload in about five seconds whether one user or 64 concurrent users are querying — while Microsoft's July 2026 benchmark blog reports roughly 6x throughput and 6–7x lower p95 latency versus the CPU path at high concurrency.If you are running complex analytical workloads — financial reporting, supply chain optimization, customer analytics — this is a performance multiplier that directly impacts time-to-insight.Practical Migration GuidanceOkay, so what should your team actually do with all of this? Here is my practical take for Azure-first enterprise teams in Southeast Asia:If You Are Already on FabricTurn on GPU acceleration. If your workspace has Data Warehouses, enable GPU acceleration from workspace settings. Run your existing queries and measure the difference. No risk, no rewrites.Start building Data Agents. Pick one domain — finance is usually the easiest win — and create a Data Agent scoped to your finance semantic model. Let your finance team try natural language queries. Iterate on the configuration.Explore Database Hub. If you are managing multiple database services, request early access to Database Hub. Unified management is a force multiplier for small DBA teams.If You Are on Azure SQL / Synapse TodayEvaluate HorizonDB for new PostgreSQL workloads. If you are starting a new AI-adjacent project that needs vector search, HorizonDB gives you relational + vector in one service. No separate vector database needed.Use the Migration Assistant. The ADF and Synapse pipelines migration assistant is in public preview. It automates the conversion of existing pipelines into Fabric. For SQL databases, a separate migration assistant (also in public preview) imports schemas through DACPACs, while a direct-connection option — generally available since June 2026 — lets the assistant connect directly to the source warehouse.Plan for Database Hub. As your database estate grows, having a unified management layer becomes critical. Start thinking about how Database Hub fits into your governance model.If You Are Considering Fabric from ScratchStart with mirroring. Fabric now mirrors SAP Datasphere (GA), Oracle (GA), SharePoint Lists (preview), Dremio (preview), and more. Use mirroring to get a low-risk copy of your existing data into OneLake without disrupting production systems.Build your semantic layer first. Before you jump to agents or AI, make sure your Power BI semantic models are solid. Fabric IQ builds on top of these, and Data Agents depend on them. A weak semantic layer means weak agents.Think about governance from day one. OneLake security reached general availability on Microsoft's announcement timeline, and now provides row-level and column-level controls, with further capabilities (Eventhouse coverage, security APIs, simplified column-level security) shipped in July 2026. Set them up early — retrofitting governance is always harder than building it in from the start.A Note on CostMicrosoft also announced Savings Plans for databases, offering up to 35% savings compared to pay-as-you-go on select services. If you are committing to Fabric long-term, this is worth evaluating with your Microsoft account team. In the Southeast Asia market, where cost optimization is often a key selling point for cloud adoption, every percentage point matters.Pitfalls to Watch Before You CommitAnnouncements are exciting; production is where the truth comes out. Based on what I have seen in the field, here is where teams are likely to stub their toes.1. Data Agents are only as good as your semantic layer. This is the single most common failure mode. If your Power BI semantic models have inconsistent naming, missing descriptions, or overlapping measures, the agent will produce plausible-sounding wrong answers. Budget real time for semantic model hygiene before you demo an agent to your CFO. A pilot with a clean, narrow domain scope beats a broad rollout every time.2. Prerequisites and capacity licensing. Data Agents require paid F2+ (or P1+) capacity. That sounds small, but in environments still running on per-user Power BI Pro licenses with no reserved capacity, this is a procurement change, not a toggle. Confirm your capacity model before promising a timeline to stakeholders.3. GPU acceleration is early access, not GA. Do not build migration business cases on the 7x headline until the feature reaches your region and your workload has been measured. Preview features change, region availability is limited, and your query mix may not match the benchmark scenarios. Treat it as upside, not baseline, in your TCO model.4. HorizonDB is in preview — plan the exit strategy too. For production-critical workloads, evaluate what happens at GA: pricing changes, supported regions, extension compatibility with your PostgreSQL toolchain. Because HorizonDB keeps full PostgreSQL wire compatibility, the blast radius of a misjudgment is smaller than a proprietary platform — but still validate your specific extensions and drivers.5. Database Hub does not replace your governance model. Unified management gives you one pane of glass, but it also centralizes the blast radius of misconfiguration. RBAC design, service principals, and audit logging across your Arc-registered SQL Servers need the same rigor as before — arguably more, since one console now touches everything.6. Mirroring is not migration. Mirrored data in OneLake is a low-risk copy for analytics — it does not retire the source system, and it adds latency considerations for anything that needs fresh data. Keep a clear line between "mirrored for analytics" and "migrated" in your roadmap, or stakeholders will assume the source database is already decommissionable.What It All Adds Up ToStepping back, the overarching story from FabCon 2026 is clear: Microsoft is building Fabric into the unified enterprise data and AI platform. Not just analytics, not just warehousing, not just databases — all of it, connected through OneLake, governed through Fabric IQ, and made accessible through Data Agents.The HorizonDB announcement signals that Microsoft is serious about owning the database layer for AI workloads. The GPU-accelerated warehouse shows they are willing to invest in hardware-level performance differentiation. And Data Agents going GA means the "talk to your data" promise is now production-ready, not just a demo.For enterprise teams in Southeast Asia, this is a moment to reassess your data platform strategy. The gap between "having data" and "using data effectively" has always been about accessibility, governance, and performance. These announcements address all three.The teams that move early on GPU acceleration and Data Agents will have a meaningful advantage in how quickly they can deliver AI-powered insights to their business stakeholders.Key TakeawaysData Agents are production-ready (GA). Start with one domain, ground it in your governed semantic model, and let business users ask questions in natural language. This is the lowest-risk, highest-impact entry point for enterprise AI on Fabric.HorizonDB changes the PostgreSQL game. With 128TB elastic storage, DiskANN filtered vector search, and built-in AI model management, it eliminates the need to stitch together separate relational and vector databases for AI workloads.GPU-accelerated Data Warehouse delivers up to 7x speedup with zero query changes — in early access preview today. Enable it from workspace settings once your tenant has access and measure the difference on your existing workloads.Database Hub unifies management across your hybrid database estate. Whether your databases are on-premises via Azure Arc, in Azure SQL, or in Cosmos DB, Database Hub gives you one interface to manage, monitor, and govern them all.Migration paths are maturing. The ADF/Synapse pipeline migration assistant, the SQL database migration assistant with DACPAC support, and the direct-connection option mean moving to Fabric is getting easier. Start with mirroring for low-risk data replication, then plan your full migration with proper governance.Got questions about Fabric, HorizonDB, or how to plan your migration? Reach out at wenfeng.my/ or connect with me on LinkedIn. --- ## Azure Landing Zone Transition Patterns — Migrating Existing Environments to Enterprise-Scale at Malaysia West URL: https://wenfeng.my/azure-landing-zone-transition-patterns-migrating-existing-environments-malaysia-west/ Author: admin Published: 2026-08-30 23:30:42 Modified: 2026-10-04 15:25:04 Azure Landing Zone Transition Patterns — Migrating Existing Environments to Enterprise-Scale at Malaysia WestMost Malaysian enterprises did not start Azure with a landing zone. They started with a subscription, got workloads running, and ended up with sprawl — ad-hoc resource groups, inconsistent policies, scattered RBAC, and no management group hierarchy. This is not a failure. It is the normal evolution of most Azure tenants.The good news is that Microsoft's Cloud Adoption Framework now has a well-defined, incrementally safe path to align brownfield environments with the Azure Landing Zone reference architecture. You do not have to tear everything down. You build the target in parallel and migrate progressively.What a Brownfield Environment Looks LikeA brownfield Azure environment is any existing deployment that was not built to ALZ from the start. Common characteristics:One or a few subscriptions doing everything (platform and workloads mixed together)No management group hierarchy, or a flat/shallow onePolicies applied ad-hoc at the subscription or resource group levelRBAC assignments scattered at multiple scopesNo centralized hub network or inconsistent connectivity designNo separation of platform functions (identity, connectivity, management) from application workloadsIf this describes your Azure tenant, you are in the majority. Microsoft's brownfield transition guidance — matured significantly in 2026 — is designed specifically for this reality.The Target: Azure Landing Zone Reference ArchitectureBefore transitioning, understand what you are moving toward. The ALZ reference architecture defines eight design areas: Letter Design Area Purpose A Azure billing and Entra ID tenant Tenant creation, enrollment, billing structure B Identity and access management Security boundary, RBAC, Entra ID integration C Resource organization Management group hierarchy, subscription design D, G, H Management Monitoring baseline, platform operations C, D Governance Policy-driven guardrails, Azure Policy at scale E Network topology and connectivity Hub-spoke, hybrid connectivity, DNS F Security Defender for Cloud, secure baselines I Platform automation and DevOps IaC, deployment pipelines Note: The legend letters come from the official Microsoft Learn design-areas diagram (learn.microsoft.com). Several letters appear under more than one design area (Management spans D, G, and H; Governance spans C and D), which is why those rows list multiple letters.The canonical management group structure:Tenant Root Group └── [Your Org] ├── Platform │ ├── Identity │ ├── Connectivity │ └── Management ├── Landing Zones │ ├── Corp (VNet-connected workloads) │ └── Online (internet-facing workloads) ├── Sandbox └── Decommissioned Workload subscriptions live under Corp or Online depending on whether they require private network connectivity.The Golden Rule: Build in Parallel, Migrate IncrementallyThe most important principle when transitioning a brownfield environment:Never modify production in place. Deploy the new structure in parallel and migrate workloads progressively.This guarantees:Active workloads are not disrupted during the transitionTeams can assess policy compliance before enforcement takes effectRollback is possible at any point — the old structure still worksMigration can be paced by team capacity and risk tolerancePhase 1: Assess Your Current StateBefore moving anything, document what you have.Inventory ResourcesUse Azure Resource Graph:Resources | summarize count_=count() by type, subscriptionId | order by count_ desc (The count_=count() alias is required so order by count_ desc resolves; bare count() cannot be used directly in order by.)Key questions to answer:How many subscriptions exist, and what is in each?Are platform resources (gateways, firewalls, DNS, monitoring) mixed with workload resources?What management groups exist today?What policies are assigned, at what scopes, and with what effects?What RBAC role assignments exist at which scopes?Identify What Cannot MoveSome resource types do not support Azure Resource Manager move operations. Cross-reference your inventory against the supported resources list before planning any migrations.Common resources that require care:Virtual Networks — peering must be removed first, then re-established afterVPN Gateways and ExpressRoute Gateways — significant reconfiguration requiredRecovery Services Vaults — active backup or replication configurations cannot be movedKey Vault — access policies and firewall rules need review post-moveManaged Identities — role assignments tied to ResourceId update on moveAudit Your Policy LandscapePolicyResources | where type == "microsoft.authorization/policyassignments" | project name, properties.scope, properties.policyDefinitionId, properties.enforcementMode This is critical input for the transition. You need to know what guardrails already exist before adding new ones.Phase 2: Deploy the Target Structure (Parallel, Not Replacing)Deploy the ALZ Management Group HierarchyDeploy the ALZ management group structure into your existing tenant without moving any subscriptions yet. This creates the target hierarchy alongside your current structure with zero impact to running workloads.Options to deploy:Azure Landing Zones Bicep accelerator (current canonical): github.com/Azure/alz-bicep-accelerator — based on Azure Verified Modules for Platform Landing ZoneAzure Landing Zones Terraform accelerator (current canonical): github.com/Azure/alz-terraform-accelerator — based on the Azure/caf-enterprise-scale/azurerm Terraform module (legacy module path: github.com/Azure/terraform-azurerm-caf-enterprise-scale)Azure Portal (accelerator UI): aka.ms/alz/acc — guides you through the Bicep or Terraform deploymentNote (Feb 2026): The classic Azure/ALZ-Bicep repo was removed on 16 February 2026. New deployments should use the alz-bicep-accelerator repo above, which is built on Azure Verified Modules (AVM) and is the documented successor.The deployment creates:The management group hierarchyALZ built-in policy assignments and initiativesRole definitionsDiagnostic settings baselineConfigure Policies in Audit-Only ModeThis is the Microsoft-recommended brownfield approach. Set all policy assignments to DoNotEnforce mode:{ "properties": { "policyDefinitionId": "/providers/Microsoft.Authorization/policySetDefinitions/...", "enforcementMode": "DoNotEnforce" } } This creates a "shadow compliance" view:Resources that violate policies appear as non-compliant in the portalNo deployments are blocked — Deny effects become no-opsDeployIfNotExists and Modify effects do not auto-remediateApplication teams evaluate compliance findings without risk of disruptionDeploy Platform SubscriptionsCreate new, clean subscriptions for platform functions: Subscription Purpose Connectivity Hub VNet, VPN/ExpressRoute gateways, Azure Firewall, DNS Identity Domain controllers, Entra ID Connect, AD DS Management Log Analytics workspace, Azure Monitor, Defender for Cloud Keep these separate from workload subscriptions from day one.Phase 3: Migrate WorkloadsMove Subscriptions into the New HierarchyMoving a subscription to a different management group does not affect existing resources within the subscription. ResourceId values do not change. The move primarily reorganizes governance boundaries.Requirements:Owner, Contributor, or Management Group Contributor role on the source subscriptionWrite permission on the target management groupAssessment of inherited policies from the new parent management groupMove Resources Across SubscriptionsWhen moving resources between resource groups or subscriptions:Source and target resource groups are locked during the moveResourceIds are updated, which breaks existing monitoring, alerts, and automationSome resource types do not support moveVNet peering must be disabled before moving virtual networksRecommendation: Minimize resource moves. Move subscriptions into the management group hierarchy first. Only move individual resources when workload separation requires it.Policy Compliance AssessmentAfter subscriptions are in the new hierarchy, review the non-compliance findings:Resources marked non-compliant under ALZ standardsGroup findings by policy and workload ownerCreate remediation plans with application teamsPrioritize by security impactPhase 4: Enable EnforcementRemediate Non-Compliant ResourcesWork through the remediation backlog systematically:Critical security findings — public IPs on databases, missing encryption, unrestricted network accessIdentity and access — excessive permissions, missing managed identitiesGovernance — missing tags, incorrect naming, resource organizationNetworking — inconsistent DNS, missing NSGs, peer connectivity gapsTransition from Audit to EnforceMove subscriptions from the audit management group to the enforced management group as they become compliant:# Check compliance status az policy state list --management-group "YourOrg-LandingZones-Audit" --query "[?complianceState=='NonCompliant']" -o table # Move when compliant az account management-group subscription add \ --name "YourOrg-LandingZones" \ --subscription "subscription-id" Decommission Old Management GroupsOnce all subscriptions are moved and enforced, decommission the temporary audit management groups and any legacy structure.Phase 5: Rebuild PlatformHub Network MigrationThe hub network migration is typically the most complex phase. The parallel approach:Deploy new hub VNet in the Connectivity subscriptionConfigure VPN/ExpressRoute gatewaysEstablish new peering to spoke VNetsMigrate DNS resolutionMigrate firewall rules and network security groupsCut over trafficDecommission old hub networkMalaysia West specific considerations: The Malaysia West region (GA May 2025) makes this urgent as enterprises plan for dual-region architectures with the upcoming Johor Bahru region. Design your hub for West-to-JB connectivity from the start.DNS MigrationConfigure Azure DNS Private Zones for internal resolution:Split-horizon DNS for hybrid name resolutionPrivate DNS zones linked to spoke VNetsDNS forwarding between Azure and on-premisesMonitoring MigrationDeploy Log Analytics workspace in the Management subscriptionConfigure diagnostic settings for all platform resourcesMigrate existing alerts and dashboardsEnable Defender for Cloud across all subscriptionsMalaysia West Specific ConsiderationsPDPA ComplianceMalaysia's Personal Data Protection Act applies to how data is organized, stored, and accessed in Azure. The ALZ governance layer provides the structure to implement PDPA controls:Data classification: Use Azure Policy to enforce sensitivity labels on storage and databasesAccess control: RBAC at management group scope ensures consistent access policiesAudit logging: Centralized monitoring in the Management subscription provides audit trailsData residency: Malaysia West region keeps data within Malaysian jurisdictionHub Connectivity DesignFor enterprises with on-premises data centers in Malaysia:ExpressRoute through local providers (Time, TM, Maxis) for production workloadsVPN Gateway for backup connectivity and branch officesConsider Virtual WAN if you have multiple branch offices or need SD-WAN integrationDual-Region PlanningWith the Johor Bahru region expected:Design hub connectivity for West-to-JB peeringUse Azure Policy to enforce region assignments for data sovereigntyPlan for cross-region disaster recovery with Azure Site RecoveryConsider availability zones in Malaysia West for intra-region resilienceCommon Failure PatternsAssessing Policy Compliance Too LateTeams often enable enforcement before understanding the compliance gap. Always start in audit mode and build the remediation backlog before flipping to enforce.Moving Recovery Services VaultsVaults with active backup or replication workloads cannot be moved in most cases. Create a new vault in the target subscription, reconfigure backups, and let the old vault's data age out per its retention policy.Forgetting ResourceId ChangesEvery resource move updates ResourceIds. Alerts, dashboards, runbooks, and ARM-based automation referencing old ResourceIds will stop working. Audit automation dependencies before any move.Not Assessing RBAC Before Moving SubscriptionsManagement group moves do not change direct RBAC on subscriptions, but they add inherited permissions from the new parent. Audit who would gain access before moving.Over-Engineering the Initial DeploymentDeploy only the management group hierarchy and policies in Phase 2. Do not try to deploy hub networking, identity, and monitoring simultaneously. Phase them based on risk and team capacity.Summary: The Transition SequencePhase 1: Assess current state └── Inventory resources, policies, RBAC └── Identify what can and cannot move Phase 2: Build target in parallel └── Deploy ALZ management group hierarchy └── Assign policies in DoNotEnforce (audit) mode └── Create new Platform subscriptions Phase 3: Migrate workloads └── Assess audit compliance findings with app teams └── Move subscriptions into new hierarchy └── Move resources across subscriptions where needed Phase 4: Enable enforcement └── Remediate non-compliant resources └── Move subscriptions from audit to enforced └── Decommission old management groups Phase 5: Rebuild platform └── Migrate hub network (parallel approach) └── Migrate DNS and monitoring └── Decommission old platform resources The brownfield transition is not a one-time project. It is a continuous improvement process that moves your Azure environment from ad-hoc to governed without the risk of a big-bang migration. Start with assessment, build the target alongside your current state, and migrate when the compliance picture is clear. The enterprises that get this right do not just save on cost — they gain the governance foundation that makes every subsequent Azure investment more secure and more manageable.Key Takeaways Build in parallel, migrate incrementally. Never modify production in place. Deploy the ALZ management group hierarchy, audit-mode policies, and platform subscriptions alongside your existing structure, then move workloads one at a time as the compliance picture clears. This guarantees rollback is always possible and active workloads are never disrupted. Start every policy in `DoNotEnforce`. Brownfield environments rarely pass ALZ baselines on day one. Use audit mode to build a remediation backlog with application teams, prioritise by security impact, and only flip to enforcement after non-compliance is driven to zero on the subscription or management group you are moving. Minimize resource moves; move subscriptions instead. Moving a subscription to a new management group leaves ResourceIds untouched and preserves monitoring, alerts, and runbooks. Cross-subscription resource moves break ResourceIds, so reserve them for cases where workload separation truly requires it. Plan for things that cannot move. Recovery Services Vaults with active backup, ExpressRoute gateways, and certain identity resources do not support ARM move. Sequence them out before the migration window: stand up a new vault, reconfigure backups, and let the old data age out per its retention policy. Treat the target as Malaysia-aware from day one. If you are running in Malaysia West, design the hub for the upcoming Johor Bahru peering, keep regulated workloads inside Malaysian jurisdiction for PDPA, and use Azure Policy to enforce region assignment so the regional expansion is a configuration change rather than a redesign. --- ## AKS at Build 2026 — Blue-Green Agent Pools, Cross-Cluster Networking, and GPU Monitoring URL: https://wenfeng.my/aks-build-2026-blue-green-agent-pools-cross-cluster-networking-gpu-monitoring/ Author: admin Published: 2026-08-30 06:30:24 Modified: 2026-08-30 06:30:24 Microsoft Build 2026 was, above all, about running AI workloads on Kubernetes. The official AKS announcement list reads like an AI platform roadmap: managed system node pools in AKS Automatic going GA, Azure Container Linux GA as the standardized host OS, AKS on bare metal in public preview for direct NVLink and RDMA access, Fleet Manager support for Arc-enabled clusters going GA, managed Ray through Anyscale on Azure, and the AI Runway plus KAITO model-serving stack.But behind the AI headline, the operational foundations kept maturing too. Three capabilities matter most to the production teams I work with in Kuala Lumpur and Singapore: blue-green node pool upgrades (now configurable with soak times and batch drains), cross-cluster networking in Azure Kubernetes Fleet Manager, and AKS-managed GPU node pools that ship DCGM metrics out of the box. None of these are brand new at Build — some are GA, some are preview — but together they answer the question every enterprise AKS owner eventually asks: how do I upgrade, connect, and monitor this thing without a maintenance window?Here is where each one actually stands, with the commands I use in real deployments.Blue-Green Node Pool Upgrades: Soak Times Instead of Upgrade WindowsIf you have ever stared at a Kubernetes upgrade at 2 AM, waiting for the last node to drain while your SLA bleeds away, blue-green node pool upgrades are the capability you have been asking for.The mechanism is now first-class in AKS: every node pool carries an upgradeStrategy. The default is Rolling — the traditional surge-and-reimage flow. Set it to BlueGreen, and upgrades behave differently: AKS provisions a full set of green nodes at the target version, drains workloads off the blue nodes in controlled batches, and keeps the blue nodes around for a final soak period before deletion. Your old capacity stays intact until you are satisfied with the new one.There are no az aks nodepool cordon or drain commands — the orchestration is declarative, driven by four tuning parameters on the node pool:--drain-batch-size — number or percentage of nodes drained per batch (default 10%)--batch-soak-duration — wait time between batches in minutes (default 15)--drain-timeout-bg — eviction timeout per node, honors Pod Disruption Budgets (default 30 minutes)--final-soak-duration — how long old nodes stay around after draining completes (default 60 minutes)Enabling the Blue-Green Strategy # New node pool with blue-green upgrade strategy az aks nodepool add \ --resource-group rg-aks-prod \ --cluster-name aks-prod-my \ --name nodepool1 \ --upgrade-strategy BlueGreen # Tune the upgrade behavior on an existing pool az aks nodepool update \ --resource-group rg-aks-prod \ --cluster-name aks-prod-my \ --name nodepool1 \ --drain-batch-size 50% \ --drain-timeout-bg 5 \ --batch-soak-duration 10 \ --final-soak-duration 10 When a cluster or node pool upgrade runs (az aks upgrade / az aks nodepool upgrade), the blue-green path executes automatically. You watch the batches soak, validate the green capacity under real traffic, and the old nodes are deleted only after the final soak completes.Why Malaysian Enterprises CareFor customers operating under strict change management — financial institutions and government-linked companies in particular — the soak windows are a compliance feature, not just an engineering one. A batch soak gives you a defined observation point to attach to your change record. The 60-minute default final soak is your rollback window: if something surfaces on the green nodes, the blue capacity is still there and still registered.Two operational notes from experience:Size for double capacity. During the upgrade you temporarily carry blue and green nodes. If your quota or budget cannot absorb a full second pool, keep batch sizes small and drains slow.The strategy applies to Kubernetes version and node image upgrades. It is not a substitute for application-level blue-green deployments, which remain your responsibility at the ingress and DNS layer.Cross-Cluster Networking with Fleet ManagerMost teams I meet are not running one cluster — they are running several, often split by data residency. A common ASEAN pattern: a primary cluster in Malaysia West for PDPA-sensitive workloads, with a second cluster in Southeast Asia (Singapore) for DR and regional failover. Historically, unifying networking across those clusters meant manual DNS work, service mesh overlays, or custom ingress glue.Azure Kubernetes Fleet Manager now covers three distinct traffic problems, and it is worth being precise about which is which:East–west (service-to-service): cross-cluster networking lets endpoints in member clusters talk directly, with network policy enforcement and global service discovery integrated with CoreDNS.North–south, public: DNS load balancing wires exported services into Azure Traffic Manager via TrafficManagerProfile and TrafficManagerBackend CRDs, giving exported services a globally available public DNS name.North–south, intra-VNet: the MultiClusterService resource makes each member cluster's Azure Load Balancer route to local endpoints and to endpoints of the same service on other member clusters.Multi-cluster networking features are still maturing — check current preview/GA status per capability before committing — but the primitives are stable enough to design against.Joining Two Clusters to a Fleet # Register the Malaysian cluster az fleet member create \ -g rg-fleet \ -f fleet-asean \ -n aks-my-central \ --member-cluster-id "/subscriptions/{sub-id}/resourceGroups/rg-aks-my/providers/Microsoft.ContainerService/managedClusters/aks-my-central" # Register the Singapore cluster az fleet member create \ -g rg-fleet \ -f fleet-asean \ -n aks-sg-dr \ --member-cluster-id "/subscriptions/{sub-id}/resourceGroups/rg-aks-sg/providers/Microsoft.ContainerService/managedClusters/aks-sg-dr" Exposing a Service Across ClustersOn the member cluster, export the service: apiVersion: networking.fleet.azure.com/v1alpha1 kind: ServiceExport metadata: name: payment-api namespace: production On the hub, declare a MultiClusterService that load-balances across all member endpoints: apiVersion: networking.fleet.azure.com/v1alpha1 kind: MultiClusterService metadata: name: payment-api namespace: production spec: ports: - port: 80 targetPort: 8080 In practice this is what makes the Malaysia-plus-Singapore pattern operationally sane: the failover story stops being a runbook full of DNS changes and becomes a placement and health-check problem that the fleet control plane handles. Note that member clusters must share a Microsoft Entra tenant, though they can span regions, resource groups, and subscriptions.GPU Monitoring Without Building Your Own Exporter StackIf you run inference servers or training jobs on AKS, you have probably operated the NVIDIA GPU Operator yourself — driver versions, device plugin, a DCGM exporter scraping somewhere, dashboards bolted on after the fact. That stack works, but it is undifferentiated operational overhead.AKS-managed GPU node pools (public preview) shift that burden to the platform. AKS installs and maintains the NVIDIA driver, the device plugin, the DCGM metrics exporter, and node-problem-detector GPU health signals for you. The DCGM exporter exposes standard Prometheus metrics — DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_GPU_TEMP, memory and ECC counters — on port 19400, labeled with kubernetes.azure.com/dcgm-exporter=enabled.Creating a Managed GPU Node Pool # Requires aks-preview extension >= 19.0.0b29 and the # ManagedGPUExperiencePreview feature flag registered az extension add --name aks-preview az feature register --namespace Microsoft.ContainerService \ --name ManagedGPUExperiencePreview az aks nodepool add \ --resource-group rg-aks-ai \ --cluster-name aks-gpu \ --name gpupool \ --node-vm-size Standard_NC40ads_H100_v5 \ --node-count 2 \ --enable-managed-gpu true Wiring DCGM Metrics into Azure Monitor az aks update \ --resource-group rg-aks-ai \ --name aks-gpu \ --enable-azure-monitor-metrics With Azure Monitor managed service for Prometheus enabled on the cluster, the DCGM metrics are scraped into your Azure Monitor workspace and become available in Grafana dashboards and PromQL queries — no exporter deployment to maintain. Alerting follows the same path as the rest of your platform signals: az monitor account create \ --resource-group rg-monitoring \ --name amw-aks-alerts From there, Prometheus-rule alerts (for example on sustained 100% utilization or rising GPU temperature) are defined in the workspace exactly like any other container metric.Two caveats worth knowing before you plan around preview limitations: managed GPU node pools currently support Linux pools only, and cluster autoscaler is not supported on them during preview — scale these pools manually. Both are typical preview constraints, but they bite real capacity plans, so model them now rather than at cutover.The Compliance Angle: PDPA and Operational ControlThe Personal Data Protection Act 2010 — including the PDPA (Amendment) Act 2024, which passed in 2024, phased in from 1 January 2025, and added mandatory data breach notification obligations (effective 1 June 2025) — is the baseline every Malaysian enterprise handling personal data must design against. The three capabilities above map onto it in concrete ways:Data residency: keeping PDPA-covered workloads on clusters in Malaysia West, with Fleet Manager handling cross-cluster routing for DR, lets you hold residency boundaries without fragmenting operations.Access control and integrity: blue-green upgrades with soak windows mean patching and version upgrades happen without availability gaps — and with an auditable observation point between batches. Combine with Entra-only identity (--disable-local-accounts) for centralized authentication and RBAC.Breach detection: native GPU and cluster metrics flowing into Azure Monitor give your SOC a single pane for anomaly detection and alerting, integrated with Sentinel — which matters now that notification obligations carry real timelines.I have been building exactly this mapping with Malaysian financial institutions. The combination of fleet-level networking for residency, blue-green upgrades for availability, and Azure Monitor for detection gives auditors something they can actually verify rather than a box-ticking narrative.Pitfalls I Keep SeeingAssuming blue-green means zero planning. You still need quota and budget for double node capacity during upgrades, and PDBs still govern drain behavior. A misconfigured PDB can stall a batch until --drain-timeout-bg expires and the upgrade fails.Treating preview as GA. Cross-cluster networking capabilities and managed GPU pools are at different maturity points. Read the current status of each feature before putting it on a critical path — especially anything feeding a DR commitment.Forgetting tenant constraints. Fleet member clusters must live in the same Microsoft Entra tenant. Cross-tenant scenarios need a different design.Self-managed GPU drift. If you stay on the self-managed GPU stack, pin driver and plugin versions and test upgrades deliberately. Version skew between driver and CUDA workloads remains a top incident cause in AI clusters.Skipping the soak. Shortening --final-soak-duration to zero to save money removes your rollback window. The old nodes are your insurance policy; keep them for at least one full validation cycle.Key TakeawaysBlue-green node pool upgrades are declarative and tunable. Set --upgrade-strategy BlueGreen and control batch size, soak durations, and drain timeouts per pool — no custom cordon/drain scripting, and a built-in rollback window.Fleet Manager separates three networking problems. East–west service discovery, public DNS load balancing via Traffic Manager, and intra-VNet MultiClusterService L4 load balancing are distinct primitives — pick the one that matches your traffic pattern.Managed GPU node pools remove the exporter tax. Driver, device plugin, DCGM metrics exporter, and health signals installed and maintained by AKS; standard DCGM_FI_* Prometheus metrics flow straight into Azure Monitor.Preview constraints shape capacity plans. No cluster autoscaler on managed GPU pools during preview; Linux-only GPU pools; verify per-feature GA status before wiring into DR.Build 2026's AI story stands on these foundations. Managed system node pools, bare metal AKS, and model serving all assume you can upgrade safely, connect clusters cleanly, and observe GPUs natively.Law Wen Feng is a Principal Solution Architect specializing in Azure infrastructure and cloud-native architecture across Southeast Asia. He writes about Azure, Kubernetes, and the intersection of technology and compliance in the region at wenfeng.my/. --- ## Microsoft Fabric Apps + Rayfin: The Open-Source BaaS That Turns Fabric Into an Application Platform URL: https://wenfeng.my/microsoft-fabric-apps-rayfin-open-source-baas/ Author: admin Published: 2026-08-29 23:30:17 Modified: 2026-10-04 15:25:15 Microsoft Fabric Apps + Rayfin: The Open-Source BaaS That Turns Fabric Into an Application PlatformAt Microsoft Build 2026, Microsoft introduced something that quietly shifts the Fabric conversation from "analytics platform" to "full application platform." The piece is Rayfin — a new, open-source SDK and CLI that lets you describe a complete application backend in TypeScript and ship it to Microsoft Fabric in a single command. It is MIT-licensed, currently in public preview, and the repo is already at 590 stars and 62 forks as of late August.If you have been stitching together Cosmos DB, App Service, Entra ID, APIM, and OneLake just to give an AI agent a writeable data plane, this is the announcement that lets you stop doing that. Let me show you what Rayfin is, what it actually replaces, and how to ship your first app.The Gap Rayfin ClosesAI coding agents can scaffold a clean frontend in seconds. The backend is still where the work piles up. The moment your app needs real data, you are stitching together a database, an identity provider, access policies, hosting, and then reconciling all of that with the governance and compliance rules your organization already enforces. That is the gap between a prototype that demos well and a system you can actually run. Prototypes are easy to build and hard to scale.Rayfin exists to close that gap. You describe your data model with TypeScript decorators, and Rayfin provisions and manages the backend for you: database, authentication, data APIs, storage, and hosting. Because the deployment target is Microsoft Fabric, application data lands directly in OneLake — unified with analytics, operational data, real-time data, and AI engines by default, and inheriting Fabric's governance, security, and compliance from day one.Microsoft positioned it as a way to "move from prompt to production without managing infrastructure." The Replit partnership — confirmed by Amjad Masad in the same launch — makes the same point: agents write the code, Fabric ships it.What Rayfin Actually IsRayfin is a fully managed Backend-as-a-Service (BaaS) runtime that runs on Microsoft Fabric. Think Firebase on Google Cloud, or Supabase on PostgreSQL — but built natively on the Fabric data estate, with the Microsoft Learn docs explicitly calling it "Backend-as-a-Service with built-in auth, data, and more."The full surface area:Decorator-driven data models. You define entities as TypeScript classes with @entity(), @text(), @int(), @uuid(), @date(), @boolean(), @decimal(), @email(), and @set(). Rayfin generates the table, the GraphQL API, and the type-safe client.Auto-generated GraphQL + REST. No hand-written controllers, no DTO classes. The schema is the API.Authentication built in. Auth is mandatory for Fabric deployments (turning it off causes npx rayfin up to fail). Fabric SSO, Entra ID, and local password auth for dev are all supported via a single YAML config.OneLake storage with native governance. Application data lands in your tenant's OneLake, with the same lineage, access controls, and audit logging as your Power BI datasets.Agent-friendly by design. The repo ships dedicated plugins for Claude Code (.claude-plugin/), GitHub Copilot (.github/), Cursor (.cursor-plugin/), Codex (.codex-plugin/), and others. Coding agents can scaffold, modify, and deploy a complete backend from a natural-language spec.MIT licensed. Fork it, vendor it, audit it, ship it on-prem if you ever need to.The GitHub repo is at github.com/microsoft/rayfin under MIT. The Microsoft product page is at aka.ms/rayfin, and the official Microsoft Learn docs live under learn.microsoft.com/en-us/fabric/apps/.How It Compares to Alternatives CapabilityRayfin (Fabric Apps)Power AppsAzure App Service + CosmosSupabase **Data platform**OneLake (Fabric)DataverseCosmos DB (your choice)PostgreSQL **Governance**Native FabricDataverse DLPManualManual **Auth**Entra ID + Fabric SSOEntra IDBring your ownSupabase Auth / OIDC **Open source**Yes (MIT)NoNoYes (Apache 2.0) **Code-first**TypeScript decoratorsLow-codeAny languageSQL + TypeScript SDK **Agent-Ready**First-class pluginsNoNoPartial **Cost model**Fabric FPUPer-user licensingApp Service plan + RU/sFree tier + paid **Maturity**Public preview (June 2026)GAGAGA The interesting row is governance. With Supabase or a hand-rolled App Service + Cosmos stack, you have to manually replicate OneLake's access controls, lineage, and audit logging for the app's data. With Rayfin, that work disappears — the data lands in OneLake and the governance model is inherited automatically.The Data Model: How It Actually WorksThe programming model is decorator-driven, not function-call-driven. The Microsoft Learn data-models doc is the source of truth here. Every entity uses a UUID id as its primary key (auto-generated if you don't supply it), and you compose fields with the typed decorators. import { entity, uuid, text, int, decimal, boolean, date, email, set } from '@microsoft/rayfin-core'; @entity() export class Product { @uuid() id!: string; @text({ min: 1, max: 200 }) name!: string; @text({ optional: true, max: 2000 }) description?: string; @decimal() price!: number; @int({ min: 0 }) stockQuantity!: number; @boolean({ default: true }) isAvailable!: boolean; @date() createdAt!: Date; @set('draft', 'published', 'archived') status!: 'draft' | 'published' | 'archived'; } @entity() export class Customer { @uuid() id!: string; @email({ unique: true }) email!: string; @text({ min: 3, max: 50 }) username!: string; @int({ min: 0, max: 150 }) age!: number; @boolean({ default: false }) isVerified!: boolean; } Relationships use @one() and @many() navigation decorators. Many-to-many is not supported — you build an explicit join entity instead. import { entity, uuid, text, date, one, many } from '@microsoft/rayfin-core'; @entity() export class Notebook { @uuid() id!: string; @text() name!: string; @date() createdAt!: Date; @many(() => Note) notes?: Note[]; } @entity() export class Note { @uuid() id!: string; @text() title!: string; @text() content!: string; @date() createdAt!: Date; @text() notebook_id!: string; @one(() => Notebook) notebook?: Notebook; } A few rules worth burning in:The TypeScript ? optional marker does not make the column nullable. Use { optional: true } on the decorator. The docs are explicit about this.Foreign key fields (notebook_id above) only need to be declared if you read or set them in code. The framework auto-generates them from the navigation decorators.Use relative imports with .js extensions in entity files so the emitted ESM JavaScript resolves correctly.Every entity has to be registered in rayfin/data/schema.ts or the API will not see it.Authentication and PermissionsAuth is a first-class YAML config, not a custom service. From the Microsoft Learn authentication doc: services: auth: enabled: true allowedRedirectUris: - http://localhost:5173 fabric: enabled: true # Fabric SSO (Entra ID) for the workspace password: enabled: true # Local username/password — dev only Key facts:Setting services.auth.enabled to false causes npx rayfin up to fail. There is no off switch for production.For production, you want fabric.enabled: true (Entra ID / Fabric SSO) and password.enabled: false.The session is exposed to your code as user.id and user.email — that is the whole identity surface area.Row- and column-level permissions are configured in a separate data-permissions layer (see the Microsoft Learn link above).Deploying to FabricThe day-to-day loop is short. From the Build 2026 lab materials and the Microsoft Learn deploy docs: # Scaffold a new project npm create @microsoft/rayfin@latest # Or initialize in an existing folder npx rayfin init # Local dev (live reload, SQLite under the hood) npx rayfin dev # Sign in with Entra ID the first time you deploy npx rayfin login # Deploy to Fabric — provisions backend, runs migrations, applies permissions npx rayfin up # Health check npx rayfin dev status A plain npx rayfin up does more than just push code. It redeploys the backend and runs any needed schema migration in one shot. There is a safety edge worth knowing about: if a migration would drop a non-empty column, the CLI will prompt before proceeding. In CI you pass --yes to skip the prompt.Practical Walkthrough: A Customer Support PortalLet's put it together end-to-end. The scenario: a small B2B SaaS company wants a support portal where customers can open and track tickets. Today this is a TypeScript/React app on App Service with Cosmos DB and Entra ID. With Rayfin, the data layer and auth collapse to a handful of files.1. Scaffold and define models npm create @microsoft/rayfin@latest support-portal cd support-portal // rayfin/data/schema.ts import { entity, uuid, text, date, set, one, many } from '@microsoft/rayfin-core'; import { Customer } from './customer.js'; import { SupportTicket } from './ticket.js'; export const schema = [Customer, SupportTicket]; // rayfin/data/customer.ts import { entity, uuid, text, email, set, many } from '@microsoft/rayfin-core'; import { SupportTicket } from './ticket.js'; @entity() export class Customer { @uuid() id!: string; @email({ unique: true }) email!: string; @text() companyName!: string; @set('free', 'pro', 'enterprise') tier!: 'free' | 'pro' | 'enterprise'; @date() createdAt!: Date; @many(() => SupportTicket) tickets?: SupportTicket[]; } // rayfin/data/ticket.ts import { entity, uuid, text, date, set, one } from '@microsoft/rayfin-core'; import { Customer } from './customer.js'; @entity() export class SupportTicket { @uuid() id!: string; @text() subject!: string; @text({ max: 5000 }) description!: string; @set('open', 'in_progress', 'resolved', 'closed') status!: 'open' | 'in_progress' | 'resolved' | 'closed'; @date() createdAt!: Date; @date() updatedAt!: Date; @text() customer_id!: string; @one(() => Customer) customer?: Customer; } 2. Wire up auth # rayfin/rayfin.yaml services: auth: enabled: true allowedRedirectUris: - http://localhost:5173 - https://support.example.com fabric: enabled: true password: enabled: false # production — Entra ID only 3. Deploy npx rayfin login # one-time Entra ID sign-in npx rayfin up # provisions backend, runs migration, returns the GraphQL endpoint What you get back: a GraphQL endpoint with full CRUD on Customer and SupportTicket, type-safe clients you can generate for any frontend, OneLake storage with lineage back to Power BI, audit logging on every read/write, and Entra ID auth for the front door. The whole thing is a few hundred lines of TypeScript and one YAML file.When Rayfin Makes SenseYou are already on Fabric. Your data is in OneLake, your governance is in Fabric, and you want to build applications that operate on the same data without ETL or a separate operational store.You need governed APIs fast. Instead of building an API tier, identity layer, and audit pipeline, you define entities and you get all three.Your team is using coding agents. The repo ships agent plugins for Claude, Copilot, Cursor, and Codex. A "build me a ticketing portal backed by Entra ID" prompt becomes a working deployable in hours, not weeks.You are a regulated Malaysian enterprise. Fabric inherits Bank Negara Malaysia, PDPA, and healthcare-sector compliance controls you have already set up. Rayfin apps get those controls for free.When It Doesn't (Yet)Heavy compute or long-running jobs. Rayfin is CRUD-shaped. For CPU-intensive work (image processing, ML inference, video transcoding), use Azure Functions or Container Apps and call them from Rayfin.Real-time event streaming at scale. If your app ingests millions of events per second, Event Hubs + Eventhouse is still the right fabric pattern; use Rayfin for the application surface on top.Polyglot persistence. Rayfin data lives in OneLake. If your app needs a graph DB, a vector DB outside of Fabric's built-in AI services, or a third-party SaaS as a system of record, you are back to stitching.Anything that needs to be in production next Tuesday. It is public preview. The API, decorators, and migration behavior can still change. Pin your Rayfin version and budget migration time.Implications for Malaysian EnterprisesFor enterprises already invested in Fabric, Rayfin changes the conversation about application development in three concrete ways.Data gravity. When your analytics, AI, and application data all live in OneLake, the data silos that plague most enterprises simply do not form. A customer portal built on Rayfin queries the same tables your Power BI dashboards use — no sync, no ETL, no drift. For Malaysian groups that are tired of paying twice for the same data (once in a transactional system, once in a warehouse), this is a real shift.Governance without overhead. Malaysian enterprises in regulated industries — banking (BNM RMiT), healthcare (Ministry of Health guidelines), government (MyGov policies) — need governance that is enforced, not just documented. Rayfin inherits Fabric's governance model, so application data gets the same access controls, lineage tracking, and compliance reporting as analytical data. The Leatherman team said it best in the Build 2026 customer story: "We appreciate how quickly we can build and iterate in Replit, but for some of our business, our data needs to stay governed and centralized in Fabric. With Rayfin, we finally have both."*Cost predictability. Rayfin runs on Fabric capacity units (FPU). If you are already paying for Fabric, the marginal cost of adding application workloads is incremental and rides on the same consumption model. No separate SaaS subscription, no per-user Power Apps licensing, no App Service plan on top of everything else.Pitfalls to WatchA few things I would not skip if I were rolling this out:Pin your Rayfin version. Public preview means the API can change. Use a specific version (@microsoft/rayfin-core@x.y.z) in package.json, not latest.Migration safety. npx rayfin up runs migrations automatically. In CI, use --yes to skip the interactive prompts, but in interactive dev sessions, read the diff it shows you. Dropping a non-empty column is destructive.Auth is mandatory. You cannot disable it. Plan your Entra ID app registration and redirect URIs before you start.No many-to-many. Use an explicit join entity. The framework will not auto-create one.The `?` optional marker lies. It is a TypeScript static-type thing, not a database-nullability thing. Use { optional: true } on the decorator or your column will be NOT NULL.OneLake geography follows your Fabric capacity. Your Rayfin app's data residency is your Fabric capacity's residency. Pick the Fabric region that matches your data residency requirements up front — switching later is a migration, not a config change.No raw SQL escape hatch (yet). The generated GraphQL API is the surface. If you need stored procedures, complex joins the GraphQL schema cannot express, or bulk imports, use a Fabric SQL Database alongside the Rayfin app and wire them together with a Fabric data pipeline.Getting StartedMake sure Fabric Apps is enabled in your Fabric workspace. New workspaces created after Build 2026 have it on by default; older workspaces need a feature flag in the admin portal.Scaffold a project: npm create @microsoft/rayfin@latestDefine entities in TypeScript using the decorators from @microsoft/rayfin-core.Register them in rayfin/data/schema.ts.Configure auth in rayfin/rayfin.yaml — Entra ID for production.Sign in: npx rayfin loginDeploy: npx rayfin upWire your frontend to the generated GraphQL endpoint and ship.The Microsoft Learn quickstart walks through the same flow with the Todo sample. The Build 2026 lab repo has a field-services app you can clone and deploy end-to-end.Key TakeawaysRayfin is Fabric's application layer. It turns Fabric from an analytics-only platform into a full application platform with governed APIs, auth, and data models — provisioned by declaring a TypeScript schema.MIT-licensed and agent-first. The SDK is open source on GitHub under MIT and ships dedicated plugins for Claude Code, GitHub Copilot, Cursor, and Codex. Coding agents can scaffold and deploy a complete backend from a natural-language spec.Data gravity is the real value. Application data in OneLake means no ETL, no sync, and unified governance across analytics and applications. The data is in one place from day one.Public preview — pin versions. The API is stabilizing but not stable. Pin your @microsoft/rayfin-core version, budget time for migration changes, and don't go to production at scale before GA.Start with governed CRUD apps. Customer portals, internal tools, partner-facing apps, and data-driven line-of-business apps are the sweet spot. Heavy compute, real-time streaming, and polyglot persistence still belong on Functions, Event Hubs, and Cosmos — and Rayfin can call them.If you are an enterprise architect in Malaysia evaluating where to put the next internal app, my honest read is this: the question is no longer "Can we build it on Azure?" It is "Should the data live in OneLake and be served from Fabric, or should it live somewhere else?" For most of the apps I see on customer workshops — portals, ticketing, approval flows, partner integrations — the answer is now the former, and Rayfin is the path of least resistance to get there. --- ## Agentic AI at Scale — Why Data Foundations Are the Real Bottleneck URL: https://wenfeng.my/agentic-ai-at-scale-why-data-foundations-are-the-real-bottleneck/ Author: admin Published: 2026-08-29 06:30:12 Modified: 2026-10-04 15:30:54 Here's a statistic that should stop every AI initiative in its tracks: eight in ten companies cite data limitations as a roadblock to scaling agentic AI. Not model quality. Not compute availability. Not budget. Data.McKinsey's April 2026 article, Building the foundations for agentic AI at scale, makes the case clearly: nearly two-thirds of enterprises worldwide have experimented with agents, but fewer than 10 percent have scaled them to deliver tangible value. The gap between experimentation and production isn't about the AI — it's about the data infrastructure the AI needs to operate autonomously.If you're planning agent deployments for your enterprise — whether it's Microsoft Copilot Studio, Azure AI Foundry, custom multi-agent pipelines, or any combination — the data readiness question needs to come before the agent architecture question. This article walks through why, and gives you a concrete four-step approach with working Azure examples.The Scale Gap: Why Pilots FailThe pattern is consistent across organizations I've worked with:The pilot works beautifully. A single agent on a well-defined dataset delivers impressive results. Everyone is excited.Scaling breaks things. When the agent needs to access data across multiple systems, departments, or formats — the quality drops, the latency increases, and the governance gap becomes visible.Trust erodes. Inconsistent results from the agent — because it's pulling from fragmented data sources — make stakeholders pull back.The initiative stalls. Not because the AI failed, but because the data foundation couldn't support the scale.McKinsey puts it bluntly: while companies have often muscled through fragmented, siloed data in the past, those issues are impossible to manage at scale — and inconsistent governance only increases the challenge of preserving data context while enforcing access control, lineage, and auditability. Shaky data leads to agent breakdowns, inconsistent decisions, and lost coordination.The analogy the article opens with is the right one: a house is only as strong as its foundation. An agent is only as reliable as the data it can access.What Agent-Ready Data Actually RequiresTraditional data architecture was designed for human-driven analytics — batch ETL, data warehousing, BI dashboards. Agent-driven systems have fundamentally different data requirements:Real-Time Access, Not Batch ProcessingAgents make decisions in seconds or milliseconds. They can't wait for a nightly ETL refresh. They need access to current data across all relevant sources, with latency in the hundreds of milliseconds range.Semantic Consistency, Not Just Data AccessAn agent that reads "revenue" from a sales system and "revenue" from a finance system needs to know they mean the same thing — or at least understand the transformation between them. Without a shared semantic layer, agents make decisions on inconsistent interpretations. McKinsey is explicit: without this shared semantic foundation, agents may act on incomplete or conflicting interpretations of the same data, increasing error rates and operational risk as scale grows.Governed Access, Not Open AccessAgents need to access data autonomously, but that access must be governed. Row-level security, column-level masking, purpose-based access controls — these need to apply to agent data access the same way they apply to human access.Quality at Source, Not Quality After the FactTraditional data quality processes clean data after ingestion. Agent systems need quality checks at the point of consumption, because agents can't distinguish high-quality data from low-quality data without explicit signals.McKinsey's Seven Principles for Agent-Ready DataThe article distills seven architectural principles that distinguish agent-ready data platforms from traditional ones:Treat data ingestion like a product. All data — batch, real-time, structured, or unstructured — enters once and is usable by everyone.Share meaning, not just data. Common definitions through ontologies and knowledge graphs so analytics, AI models, and agents interpret data the same way.Use one data foundation for analytics and AI. Build data once and use it everywhere — reports, machine learning, gen AI — rather than running separate pipelines and platforms.Build trust into the platform by default. Security, access controls, privacy, and AI governance should be automatic, not added later or managed manually.Expose capabilities through stable interfaces. Clear APIs and model access points so teams can build without rework.Make behavior visible and measurable. Continuously track data quality, model performance, speed, and cost.Provide a controlled way to run AI agents. A shared execution layer that enforces enterprise rules and guardrails.These principles aren't theoretical. They map directly to architectural decisions you can make today.The Four-Step ApproachMcKinsey's prescription is four coordinated steps that link strategy, technology, and people.Step 1: Identify High-Impact Workflows to AgentifyDon't try to agentify everything. Start with a few high-value, end-to-end workflows where agent autonomy can unlock significant impact, prioritized by value potential, feasibility, and strategic fit.Practical starting points for Malaysian enterprises:Invoice processing — agent reads incoming invoices, extracts data, matches to POs, routes for approvalCustomer inquiry routing — agent classifies incoming queries, routes to the right team, provides contextCompliance monitoring — agent monitors regulatory changes, maps them to existing policies, flags gapsReport generation — agent pulls data from multiple sources, generates structured reports with narrativeFor each workflow, map the data sources the agent needs to access. If the agent needs data from 3+ systems with different formats, different access controls, and different update frequencies — that's your data readiness signal.Step 2: Modernize Each Layer of the Data ArchitectureThe article walks through the layers of the data stack that need modernization — and importantly, it says to modernize existing platforms rather than rebuild from scratch, building modular, evolutionary architectures:Data Source Layer: automate ingestion, quality checks, security, and lineage tracking directly into pipelines — not as one-time reviews.Data Platform Layer: connect systems and orchestrate access, synchronization, and real-time interaction. Include vector stores and embedding services for unstructured data, plus agent interoperability standards — MCP for structured context sharing, A2A for agent-to-agent coordination, and AP2 for trusted transactional interactions.Semantic Layer: codify business meaning through ontologies and knowledge graphs. This is the layer that prevents agents from acting on conflicting interpretations of the same term.Data Products: package curated data as reusable, business-ready assets with clear ownership, quality standards, semantics, and interfaces. Agents consume data products, not raw tables.Data Consumption Layer: deliver intelligence into workflows via data APIs, retrieval interfaces, and agentic orchestration services that dynamically assemble context rather than relying on predefined queries.Governance & Access Controls: use a medallion architecture that progressively curates data from raw to agent-ready form while preserving lineage and auditability, and an AI gateway that governs model access to unstructured data, enforces usage policies, and records how data is retrieved and used.Practical foundation in Azure: Fabric OneLake gives you the unified storage layer, with workspace-level isolation and RBAC you can deploy as code: // Fabric workspace for the agent-ready data foundation resource fabricWorkspace 'Microsoft.Fabric/workspaces@2023-11-01' = { name: 'ws-agent-data-foundation' location: 'southeastasia' properties: {} } // OneLake capacity assignment resource capacityAssignment 'Microsoft.Fabric/capacityAssignments@2023-11-01' existing = { name: 'ca-agent-foundation' } # Verify workspace capacity and permissions after deploy az fabric list workspaces --query "[?name=='ws-agent-data-foundation']" -o table # Grant the agent service principal read access to curated lakehouse schemas az role assignment create \ --assignee <agent-sp-object-id> \ --role "Reader" \ --scope /subscriptions/<sub>/resourceGroups/rg-data-foundation Step 3: Ensure Data Quality Is in PlaceData quality for agent systems is different from data quality for BI. McKinsey calls for a move from periodic cleanup to continuous, real-time quality management, covering structured data, unstructured data, and agent-generated outputs alike. Quality DimensionBI RequirementAgent Requirement FreshnessDaily/hourlyReal-time (seconds) Completeness>95%>99% (agents can't handle missing gracefully) ConsistencyAcross reportsAcross all sources simultaneously AccuracyHuman-verifiableAgent-verifiable (automated checks) LineageDocumentationAutomated, queryable by agents Unstructured data — documents, emails, tickets — must be held to the same standards: tagging, classification, vector embeddings, and graph-based structuring. And when agents generate data, the same quality, lineage, and reconciliation standards apply to their outputs.A minimal automated quality gate in Python that an ingestion pipeline can run before data reaches the agent-ready layer: from datetime import datetime, timedelta, timezone import pandas as pd def quality_gate(df: pd.DataFrame) -> dict: """Continuous quality checks before data reaches agent consumption.""" now = datetime.now(timezone.utc) checks = { "completeness": df["order_id"].notna().mean(), "freshness_hours": (now - df["updated_at"].max()).total_seconds() / 3600, "duplicate_rate": df.duplicated(subset=["order_id"]).mean(), } checks["pass"] = ( checks["completeness"] >= 0.99 and checks["freshness_hours"] <= 1.0 and checks["duplicate_rate"] <= 0.001 ) return checks # Fail closed: block the pipeline if the gate fails result = quality_gate(df) if not result["pass"]: raise RuntimeError(f"Quality gate failed: {result}") Step 4: Build an Operating and Governance Model for Agentic AIThis is the most underappreciated step. McKinsey is clear that scaling requires an organizational reboot, not just a technology one: human roles shift from execution to supervision and orchestration of agent-driven workflows. In a hybrid human-agent work environment, clear governance is essential for agents to operate transparently and safely at scale.What this means practically:Define what agents can do, which data they can access, and when human approval is requiredImplement audit trails for every agent data accessCreate escalation paths for agent decisions above a defined risk thresholdMonitor agent behavior for drift — are agents accessing data they shouldn't?Governance architecture: Agent Decision │ ▼ ┌──────────────┐ ┌──────────────────┐ │ Policy Engine │────▶│ Human Approval │ │ (risk check) │ │ (if needed) │ └──────┬───────┘ └──────────────────┘ │ ▼ ┌──────────────┐ │ Audit Log │ │ (every │ │ access) │ └──────────────┘ For enterprises running agents through Azure, the audit trail piece is configuration, not custom code — route agent identity access through managed identities and capture sign-in logs: # Verify agent service principal sign-in activity (audit trail) az rest --method get --url \ "https://graph.microsoft.com/v1.0/auditLogs/signIns?\$filter=appId eq '<agent-app-id>'&\$top=50" \ --query "value[].{time:createdDateTime,ip:ipAddress,status:status.errorCode}" -o table Connecting This to Malaysian Enterprise RealityMost Malaysian enterprises I work with have data scattered across:On-premises SQL Server databases (often running legacy LOB applications)SaaS applications (Salesforce, Dynamics 365, local banking systems)Cloud storage (Azure Blob, SharePoint, OneDrive)Spreadsheets (the universal data store that nobody wants to admit exists)The gap between this reality and agent-ready data is significant. But it's not insurmountable — it requires prioritized investment in data infrastructure before agent deployment.Recommended approach:Start with data inventory — map all data sources, formats, access controls, and update frequenciesBuild the semantic layer — even a simple business glossary shared across systems dramatically improves agent reliabilityPilot with controlled data scope — start agents with access to 1-2 well-governed data sources, expand graduallyInvest in Fabric OneLake — as the unified data foundation, it reduces the integration complexity that makes agent deployment brittleImplement governance before scaling — the pilot-to-production gap is almost always a governance gapPitfalls to AvoidFrom the research and from deployments I've watched closely:Skipping the semantic layer. Teams jump straight to RAG pipelines over raw tables. The agents work in demos, then make contradictory decisions in production because "revenue" means different things in different systems. Build the shared definitions first — even a minimal ontology pays for itself.Treating agent access like service account access. Reusing a shared credential for all agents destroys auditability. Every agent workload needs its own managed identity with scoped permissions, so you can trace which agent accessed what and when.Bolting governance on after the pilot. Retrofitting row-level security, lineage, and audit logging onto a working pilot is 5-10x more expensive than designing it in from day one. The pilot that "just works" with open access becomes the production blocker.Ignoring agent-generated data. Agents that write data back — updating records, creating tickets, generating reports — produce data that must flow through the same governed interfaces as human-generated data. Uncontrolled agent writes are a silent corruption path.Rebuilding instead of modernizing. McKinsey warns against the temptation to rebuild everything from scratch. The strongest organizations build modular, evolutionary architectures where components can be replaced as new technologies emerge. Your existing SQL Server estate is not the enemy — ungoverned access to it is.Key TakeawaysData is the bottleneck, not AI capability. Eight in ten companies cite data limitations as a roadblock to scaling agentic AI, and fewer than 10 percent of enterprises have scaled agents to deliver tangible value. Fix your data foundations before investing in agent architecture.Agent-ready data has different requirements than BI-ready data. Real-time access, semantic consistency, governed autonomous access, and quality at source are non-negotiable for agent systems.Follow the four steps in order: identify high-impact workflows, modernize the data architecture layers, enforce continuous data quality, and build the operating and governance model.Govern before you scale. The pilot-to-production gap is a governance gap. Every agent needs its own identity, audit trail, and escalation path before you expand data access beyond a controlled pilot scope.Modernize, don't rebuild. Modular, evolutionary architectures beat greenfield rewrites — especially for Malaysian enterprises with significant on-prem and SaaS estates.Source: McKinsey, "Building the foundations for agentic AI at scale," April 2, 2026. Related reading: Microsoft Fabric Data Agents — Building Agentic AI on Your Enterprise Data, AI Agent Governance in Enterprise, and FinOps for AI Agents. --- ## SQL Server 2025: The AI-Ready Database With Vector Search, Copilot, and Fabric Mirroring URL: https://wenfeng.my/sql-server-2025-ai-ready-database-vector-search-copilot-fabric/ Author: admin Published: 2026-08-29 02:00:13 Modified: 2026-10-04 15:25:26 By Law Wen Feng, Principal Solution ArchitectEvery enterprise AI project I've architected in the last three years hit the same wall: the data stack sprawl. Your transactional data lives in SQL Server, your embeddings live in a vector database, your analytics live in a lake, and your developers need three connection strings, three security models, and three backup strategies to build one chatbot.SQL Server 2025 is Microsoft's answer to that sprawl. It's not a single headline feature — it's three distinct moves that together change what a SQL Server instance can do: native vector search built into the database engine, GitHub Copilot inside SQL Server Management Studio, and Fabric Mirroring that streams on-premises SQL Server data continuously into Microsoft Fabric without ETL pipelines.I've spent the last few weeks digging into the Microsoft Learn documentation and testing what's real versus what's marketing. Here's the fact-checked picture, with working code.The Problem: AI Data Stack SprawlThe standard RAG architecture that every vendor demo shows you looks simple: chunk your documents, embed them, store the vectors, retrieve the top matches, feed them to a language model.What the demo doesn't show is the operational reality. In practice you end up with:A relational database for the transactional data your business runs on.A dedicated vector store for embeddings, with its own access control, monitoring, and DR plan.A synchronization job that keeps the two consistent — and pages you at 2 AM when it drifts.An ETL pipeline if the analytics team wants any of it in the lake.For a 10-person startup, that's manageable. For the enterprises I work with across Malaysia, Singapore, and the broader ASEAN region — most running lean database teams — every additional data platform is a real operational tax. This is the gap SQL Server 2025 attacks directly.What SQL Server 2025 Actually ChangesThree capabilities matter for AI workloads:Native vector search. A VECTOR data type, DiskANN-based approximate nearest neighbor indexes, and T-SQL functions for similarity search — all inside the database engine. No extension, no sidecar service.GitHub Copilot in SSMS. Natural-language T-SQL assistance, code completions, and now an autonomous Agent mode, directly in SQL Server Management Studio 22.Fabric Mirroring for SQL Server. Continuous, near-real-time replication of your SQL Server databases into Microsoft Fabric's OneLake as open Delta tables — including, for the first time, from on-premises SQL Server 2025 instances.Let me walk through each one.Native Vector Search: Semantic Queries in Plain T-SQLThe headline feature. SQL Server 2025 stores embeddings in a native VECTOR column and searches them with familiar T-SQL. If your team can write a JOIN, they can write a vector search.The Core WorkflowFirst, register your embedding endpoint as an external model: CREATE EXTERNAL MODEL EmbeddingModel WITH ( LOCATION = 'https://aoai-wenfeng-poc.openai.azure.com/openai/deployments/text-embedding-3-small/embeddings?api-version=2023-05-15', API_FORMAT = 'Azure OpenAI', MODEL_TYPE = EMBEDDINGS, MODEL = 'text-embedding-3-small', CREDENTIAL = AzureOpenAICredential ); The API_FORMAT accepts Azure OpenAI, OpenAI, Ollama, or ONNX Runtime — so local, air-gapped embedding models are also supported via Ollama or ONNX. That matters for regulated industries in this region that can't send text to a cloud endpoint.Next, a table with a vector column. Note that the dimension must match your model's output — 1536 for text-embedding-3-small and text-embedding-ada-002: CREATE TABLE KnowledgeBase ( ArticleID INT IDENTITY PRIMARY KEY, Title NVARCHAR(500) NOT NULL, Content NVARCHAR(MAX) NOT NULL, Category NVARCHAR(100), Embedding VECTOR(1536) ); Then the DiskANN index: CREATE VECTOR INDEX idx_KB_Embedding ON KnowledgeBase (Embedding) WITH (METRIC = 'cosine', TYPE = 'DiskANN', MAXDOP = 4); Microsoft chose DiskANN deliberately: it's an SSD-friendly approximate nearest neighbor algorithm, so you get billion-scale-class indexing behavior without keeping the entire index in RAM. The supported distance metrics are cosine, euclidean, and dot.Finally, the semantic query itself: DECLARE @qv VECTOR(1536) = AI_GENERATE_EMBEDDINGS(N'I forgot my login credentials' USE MODEL = EmbeddingModel); SELECT TOP (5) k.ArticleID, k.Title, k.Category, v.distance FROM VECTOR_SEARCH( TABLE = KnowledgeBase AS k, COLUMN = Embedding, SIMILAR_TO = @qv, METRIC = 'cosine', TOP_N = 5 ) AS v ORDER BY v.distance; One syntax note: TOP_N is the parameter for the earlier-version vector indexes that SQL Server 2025 ships. The newer SELECT TOP (N) WITH APPROXIMATE form applies to the latest vector index version, which at the time of writing is available only in Azure SQL Database and SQL database in Microsoft Fabric. On SQL Server 2025, use VECTOR_SEARCH with TOP_N.Exact Search Is Sometimes the Right AnswerYou don't always need an ANN index. For exact k-nearest-neighbor search, VECTOR_DISTANCE() scans without an index: SELECT TOP (10) ArticleID, Title, VECTOR_DISTANCE('cosine', @qv, Embedding) AS distance FROM KnowledgeBase ORDER BY distance; Perfect recall, simpler operations. For catalogs and knowledge bases under roughly 50,000 vectors, exact search is often fast enough — don't over-engineer.Generating Embeddings Without Leaving the DatabaseThe AI_GENERATE_EMBEDDINGS() function above deserves its own section, because it's quietly one of the most useful additions. SQL Server calls your embedding endpoint directly from T-SQL — on insert, in a query, or in a batch job. No application round-trip.Three prerequisites that the documentation buries and every team misses: -- 1. Vector features are preview in SQL Server 2025; enable them per database ALTER DATABASE SCOPED CONFIGURATION SET PREVIEW_FEATURES = ON; -- 2. Allow the engine to call external REST endpoints (server level) EXECUTE sp_configure 'external rest endpoint enabled', 1; RECONFIGURE WITH OVERRIDE; -- 3. Grant users the right to execute the model GRANT EXECUTE ON EXTERNAL MODEL::EmbeddingModel TO [AppRole]; The external model authenticates through a DATABASE SCOPED CREDENTIAL holding your Azure OpenAI API key. On Azure SQL Database and SQL database in Fabric the REST endpoint option is on by default; on SQL Server and SQL Managed Instance you must enable it explicitly.The alternative pattern — and the one most production applications use — is generating embeddings in the application layer and inserting them as a JSON cast: import json import os import pyodbc from openai import AzureOpenAI client = AzureOpenAI( azure_endpoint="https://aoai-wenfeng-poc.openai.azure.com", api_version="2023-05-15", api_key=os.environ["AZURE_OPENAI_KEY"], ) resp = client.embeddings.create( model="text-embedding-3-small", input="How do I reset my password?", ) vec = json.dumps(resp.data[0].embedding) conn = pyodbc.connect(conn_str) cur = conn.cursor() cur.execute( "INSERT INTO KnowledgeBase (Title, Content, Embedding) " "VALUES (?, ?, CAST(? AS VECTOR(1536)))", ("Reset password", "Go to Settings > Security...", vec), ) conn.commit() Use in-database embedding for bulk backfills and pure-SQL pipelines; use application-side embedding when you need control, batching, and retry logic. Both land in the same VECTOR column.GitHub Copilot in SSMS: Natural Language Meets Your SchemaThe Copilot story has evolved, so let me state the current situation precisely. The AI assistant in SSMS is now GitHub Copilot in SQL Server Management Studio, and it requires SSMS 22 or later. You sign in with a GitHub account that has Copilot access — or sign up for Copilot Free directly from the SSMS badge in the top-right corner.What it actually does:Chat and inline chat — ask questions about your database in plain English, get T-SQL back. "Show me all orders from the last 30 days for customers in Kuala Lumpur" becomes a working query against your real schema.Code completions — inline suggestions in the query editor, available from SSMS 22.2.Slash commands — /doc, /explain, /fix, /optimize for documenting and repairing T-SQL.Database instructions — you can store business rules and context in the database itself so Copilot generates more accurate queries.Agent mode (preview) — from SSMS 22.7, give Copilot a high-level goal and it works through it autonomously: executing queries, reading execution plans, and modifying schema with your approval, extensible via MCP servers.The security posture is the part that matters for my enterprise clients: Copilot executes queries under your login's permissions — if you can't SELECT from Sales.Orders, neither can Copilot. And per Microsoft's documentation, prompts, responses, and system metadata are not retained, and your data is not used to train models.Fabric Mirroring: Your On-Prem Data in OneLake, Without ETLThis is the feature in the title, and arguably the most strategically important of the three. Mirroring in Fabric continuously replicates SQL Server databases into OneLake as open Delta/Parquet tables, with near-real-time latency. Inside Fabric, each mirrored database gets an auto-generated SQL analytics endpoint — queryable with T-SQL from SSMS, VS Code, or Power BI.Why this matters: Synapse Link for SQL Server is discontinued in SQL Server 2025. Mirroring in Fabric is the replacement path, and it's a genuinely better one — no ETL pipeline, no staging database, and the data lands in an open format.The SQL Server 2025 specifics you need to know, straight from the documentation:SQL Server 2025 mirroring is supported for on-premises instances. It is currently not supported for SQL Server 2025 running in Azure Virtual Machines, and not on Linux.It requires Azure Arc onboarding with the Azure Extension for SQL Server.SQL Server 2025 uses the Fabric mirroring change feed feature — not classic Change Data Capture. Older versions (SQL Server 2016–2022, Standard/Enterprise/Developer editions) mirror via CDC.SQL Server 2025 adds resource governor workload groups to control the CPU and I/O that mirroring consumes, plus automatic reseed configuration to prevent unbounded growth of the change feed.The architecture this unlocks is the one I now recommend for clients with a large on-premises SQL estate: ┌────────────────────────────────────────────┐ │ Apps & APIs (OLTP, vectors, JSON) │ ├────────────────────────────────────────────┤ │ SQL Server 2025 (on-premises, Arc-joined)│ │ VECTOR columns + DiskANN + Copilot │ ├────────────────────────────────────────────┤ │ Fabric Mirroring (change feed, Arc) │ ├────────────────────────────────────────────┤ │ Microsoft Fabric OneLake (Delta tables) │ │ SQL analytics endpoint + Power BI │ └────────────────────────────────────────────┘ Operational data stays in SQL Server 2025 where your AI application serves it; analytics and BI read continuously refreshed Delta copies in Fabric. No nightly ETL window, no "the dashboard is stale" conversations.Pitfalls I See Teams HitFrom testing and early client engagements, these are the traps:Forgetting `PREVIEW_FEATURES`. Vector search in SQL Server 2025 is a preview feature. Without ALTER DATABASE SCOPED CONFIGURATION SET PREVIEW_FEATURES = ON on the target database, vector operations fail — and Microsoft explicitly cautions that preview features aren't recommended for production. Plan your GA timing accordingly.Dimension mismatch. VECTOR(1536) must match your embedding model's output. Switching from text-embedding-ada-002 to text-embedding-3-large (3072 dimensions) means a new column and a full re-embedding of existing data. Embeddings are model-specific: you cannot mix them.Metric mismatch between index and query. An ANN index is only used if the query's distance metric matches the index's metric. If you build the index with cosine and query with euclidean, SQL Server silently falls back to exact kNN and just warns you.Assuming mirroring works from Azure VMs. For SQL Server 2025, Fabric Mirroring currently supports on-premises instances only. If your SQL Server runs in an Azure VM, mirror via the Azure SQL Database or Managed Instance paths instead — or wait.Treating Copilot as a security boundary. Copilot respects your login's permissions, which is exactly as safe as your permission model. If your developers connect with db_owner or sysadmin accounts — as too many do — Copilot inherits that reach. Use least-privilege logins before handing anyone an AI assistant.Re-embedding costs. Every content update invalidates its embedding. Budget the Azure OpenAI token cost of re-embedding into your change workflow, especially for large knowledge bases that update frequently.When You Still Need a Dedicated Vector DatabaseHonest guidance, because SQL Server vector search is not always the answer: Choose SQL Server 2025Choose a dedicated vector store Vectors must JOIN with relational data (inventory, pricing, permissions)Billion-vector scale or thousands of similarity queries per second Dataset under a few million vectorsMultimodal workloads (image, audio, video embeddings) You want one backup, one security model, one connection stringVectors are fully independent of transactional data ACID transactions around vector writesYou need specialized hybrid-search tuning For the use cases I see most in this region — internal knowledge bases, customer support RAG, product catalog search — SQL Server 2025 is simpler, cheaper, and operationally leaner than bolting on a second database.Key TakeawaysVector search is native T-SQL. The VECTOR type, DiskANN indexes, VECTOR_SEARCH(), and VECTOR_DISTANCE() bring semantic search into the database engine — but it's a preview feature gated behind PREVIEW_FEATURES = ON, so treat production timing carefully.Embeddings can be generated inside the engine. AI_GENERATE_EMBEDDINGS() plus an external model (Azure OpenAI, OpenAI, Ollama, or ONNX) keeps the whole RAG pipeline in one database — or generate embeddings in your app and CAST the JSON into a vector column.Copilot in SSMS is GitHub Copilot. It requires SSMS 22+, signs in with a GitHub account, executes queries under your permissions, and now includes completions, database instructions, and a preview Agent mode.Fabric Mirroring replaces Synapse Link — and for the first time it reaches on-premises SQL Server 2025 via Azure Arc and the change feed, streaming your operational data into OneLake as open Delta tables without ETL.The strategic play is consolidation. One engine for transactional data, vectors, and JSON; one continuous mirror into Fabric for analytics. Before you buy a second database for AI, prove you actually need it. --- ## Azure Malaysia West: One Year Later — Lessons, Gaps, and the Road to Southeast Asia 3 URL: https://wenfeng.my/azure-malaysia-west-one-year-later-lessons-gaps-southeast-asia-3/ Author: admin Published: 2026-08-28 23:30:04 Modified: 2026-10-04 15:25:37 It has been just over a year since Microsoft flipped the switch on the Malaysia West cloud region. For those of us who have been tracking Azure's expansion across Southeast Asia, this was a watershed moment — Microsoft's first hyperscale datacenter region on Malaysian soil, strategically located in Greater Kuala Lumpur, built with three Availability Zones, and backed by a US$2.2 billion investment in Malaysia's cloud and AI infrastructure.I have spent the past twelve months working with enterprise teams across Malaysia — banks, telcos, oil and gas majors, government agencies — all wrestling with the same fundamental question: What can we actually run in Malaysia West today, and what still has to route elsewhere?This article is my honest assessment of where things stand. Some of what I found surprised me — including one major AI service gap that almost every draft architecture I review gets wrong. Here is what a year of hands-on work has taught me, and how Malaysian enterprises should be thinking about their Azure strategy as the Johor Bahru region — Southeast Asia 3 — moves from announcement to reality.The Malaysia West Region: What Is Actually LiveWhen Malaysia West went generally available in May 2025, Microsoft positioned it as an AI-ready, hyperscale region with full in-country data residency. That much is true. But "full" is a word that deserves scrutiny, and marketing announcements are not architecture inputs. So instead of trusting the launch press release, I verified service availability directly against the Azure resource provider registry. Here is the picture as of this writing.Verified Available in Malaysia WestThe region qualifies as a "recommended" Azure region — three physically independent Availability Zones with low inter-zone latency. The core platform is genuinely broad:Compute & Application PlatformAzure Virtual Machines (broad SKU support)Azure Kubernetes Service (AKS)Azure App Service and Azure FunctionsAzure Container Apps and Azure Container InstancesData & StorageAzure SQL DatabaseAzure Database for PostgreSQL Flexible ServerAzure Database for MySQL Flexible ServerAzure Cosmos DBAzure Cache for Redis (classic and Enterprise tiers)Azure Blob Storage and Azure FilesAzure NetApp FilesManaged DisksNetworkingVirtual Networks, Azure Firewall, Application GatewayExpressRoute and VPN GatewayAzure BastionDDoS Network Protection (plan creation in-region)AI & Data PlatformAzure Machine LearningAzure AI SearchEvent Hubs and Service BusLogic Apps and API ManagementSecurity & IdentityMicrosoft Entra ID (global service; Malaysian tenants can anchor data in Malaysia West)Azure Key VaultLog Analytics workspaces — the foundation Microsoft Sentinel runs onRecovery Services vaults (backup and Site Recovery)This is a solid foundation. If you are running a standard three-tier web application with a SQL backend, containerized microservices on AKS, or machine learning inference on Azure ML, you can deploy it entirely within Malaysian borders today.What Still Routes to Singapore — The Part Most People Get WrongHere is the finding that matters most, and it contradicts what I see in many otherwise-careful architecture documents: Azure OpenAI Service is not available in Malaysia West. The Cognitive Services resource provider — the one Azure OpenAI is built on — does not list Malaysia West among its regions. Any workload that needs GPT models, embeddings, or Azure OpenAI endpoints must route to Southeast Asia (Singapore) or another supported region.In 2026, that is a significant gap. Most AI-enabled application architectures assume Azure OpenAI is available in the primary region. If you are subject to in-country data residency requirements — and in Malaysia, financial services and government workloads increasingly are — this one gap can force a redesign.Other verified gaps in Malaysia West:Azure AI Bot Service — not available in the region; the provider registry lists only Global, West Europe, West US, and Central IndiaWorth noting in the other direction: services some teams assume are missing — Azure Cache for Redis and DDoS Network Protection among them — are in fact listed for Malaysia West in the provider registry. Verify rather than assume, in both directions; availability data, not tribal knowledge, should drive your region decision.The practical implication: if you are building a greenfield application, audit every service dependency against the Azure products-by-region data before you write your Bicep templates. Do not assume availability just because the region exists. You can run this audit yourself in minutes: # Check whether a specific resource type lists Malaysia West as an available location az provider show --namespace Microsoft.CognitiveServices \ --query "resourceTypes[?resourceType=='accounts'].locations" -o json # Or list everything your subscription has registered az provider list --query "[?registrationState=='Registered'].namespace" -o table Latency Benchmarks: What the Numbers Actually Tell YouLatency is the number that matters most for architecture decisions. Microsoft publishes P50 round-trip latency statistics between Azure regions. Here are the official numbers from Malaysia West as published by Microsoft — not estimates:Malaysia West Inter-Region Latency (Official P50 Round-Trip) Destination RegionRound-Trip Latency Southeast Asia (Singapore)~9 ms Indonesia Central (Jakarta)~21 ms East Asia (Hong Kong)~35 ms South India (Chennai)~41 ms Central India (Pune)~59 ms Korea Central (Seoul)~69 ms Japan East (Tokyo)~77 ms Australia East (Sydney)~100 ms West Europe (Amsterdam)~166 ms East US (Virginia)~229 ms Within Malaysia West PathLatency Zone-to-Zone (same region)~1–2 ms The Malaysia West to Singapore number — roughly 9 ms round-trip on the Azure backbone — is excellent. For most workloads, this means:Asynchronous database replication to Singapore works well for DR purposesActive-passive failover architectures are straightforward to implementActive-active is feasible but requires careful consistency design — the round-trip is small, but application-level eventual consistency must still be engineered in deliberatelySynchronous writes to Singapore add a real, measurable cost to every transaction — not recommended for latency-sensitive pathsOne critical point: these are Azure backbone numbers. Your actual application latency depends on the full path — client to Azure edge, edge to region, inter-service calls within the region, and any internet egress. I recommend every team run their own measurements before committing to an architecture: # Deploy a small test VM in each region, then measure real connectivity az network watcher test-connectivity \ -g rg-mw-test --source-resource vm-mw-test \ --dest-resource vm-sea-test --dest-port 1433 The Johor Bahru Factor: Southeast Asia 3 Changes the CalculusOn 4 November 2025, Microsoft announced Southeast Asia 3 — a second Azure cloud region to be built in Johor Bahru, Malaysia. This is not just another datacenter announcement. It fundamentally changes the strategic picture for Malaysian enterprises.What We KnowLocation: Johor Bahru, connected across the Causeway to Singapore's ecosystem and to Microsoft's global Azure backbone — more than 600,000 km of subsea and terrestrial fibreFocus: AI transformation and advanced workloads; the region will add to Azure's 70+ global regionsTimeline: No confirmed GA date yet; construction and community programs are underwaySustainability: Backup generators powered by renewable biofuel, and a closed-loop chip-level cooling design that achieves zero-water evaporation — eliminating the need for a continuous fresh water supplyCommunity investment: The Skills2Work Datacenter Foundations Program, run with The Asia Foundation and Universiti Teknologi Malaysia, will equip 150 recent graduates with hands-on datacenter training; a separate UNDP Malaysia initiative is training 200 trainers from 20 social enterprises to reach approximately 10,000 community members; and Microsoft is partnering with Belia Prihatin and WWF Malaysia on a three-year mangrove restoration project in JohorWhy This Matters for Architecture Planning1. The Paired Region Problem, Eventually SolvedToday, Malaysia West has no formal Azure paired region — Microsoft's own reliability documentation lists it under non-paired regions. Southeast Asia (Singapore) is the de facto secondary, but it is not a paired region in Azure's platform sense — meaning you do not get automatic staggered updates or paired-region features for services like geo-redundant storage.When Southeast Asia 3 goes live, an in-country pairing becomes realistic. That unlocks:Formal DR targets with platform-level supportStaggered platform updates across the pairGeo-redundant storage patterns that never leave Malaysian borders2. Cross-Border Data Residency ImplicationsMalaysia's PDPA amendments — now in force with mandatory breach notification and Data Protection Officer obligations — tighten cross-border transfer rules. While the PDPA does not mandate data localization for all data types, sector regulators, particularly Bank Negara Malaysia for financial services, lean heavily toward in-country storage.Having two Malaysian regions means:DR within Malaysian borders, without cross-border transfer analysisHigher availability for in-country workloadsA future option for active-active entirely within Malaysia3. The JB–Singapore CorridorJohor Bahru's physical proximity to Singapore creates a unique architectural opportunity: workloads distributed across Malaysia West (Kuala Lumpur), Southeast Asia 3 (Johor Bahru), and Southeast Asia (Singapore) — three regions with latencies low enough for most distributed-systems patterns.Practical Recommendations for Malaysian EnterprisesBased on a year of working with teams deploying to Malaysia West, here is my honest guidance.1. Start with a Region AuditBefore migrating anything, map every Azure service your current architecture uses against Malaysia West's actual availability. If any dependency is missing — and if your stack includes Azure OpenAI, it is — decide now whether to wait, redesign, or accept the Singapore fallback with its latency and compliance implications.2. Design for the Paired Region That Does Not Exist YetMalaysia West has no formal paired region today. Your DR strategy must be explicitly configured against Southeast Asia (Singapore): # Create a Recovery Services vault in Southeast Asia as the DR target az backup vault create \ --resource-group rg-dr-backup \ --name rsv-sea-dr \ --location southeastasia Azure-to-Azure replication through Site Recovery runs against that same vault. The CLI path exists (az site-recovery protected-item create) but requires fabric, protection container, and policy IDs, so most teams enable replication through the portal, Bicep, or Terraform instead. The important point is not the tooling: do not wait for Southeast Asia 3 to design your DR. Build it now with Singapore, and plan to migrate the DR target to Johor Bahru when it becomes available.3. Treat Latency as a First-Class Architecture ConstraintRun your own benchmarks. The published numbers are backbone-to-backbone; your real-world latency includes the client path, TLS negotiation, and application-layer processing. If your application requires single-digit-millisecond database response times and you are tempted by cross-region writes, redesign instead: in-region reads, asynchronous replication, and event-driven synchronization.4. Plan for Southeast Asia 3 NowThe Johor Bahru region announcement is a planning input, not just news. Start documenting which workloads you would move from Singapore DR to in-country DR, and what that change enables — for regulated workloads, keeping all data, primary and DR, within Malaysian borders will likely become a regulatory expectation, not a nice-to-have.5. Engage Microsoft Early on Service RoadmapFor services not yet in Malaysia West — Azure OpenAI being the big one — engage your Microsoft account team or solution architect. Regional service rollouts are demand-driven, and customer architecture requirements are genuine leverage. If enough Malaysian enterprises signal that Azure OpenAI in Malaysia West is the gating factor for their AI workloads, that changes the roadmap conversation.Cost ConsiderationsMalaysia West pricing is generally competitive with Southeast Asia (Singapore), with a few nuances:Compute: VM pricing roughly comparable to Southeast Asia, with occasional promotional pricing for new-region adoptionStorage: Standard and Premium tiers aligned with Southeast AsiaCross-region egress: Data transfer from Malaysia West to other regions incurs Azure’s standard inter-region bandwidth charges. As of August 2026, intra-continental APAC inter-region transfer (e.g. Malaysia West ↔ Singapore, Jakarta, Hong Kong, Korea, Japan) is $0.02 per GB; inter-continental (e.g. Malaysia West ↔ Europe or the US) is $0.05 per GB. Internet egress to destinations outside Azure is metered separately and starts at $0.087/GB after the first 100 GB free. Verify the current rates in the Azure Pricing Calculator before you budget a DR architecture that replicates terabytes daily.Reserved Instances: Available in Malaysia West. Savings depend on the service and term — for Azure VMs, 1-year reservations typically save around 30–40% and 3-year reservations up to ~65% versus pay-as-you-go. Across the full Azure reservation portfolio, published savings range from about 11% to 72% depending on the service. For Azure SQL and Azure Cache for Redis, the upper end is closer to 30–55%. Use the Azure Pricing Calculator for an exact figure on your specific SKU and region. # List VM sizes actually available in Malaysia West # Note: `az vm list-sizes` is deprecated in favor of `az vm list-skus`. # The query projection below targets the older list-sizes output schema. az vm list-sizes --location malaysiawest \ --query "[].{name:name, vCPUs:numberOfCores, memoryMB:memoryInMb}" -o table My recommendation: start with pay-as-you-go to validate architecture and service availability, then commit to reservations once performance and gaps are confirmed.Key TakeawaysMalaysia West is production-ready for most workloads — compute, databases, storage, networking, Azure ML, and AI Search are all in-region. Verify every dependency against the products-by-region data before committing; the gaps are real and they matter.The big gap is Azure OpenAI. The Cognitive Services provider does not list Malaysia West. Any AI workload built on Azure OpenAI routes to Singapore today — plan for it explicitly, and press Microsoft on the roadmap if in-country AI matters to you.Malaysia West to Singapore latency (~9 ms official P50 round-trip) is excellent for async DR, but design for in-region reads and asynchronous replication rather than synchronous cross-region writes.Malaysia West has no formal paired region. Singapore is the practical DR target, but you must configure replication and failover explicitly. Do not rely on platform-level paired-region features.Southeast Asia 3 (Johor Bahru) changes everything when it lands — in-country DR, a potential paired region, and a KL–JB–Singapore corridor for distributed architectures. Start planning for it now, even though GA is not yet dated.Law Wen Feng is a Principal Solution Architect based in Malaysia, specializing in Azure infrastructure, cloud strategy, and AI-first architecture for Southeast Asian enterprises. Connect at [wenfeng.my/](https://wenfeng.my/). --- ## Azure API Management as AI Gateway: MCP, A2A, Agent Traffic URL: https://wenfeng.my/azure-api-management-as-ai-gateway-governing-mcp-a2a-and-agent-traffic-at-build-2026/ Author: admin Published: 2026-08-28 06:30:01 Modified: 2026-10-04 15:34:02 If you run multi-agent systems in production, every agent interaction is ultimately API traffic. Every model call, every MCP tool invocation, every agent-to-agent handoff crosses the network as a request that someone needs to authenticate, throttle, log, and pay for. I covered the fundamentals of this earlier — access control, throttling, and observability for LLM endpoints — in my first article on Azure API Management as AI Gateway.Since Build 2026, Microsoft has extended that story significantly. Azure API Management (APIM) and Azure API Center now reach well beyond language model endpoints: the gateway can import and govern remote MCP servers, A2A agent APIs, and a unified model API that fronts multiple model providers behind a single OpenAI-compatible endpoint. In other words, APIM is evolving from an LLM gateway into the governance plane for the entire agent stack.This article walks through what actually shipped, what it looks like in code, and where the sharp edges are.The Problem: Fragmented Governance for Agent TrafficA production multi-agent system typically fans out across four kinds of endpoints:LLM provider APIs — Azure OpenAI or Microsoft Foundry, Anthropic Claude, Google Vertex AI, each with its own request schema, rate limits, and token accountingMCP tool servers — the databases, SaaS tools, and internal services exposed to agents via the Model Context ProtocolA2A agent APIs — agent-to-agent communication, typically JSON-RPC-based, between collaborating agentsTraditional internal APIs — the REST services your agents still call underneath everythingIn most environments I review, each of these is governed separately — or not governed at all. Rate limits live in application code. Logging is whatever the agent framework emits. Token spend is reconstructed from invoices at month-end. When a runaway agent loop burns through a model quota, or a prompt injection attempt reaches a tool server, you find out after the fact.The architectural answer is simple to state and hard to build: put one control plane in front of all of it. That is exactly where APIM is heading.What Actually ShippedThe AI gateway in Azure API Management is not a separate SKU — it is a set of capabilities documented as applying across API Management tiers, with some features limited to the v2 tiers. Here is what matters for agent workloads.1. Multi-provider model governanceThe gateway can manage language model APIs conforming to three schemas:OpenAI Chat Completions and Responses APIAnthropic Messages API — currently supported in the APIM v2 tiersGoogle Vertex AI APIModels can live in Microsoft Foundry or with non-Microsoft providers such as Amazon Bedrock. You import each endpoint once and apply the same policies — throttling, caching, content safety, logging — regardless of provider.2. Unified model API (preview)This is the most architecturally significant addition. The unified model API exposes multiple model backends through a single OpenAI-compatible endpoint, handles format translation automatically, and lets you apply governance policies once across all models. Your agents speak one API shape; APIM decides which backend answers, and you can route and fail over across providers without touching application code. import os from openai import OpenAI # One client, one API shape — APIM decides which backend answers client = OpenAI( base_url="https://apim-ai-gw.azure-api.net/llm/v1", api_key=os.environ["APIM_SUBSCRIPTION_KEY"], ) response = client.chat.completions.create( model="gpt", # client-facing alias; APIM maps it to the configured backend messages=[{"role": "user", "content": "Draft the incident summary"}], ) print(response.usage.total_tokens) Swapping a model, rebalancing between a PTU deployment and pay-as-you-go, or failing over to another provider becomes a gateway configuration change, not a code change. Because it is in preview, I would not build a contractual production dependency on its exact semantics yet — but it is clearly the direction of travel, and worth piloting now.3. MCP server governanceAPIM can now expose existing REST APIs as MCP servers and pass through to existing remote MCP servers. On the inventory side, Azure API Center maintains a registry of remote and local MCP servers — title, description, use cases, runtime URLs — and syncs APIs and MCP servers between APIM and API Center. Registered MCP servers can also be integrated with Microsoft Foundry's tool catalogs, so governed tools become discoverable to Foundry agents.This matters because MCP sprawl is real. Without a registry, every team wires its agents to whichever MCP servers it happens to know about, and nobody can answer "which tools can our agents reach, and who approved them?"4. A2A agent API governanceAPIM can import A2A agent APIs, bringing agent-to-agent traffic under the same gateway discipline as everything else: authentication, throttling, logging, monitoring. If your architecture has three or more agents collaborating over A2A, this is where you get an audit trail of inter-agent calls instead of a black box.5. Token controls and semantic cachingTwo policy families carry the economics:`llm-token-limit` enforces tokens-per-minute or token quotas per counter key (subscription, IP, or any expression), with optional prompt-token precalculation on the APIM side so over-limit prompts never reach the backend.`llm-semantic-cache-store` / `llm-semantic-cache-lookup` cache completions by vector similarity using the Embeddings API, backed by Azure Managed Redis or any RediSearch-compatible cache. Workloads with repetitive prompts see real reductions in backend token consumption and latency.6. Token observability and message loggingThe gateway emits token metrics with custom dimensions via llm-emit-token-metric, streams prompts and completions to Azure Monitor through diagnostic settings, and tracks token usage per consumer in Application Insights — with a built-in analytics workbook for consumption patterns. That combination is what makes per-agent, per-workflow cost attribution possible.The Gateway, DeployedHere is the practical path. First, the infrastructure — pick your weapon: # Azure CLI: v2 tiers provision through ARM/Bicep, so deploy a template az group create --name rg-ai-gateway --location malaysiawest az deployment group create \ --resource-group rg-ai-gateway \ --template-file apim-ai-gateway.bicep \ --parameters apimName=apim-ai-gw One note before the templates: az apim create has not caught up with the v2 tiers — its --sku-name argument still accepts only the classic SKUs (Developer, Basic, Standard, Premium, Consumption). That is why the CLI path here deploys a template instead of calling the APIM command directly; v2 instances are created through ARM/Bicep, Terraform, or the portal. # Terraform resource "azurerm_api_management" "ai_gateway" { name = "apim-ai-gw" location = "malaysiawest" resource_group_name = azurerm_resource_group.rg.name publisher_name = "Platform Engineering" publisher_email = "platform@contoso.com" sku_name = "PremiumV2_1" } // Bicep resource apim 'Microsoft.ApiManagement/service@2025-09-01-preview' = { name: 'apim-ai-gw' location: 'malaysiawest' sku: { name: 'PremiumV2' capacity: 1 } properties: { publisherEmail: 'platform@contoso.com' publisherName: 'Platform Engineering' } } Then the governance layer. A typical inbound policy for an agent-facing model API combines token rate limiting with per-agent metric emission: <!-- Inbound: cap token throughput per subscription --> <llm-token-limit counter-key="@(context.Subscription.Id)" tokens-per-minute="20000" estimate-prompt-tokens="false" remaining-tokens-variable-name="remainingTokens" /> <!-- Emit token metrics with dimensions for cost attribution --> <llm-emit-token-metric namespace="llm-metrics"> <dimension name="API ID" value="@(context.Api.Id)" /> <dimension name="Agent ID" value="@(context.Request.Headers.GetValueOrDefault("x-agent-id", "unknown"))" /> </llm-emit-token-metric> The x-agent-id dimension is a small trick worth adopting: require each agent to identify itself in a header, and your token cost dashboard suddenly breaks down by agent instead of by subscription blob.Verify the path end to end with a direct call: curl -sS "https://apim-ai-gw.azure-api.net/llm/v1/chat/completions" \ -H "Ocp-Apim-Subscription-Key: $APIM_KEY" \ -H "Content-Type: application/json" \ -H "x-agent-id: support-agent-01" \ -d '{ "model": "gpt", "messages": [{"role": "user", "content": "Summarize the SLA policy"}] }' | jq '.usage' On the security side, the gateway authenticates to Azure-hosted model services with managed identity — no model API keys sitting in agent configuration — and can moderate prompts through Azure AI Content Safety policies before they reach the backend. OAuth flows for apps and agents accessing APIs or MCP servers run through APIM's credential manager.The Reference Architecture ┌──────────────────────────────┐ │ Azure API Management │ │ (AI Gateway) │ │ │ Agents ────────► │ Unified Model API (preview) │ MCP clients ───► │ MCP passthrough / exposure │ A2A peers ─────► │ A2A agent APIs │ │ │ │ llm-token-limit • semantic │ │ cache • content safety • │ │ token metrics • logging │ └──────┬──────────┬─────────────┘ │ │ ┌────────────┴──┐ ┌────┴───────────────┐ │ Model backends │ │ Tool / agent layer │ │ Foundry, AOAI, │ │ MCP servers, A2A │ │ Anthropic, │ │ agents, internal │ │ Vertex, Bedrock│ │ REST APIs │ └───────────────┘ └────────────────────┘ ▲ │ Azure API Center (inventory: APIs, MCP servers, agents) Agents and MCP clients point at the gateway, not at backends directly. API Center holds the inventory so platform teams can answer what exists, who owns it, and whether it is approved.Pitfalls I Keep Seeing1. Anthropic Messages API requires the v2 tiers. If you run classic-tier APIM today and plan to front Claude through the gateway, budget for migration — this is documented as v2-only for now.2. Prompt and completion logging is off by default. You enable it per API via diagnostic settings, and you should think hard before you do: prompts routinely contain personal data and secrets. Messages log in 32 KB chunks with a 2 MB cap per direction, so treat this as an auditing decision with a retention policy, not a default.3. Semantic caching is extra infrastructure, not a checkbox. You need Azure Managed Redis or another RediSearch-compatible cache wired to APIM. Cache hits also mean stale answers — tune the similarity threshold and TTL deliberately for workloads where "close enough" responses are acceptable.4. Token data is only as good as the backend response. Microsoft documents that when token usage is missing — broken streams, terminated connections — usage may be logged inaccurately or not at all. Do not build billing-grade chargeback on gateway logs alone; reconcile against provider meters.5. A gateway only in front of the LLM is half a gateway. If agents can reach MCP tool servers and A2A peers directly, your governance has side doors. The point of this release is that MCP and A2A traffic can go through the same plane — use it.6. Preview means preview. The unified model API and the Foundry integration are previews. Pilot them, design around them, but keep a fallback path until they GA.When This MattersThis is worth your attention when you have multi-model strategies, agents connecting to several MCP servers, A2A chains, regulatory audit requirements, or token spend that needs attribution by team and workflow. For a single agent on a single Azure OpenAI deployment, the gateway fundamentals from my earlier article are sufficient — you can wait for this layer to mature.Key TakeawaysAPIM now fronts the whole agent stack — LLM endpoints, MCP servers, and A2A agent APIs through one gateway, with inventory in Azure API Center.The unified model API (preview) is the headline bet: one OpenAI-compatible endpoint, automatic format translation, cross-provider routing and failover without code changes.Token governance is policy-level: llm-token-limit for quotas, semantic caching for cost and latency, llm-emit-token-metric with custom dimensions for per-agent cost attribution.Observability is real but bounded: prompts and completions log to Azure Monitor on opt-in, and token data can be incomplete on broken streams — reconcile before you charge back.Close the side doors: the value only lands if MCP and A2A traffic route through the gateway too, with managed identity instead of scattered API keys.Continue the series: Azure API Management as AI Gateway — Managing Access, Throttling, and Observability for LLM Endpoints, Multi-Agent AI Systems in 2026 — Architecture Patterns (MCP, A2A, Swarm) for Production, and Copilot Studio vs Azure AI Foundry — The Enterprise Decision Framework. Related Reading Azure API Management as AI Gateway: Managing Access, Throttling, and Observabili The 2026 Multi-Agent Protocol Stack: MCP + A2A + ADK Define the Enterprise Inter Multi-Agent AI Systems in 2026 — Architecture Patterns (MCP, A2A, Swarm) for Pro --- ## Agentic AI at Build 2025: Copilot Coding, SRE Agents, Project Amelie URL: https://wenfeng.my/agentic-ai-build-2025-copilot-coding-agent-sre-agent-project-amelie/ Author: admin Published: 2026-08-27 23:30:06 Modified: 2026-10-04 15:33:51 Microsoft Build 2025 was the year the company stopped talking about AI assistants and started shipping AI workers. In May 2025, Microsoft introduced three flagship autonomous agents: the GitHub Copilot coding agent, the Azure SRE Agent, and Project Amelie — alongside the general availability of Azure AI Foundry Agent Service and the unveiling of Microsoft Discovery, an agentic research platform for scientists and engineers.The framing then was ambitious. The question now, a year later, is more useful for anyone planning real deployments: what did these agents become, and which of them are you safe to bet production workloads on?As an architect working with Azure-first enterprises across Malaysia and Southeast Asia, I've spent the past year fielding variations of that exact question. Here is my assessment — what each agent does, where it stands today, the security model you need around it, and a practical adoption sequence.The Shift: From Copilot to AgentBefore the specifics, the conceptual leap matters. The copilot pattern that dominated 2023–2024 is reactive: you ask, it suggests, you accept or reject. The agentic pattern Microsoft went all-in on at Build 2025 is different: an agent receives a goal, decomposes it into steps, executes those steps with tools, validates the outcome, and iterates until the goal is met — with human oversight concentrated at approval gates rather than on every keystroke.This is the same architecture I run for my own content pipeline with Hermes and OpenClaw: research agents, writer agents, review agents, each with its own scope and guardrails. What changed at Build 2025 is that Microsoft began packaging these patterns into managed products with enterprise governance — managed identities, RBAC, audit telemetry, approval workflows — instead of leaving every organization to build that scaffolding itself.GitHub Copilot Coding AgentWhat Microsoft AnnouncedAnnounced on May 19, 2025 — day one of Build — the coding agent was GitHub's headline agentic release. Internally codenamed Project Padawan, it runs entirely on GitHub Actions. You assign a GitHub issue to Copilot, exactly as you would assign it to a teammate, and the agent takes it from there: it boots a virtual machine, clones the repository, analyses the codebase using retrieval-augmented generation powered by GitHub code search, pushes commits to a draft pull request, and tags you for review. Two capabilities deserve emphasis: Model Context Protocol (MCP) support lets the agent pull in external data and tooling, and vision models let it read screenshots and mockups attached to issues.GitHub shipped it with a genuinely thoughtful default security posture:The agent can only push to branches it created — your default branch stays untouched.The developer who asks the agent to open a pull request cannot approve it, so required-review rules are always honoured.The agent's internet access is limited to a trusted, customisable allowlist.GitHub Actions workflows will not run on the agent's code without your approval.Existing repository rulesets and organisation policies apply as normal.GitHub's guidance for writing effective issue prompts goes by the acronym WRAP, and availability at launch covered Copilot Enterprise and Copilot Pro+ customers, with billing moving to one premium request per model request from June 4, 2025.Where It Stands TodayThe agent graduated from preview to general availability, and GitHub's documentation now brands it the Copilot cloud agent — with additions that extend the original model, including automations that run the agent on a schedule or in response to events, and explicit controls for rationale, confidence, and approval on automated issues. The direction of travel is clear: from "assign one issue" toward supervised fleets of agents working a backlog.Production Readiness and SecurityMy rating: production-ready for well-tested repositories with strong conventions; use with guardrails everywhere else. The agent's self-correction loop only works if there is something to correct against — a test suite. In repositories without tests, the agent has no validation mechanism, and "looks right" is not the same as "works".Security implications worth addressing before broad rollout:Prompt injection via issues. Anyone who can open an issue can feed instructions to your agent. Treat issue bodies as untrusted input, and keep issue creation permissions tight on repositories the agent works in.Dependency hygiene. The agent may introduce new dependencies. Pin versions and run dependency review in CI for agent PRs exactly as you would for human ones.Secrets scanning is non-negotiable. Enforce push protection regardless of whether a human or an agent wrote the commit.Azure SRE AgentWhat Microsoft AnnouncedThe Azure SRE Agent, introduced in preview at Build 2025, applies LLM reasoning to site reliability engineering: correlating alerts, metrics, logs, and recent changes to reach root cause faster, and proposing mitigations that humans approve before execution.Where It Stands Today: GA, Priced in Agent UnitsThe SRE Agent is now generally available, and it is worth understanding how far the product has come — and its pricing model, because there is no free tier.Billing is based on Azure Agent Units (AAUs), a standardised measure of agentic processing used across prebuilt Azure agents. A monthly bill combines always-on charges (4 AAUs per agent-hour for an agent that simply exists) with active-flow charges metered by token consumption per configured model. You can cap active-flow spending with a monthly AAU allocation limit in the agent's consumption settings — do this on day one, not after the first surprise invoice.Architecturally, the agent now operates through five extension primitives:Skills — discrete capabilities, including marketplace runbooks and Azure CLI scripts.Subagents — six built-in general-purpose subagents (Explore, Plan, CodeReview, Bash, Verification, GeneralPurpose) that let the agent parallelise investigation, planning, review, and verification work.Python tools — custom logic for scenarios that need code rather than configuration.MCP servers — more than 40 managed connectors (Datadog, New Relic, Splunk, Elasticsearch, Dynatrace, and more), plus custom tools.Agent hooks — event-triggered automations before investigation or after resolution; command hooks for deterministic CLI operations, prompt hooks for structured, policy-evaluable output.Two governance mechanisms matter most for production. A permission gate evaluates every proposed tool call before it runs — operators can require human approval, enforce policy rules, or block disallowed operations, even inside fully automated workflows. And audit telemetry routes to your own Application Insights instance, which is exactly what Malaysian enterprises under PDPA need to demonstrate that agent actions are traceable and attributable.Access to the agent itself is governed by three built-in RBAC roles: SRE Agent Administrator, SRE Agent Standard User, and SRE Agent Reader. Whoever creates the agent automatically receives Administrator.Practical SetupYou create the agent in the Azure portal (search for Azure SRE Agent, select Create, and associate the resource groups you want it to manage), then govern access with standard Azure CLI: # Give your SRE lead full administration of the agent az role assignment create \ --assignee <sre-lead-object-id> \ --role "SRE Agent Administrator" \ --scope "/subscriptions/<sub-id>/resourceGroups/rg-sre-agent-prod" # Give on-call engineers read-only visibility first az role assignment create \ --assignee <oncall-group-object-id> \ --role "SRE Agent Reader" \ --scope "/subscriptions/<sub-id>/resourceGroups/rg-sre-agent-prod" Runbooks are where operational value concentrates. Any Azure CLI operation can be automated through the agent as a skill — for example, a diagnostic-first runbook for an App Service degradation: #!/usr/bin/env bash # runbook-appservice-degraded.sh — invoked by SRE Agent via a skill RESOURCE_GROUP="rg-prod-web" APP_NAME="app-payments-api" az monitor metrics list \ --resource "/subscriptions/$SUB_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.Web/sites/$APP_NAME" \ --metric Http5xx,MemoryWorkingSet --interval PT5M az webapp log tail --resource-group "$RESOURCE_GROUP" --name "$APP_NAME" --duration 10 # Remediation is proposed, not executed — the permission gate # routes the restart to a human approver before execution. And make sure the agent has telemetry worth reasoning over — Application Insights deployed like any other infrastructure: resource appInsights 'Microsoft.Insights/components@2020-02-02' = { name: 'ai-prod-payments' location: 'malaysiawest' kind: 'web' properties: { Application_Type: 'web' RetentionInDays: 90 } } Security PostureMy rating: production-ready for investigation, triage, and supervised remediation. Three rules:Scope managed resource groups tightly. The agent can manage the full range of Azure services — compute, storage, networking, databases, monitoring. Associate only what it should touch.Keep the permission gate in approval mode for production writes. Let the agent propose; let a human approve. Graduate to policy-gated automation only for operations you would happily run unattended at 3 AM.Route and review the audit telemetry. It lands in your own App Insights — build the alerts and dashboards before you need them in an incident.Project AmelieProject Amelie was billed at Build 2025 as Microsoft's first Foundry autonomous agent. Give it a single prompt describing a machine learning problem, and it builds a fully validated ML pipeline — detailed evaluation metrics, a trained model, and ready-to-use, reproducible Python code. The validation step was the point: not just a trained model, but the evidence that the model works.Honest assessment a year later: Amelie is still early. Unlike the coding agent and the SRE Agent, it has not yet surfaced as a broadly documented, generally available Azure service. Treat it as an exploration and prototyping capability, not something to architect production ML workloads around today. If you need agentic ML pipeline automation now, build on Azure AI Foundry Agent Service — GA since Build 2025 with multi-agent orchestration capabilities — where you control the architecture, the governance, and the blast radius yourself.The strategic signal still stands, though: Microsoft is moving toward agents that don't just suggest pipeline code but generate, validate, and document entire ML workflows. Plan your data platform governance for that future; don't wait on a specific product timeline.Adoption Roadmap for Malaysian EnterprisesHere is the sequence I recommend for Azure-first organisations in this market:Phase 1 — Copilot cloud agent on internal tooling (months 1–3). Start where the risk is lowest and the feedback loop is fastest: internal utilities, test coverage, documentation, small bugs. Enable it on two or three repositories, enforce branch protection and required reviews, and measure review cycle time and defect escape rate. Premium-request billing makes usage visible — budget for it.Phase 2 — SRE Agent in supervised mode (months 2–6). Deploy alongside your existing on-call process. Let it investigate alerts and propose root causes while your engineers make every decision. Track mean time to identify and mean time to recover before and after. The agent's institutional knowledge compounds — Microsoft's own progression table has it learning your topology and failure patterns within the first month — so early deployment pays dividends later.Phase 3 — Build your own agents on Foundry Agent Service (months 6–12). The prebuilt agents teach you the governance patterns — permission gates, scoped identities, audit telemetry, approval workflows. Apply those patterns to domain-specific agents for your own workflows. Keep Amelie on your radar for ML prototyping as it matures.The regional context reinforces the timeline: Malaysia West is live in Kuala Lumpur, and Microsoft announced Southeast Asia 3 in Johor Bahru on 4 November 2025. Data residency and in-country disaster recovery are becoming real architectural options here. Pair that infrastructure with PDPA-aligned agent governance now, and you won't be retrofitting it later.Pitfalls I See RepeatedlyTreating agents as set-and-forget. The permission gate and approval workflows exist for a reason. An autonomous agent with broad permissions and no review loop is not automation; it is unmanaged risk.Pointing the coding agent at repositories without tests. No test suite means no validation loop, and the agent's core strength — iterating until green — disappears.Ignoring agent economics. Premium requests for the coding agent, AAUs for the SRE Agent. Set the monthly AAU allocation limit when you create the agent, and review consumption weekly for the first month.Over-scoping the SRE Agent's managed resource groups. Associate production resource groups only when your approval-gate discipline is proven. Start with non-production.Planning production on preview-era capabilities. Amelie at Build 2025 was a vision of where the platform is going. Architect on what is GA today; pilot what is preview.Key TakeawaysBuild 2025 was the inflection point. The Copilot coding agent, Azure SRE Agent, and Project Amelie — announced May 2025 — marked Microsoft's shift from assistive AI to autonomous agents with enterprise governance built in.Two of the three are production-ready today. The Copilot cloud agent is GA for well-tested repositories; the SRE Agent is GA with AAU-based pricing, a permission gate, and audit telemetry to your own App Insights. Amelie remains early-stage.Security is the gating factor, not capability. Scoped identities, permission gates, required reviews, secrets scanning, and audit trails are the minimum viable governance for autonomous agents — and the prebuilt agents finally give you these controls out of the box.Budget for agent consumption from day one. One premium request per model request, 4 AAUs per agent-hour always-on, token-metered active flow. Agents bill like workloads, because they are workloads.Build governance muscle now. Malaysian enterprises that learn to operate supervised agents in Malaysia West today will be the ones ready to scale them — across Southeast Asia 3 in Johor Bahru tomorrow. --- ## Microsoft Fabric June 2026: AI Functions GA, GPU Data Warehouse, and Data Agents Go Production URL: https://wenfeng.my/microsoft-fabric-june-2026-ai-functions-ga-gpu-data-warehouse-data-agents/ Author: admin Published: 2026-08-27 06:30:17 Modified: 2026-10-04 15:25:59 Fabric's June 2026 update is one of the heavier monthly releases I've seen since the platform launched. The themes are clear: AI capabilities moving from preview to production, GPU-accelerated warehouse capacity, and Data Agents reaching a general-availability state inside Microsoft 365 Copilot.If you've been evaluating Fabric as a warehouse-plus-Power BI play, this release shifts what the platform can actually do. You can now enrich, classify, and summarize data inside T-SQL and PySpark without wiring external endpoints, and business users can query governed data in natural language from within Copilot.Here's what the update actually delivers, what's genuinely production-ready versus still in preview, and what I'd adopt first.AI Functions: gpt-5-mini Is Now the DefaultThe headline change: Fabric AI Functions now default to gpt-5-mini with reasoning_effort set to low. The model carries a 400,000-token context window and a 128,000-token maximum output. For more complex transformations, you can configure gpt-5.1 or raise reasoning_effort to trade compute for higher-quality results.AI Functions work across pandas, PySpark, Dataflow Gen2, and the Data Warehouse SQL surface. Fabric manages the endpoint for you — no Azure OpenAI resource to provision, no keys to rotate.One important detail: the warehouse SQL surface is still labeled Preview in Microsoft Learn, and the function set there is specific. There is no generic "generate anything" function. What you actually get: -- Real AI Functions available in Fabric warehouse SQL SELECT AI_ANALYZE_SENTIMENT('This hotel was great!') AS sentiment; SELECT AI_CLASSIFY('Room was dirty', 'service', 'dirt', 'food') AS classification; SELECT AI_EXTRACT('Check-in was late and room dirty', 'sentiment', 'problem') AS extraction; SELECT AI_SUMMARIZE('The hotel was clean and staff were friendly.') AS summary; SELECT AI_GENERATE_RESPONSE('Reply in 20 words:', 'The room was noisy.') AS response; SELECT AI_TRANSLATE('The hotel was great', 'de') AS translation_de; SELECT AI_FIX_GRAMMAR('Th room are clean and staff were nice') AS fixed_text; The practical pattern is to run these over a warehouse table — say, enriching a week of support tickets with classification and sentiment — and materialize the results rather than recomputing on every query. Microsoft's own guidance warns that these functions call external AI APIs and can affect query performance, so precompute and stage the results.The pricing model mattersAI function calls through the built-in Fabric endpoint are billed to your Fabric capacity under the Copilot and AI meter. Consumption is measured in CU seconds per 1,000 tokens, and the difference between models is significant: ModelInput (per 1K tokens)Cached inputOutput (per 1K tokens) gpt-5-mini8.40 CU seconds0.84 CU seconds67.23 CU seconds gpt-5.142.02 CU seconds4.20 CU seconds336.13 CU seconds That's roughly a 5x cost delta between the two models on input, and the same ratio on output. Cached input is a 10x discount over fresh input, which rewards stable, repeated prompts over a dataset. Run enrichment as a batch job over materialized text, not inline in a live dashboard query.In pandas and PySpark notebooks, the interface is just as direct — classify a column with one call, then read token usage back from df.ai.stats: # PySpark AI Functions — default model is gpt-5-mini, reasoning_effort="low" df = spark.table("support_tickets_lakehouse.raw_tickets") enriched = df.ai.classify( labels=["billing", "technical", "account"], input_col="body", output_col="category", ) # Track consumption before you scale the job stats = enriched.ai.stats print(stats) # cached_tokens, input_tokens, output_tokens, reasoning_tokens, model You can set progress_bar_mode="stats" to see live capacity-unit estimates while a function runs. If you're budgeting a large enrichment job, this is how you do it.If the built-in endpoint doesn't fit your requirements, pandas and PySpark also support bringing your own model via a custom Azure OpenAI, Microsoft Foundry, or OpenAI-compatible endpoint. Fabric doesn't charge for those model calls — your provider bills token usage — but Fabric compute still applies.GPU-Accelerated Data Warehouse (Preview)The June update introduces GPU acceleration for the Data Warehouse — one of the most-watched preview features in this release. The stated benefit is faster query execution and reduced processing times on large datasets, with a workspace-level setting for administrators to enable GPU-accelerated queries and monitoring that shows which queries were accelerated and what improvement they achieved.Two honest caveats. First, this is Preview — treat it as an evaluation target, not a production dependency. Second, the June update documentation positions GPU acceleration around query performance on large datasets; it does not yet document a separate GPU tier sized specifically for AI-function workloads. If your architecture review depends on GPU behavior for AI Functions specifically, wait for the GA documentation before committing.What's directionally clear: Fabric is investing in compute headroom for the AI-enriched analytics pattern — queries that mix traditional aggregation with model calls. If you're designing a platform where BI and AI workloads share one warehouse, this is the piece to watch.The warehouse got operational improvements alongside GPU acceleration:Warehouse Monitor — visibility into running and completed queries, with the ability to spot expensive or failed ones and cancel runaway queriesCI/CD for SQL Analytics Endpoints — schema changes managed through database projects in Git and deployed through pipelinesTime Travel support — query historical versions of warehouse data for auditing and recovery scenariosData Agents: Generally Available in Microsoft 365 CopilotThis is the feature with the broadest organizational impact. The Fabric data agent is now a generally available feature for building conversational Q&A over your enterprise data — and the June update continues the push to surface data agents inside Microsoft 365 Copilot, so business users can ask natural-language questions from Teams, M365 Chat, and Copilot surfaces without opening Fabric.The governance story is the reason I'd take this seriously for production:Same security model, same results. Row-Level Security and Column-Level Security still apply. Access to a Power BI semantic model through a data agent is governed by Read permission on the model — no workspace-level access required, but no more access than that.Microsoft Purview integration. When data agents are surfaced through M365 Copilot, Purview governance policies continue to apply. Users can only access what their credentials and Purview policies allow, regardless of entry point.Precedence layers. Organizational policies and workspace governance settings override developer instructions and user prompts. The agent operates within approved boundaries no matter how it's configured.Auditability. Agent interactions can be logged and discoverable through Purview Audit and eDiscovery.On the capability side, the June release adds improvements to NL2SQL accuracy, multi-source query routing (the agent determines which source holds the answer), and a built-in code interpreter for more advanced analysis without leaving the agent.Provisioning the capacity underneathData agents, AI Functions, and warehouse workloads all consume Fabric capacity (F SKUs). If you're standing up a dedicated capacity for this, the Azure CLI covers it: az fabric capacity create \ --resource-group rg-fabric-prod \ --capacity-name fab-prod-my \ --administration "{members:[admin@wenfeng.my]}" \ --sku "{name:F4,tier:Fabric}" \ --location southeastasia Or declaratively with Bicep: resource fabricCapacity 'Microsoft.Fabric/capacities@2025-01-15-preview' = { name: 'fab-prod-my' location: 'southeastasia' sku: { name: 'F4' tier: 'Fabric' } properties: { administration: { members: [ 'admin@wenfeng.my' ] } } } Start small — F2 or F4 — pilot the agent and AI Functions on it, then scale the SKU with az fabric capacity update --sku "{name:F8,tier:Fabric}" once you have real consumption data.OneLake Storage Tiers and Lifecycle Management (Preview)OneLake gains storage tiers with lifecycle management — rules that automatically move files to lower-cost storage based on criteria like created date, modified date, or last-accessed date. There's also item size reporting for visibility into what's consuming storage across the platform.This is the cost lever. If you're accumulating years of lakehouse data, lifecycle rules tier cold data without manual intervention — the same pattern Azure Blob Storage lifecycle management gave you, now native to OneLake. It's Preview, so validate behavior on a non-critical workspace before applying rules broadly.What I'd Adopt FirstBased on what's production-ready versus preview, my adoption order:AI Functions with gpt-5-mini — the cheapest entry point into in-platform AI enrichment. Batch enrichment over materialized tables, with df.ai.stats to track CU burn.Data Agents — GA, with the security model intact. Pilot with a controlled user group on read-only sources where RLS/CLS already exists.OneLake lifecycle management — set rules early; cost savings compound monthly.Warehouse Monitor + CI/CD for SQL endpoints — operational hygiene before you scale AI workloads.GPU-accelerated warehouse — evaluate in preview once your AI Functions usage grows; wait for GA before depending on it.Pitfalls to AvoidDon't write `AI_GENERATE()` and expect it to work. The warehouse AI function set is specific (AI_CLASSIFY, AI_EXTRACT, AI_GENERATE_RESPONSE, etc.). Check the current documentation — the surface area evolves between updates.Don't run AI Functions inline in interactive queries. Each call consumes capacity. Materialize results into staging tables and query those.Don't assume gpt-5.1 for everything. At ~5x the CU cost of gpt-5-mini, reserve it for tasks where reasoning quality actually changes the output.Don't deploy data agents without checking RLS/CLS coverage. The agent respects your security model — which means gaps in your model become gaps in the agent's answers.Don't conflate Preview with GA. GPU warehouse acceleration and OneLake storage tiers are Preview this cycle. Design for them, but don't make production SLAs depend on them yet.Key TakeawaysAI Functions default to gpt-5-mini with `reasoning_effort=low` — 400K context, 128K max output — and bill to Fabric capacity in CU seconds. gpt-5.1 is available but costs ~5x more per 1,000 tokens.The warehouse AI function surface is specific and still Preview — AI_CLASSIFY, AI_EXTRACT, AI_SUMMARIZE, AI_TRANSLATE, AI_FIX_GRAMMAR, AI_ANALYZE_SENTIMENT, AI_GENERATE_RESPONSE. Materialize results; don't run them inline.Data Agents are generally available and reach users through Microsoft 365 Copilot, with Purview governance, RLS/CLS, and auditability carried through. Start with read-only pilots on governed sources.GPU-accelerated warehouse and OneLake storage tiers arrive in Preview — evaluate both, but don't build production SLAs on them until GA.Budget with real numbers. Use df.ai.stats and the Capacity Metrics app to measure AI Functions consumption before scaling the workload.Related reading: OneLake Security GA: Designing Row-Level, Column-Level, and Agent-Safe Access for Fabric, Microsoft Fabric Data Agents: Building Agentic AI on Your Enterprise Data, and Microsoft Fabric at Build 2026: What the Agentic AI Announcements Actually Mean. --- ## Malaysia Budget 2026: Cloud, AI, and Digital Transformation Incentives Every Enterprise Should Know URL: https://wenfeng.my/malaysia-budget-2026-cloud-ai-digital-transformation-incentives/ Author: admin Published: 2026-08-26 23:30:07 Modified: 2026-10-04 15:30:54 By Law Wen Feng, Principal Solution ArchitectMalaysia's Budget 2026 marks a turning point for enterprises navigating the cloud and AI landscape. The government has sent a clear signal: digital transformation is not optional — and it's being backed by real money.For years, I've helped Malaysian enterprises architect cloud solutions that balance performance, compliance, and cost. What's changed with Budget 2026 is that the fiscal environment itself is now an accelerant. Tax incentives, grants, and regulatory frameworks have aligned in ways that make 2026 the right time to move — if you know what's on the table and how to use it.Here's what every enterprise decision-maker in Malaysia should understand about the cloud, AI, and digital transformation incentives embedded in Budget 2026.Tax Incentives for Cloud and Digital InvestmentThe most immediate financial impact of Budget 2026 for cloud adopters comes through the enhanced capital allowance and deduction framework for ICT and digital investment.Accelerated Capital Allowance on ICT Equipment and SoftwareBudget 2026 introduces an Accelerated Capital Allowance (ACA) on qualifying capital expenditure incurred from 11 October 2025 to 31 December 2026 for the purchase of:Heavy machinery from local manufacturersPlant and general machinery acquired from local manufacturersICT equipment and computer softwareConsultation, licensing and incidental fees related to customized computer software developmentThe ACA provides a 20% initial allowance plus a 40% annual allowance, fully claimable within 2 years — compressing what would normally be a multi-year write-down into a much shorter period. For CFOs, this materially improves the cash-flow math on on-premises modernisation projects.What this means in practice: The ACA applies to capital expenditure on qualifying assets, not to cloud subscription fees themselves. Enterprises migrating workloads should work with their tax advisors to determine which migration-related capital expenditure (hardware, software licenses, professional services where qualifying) falls within scope. A mid-sized financial services firm spending approximately RM 2.8 million on qualifying migration costs could accelerate a significant portion of that deduction versus the typical multi-year write-down — but the exact qualifying scope must be confirmed against the gazetted rules before planning decisions are made on this basis.Verify before you plan. The ACA covers capital expenditure on ICT equipment and software. Cloud subscription (OPEX) costs are treated separately under existing deduction rules. Confirm qualifying scope with LHDN or your tax advisor for your specific project.SST on Cloud ServicesBudget 2026 did not announce a blanket service tax exemption on cloud services. Malaysia's service tax framework continues to apply to taxable digital services, including cloud services provided by foreign service providers. For enterprises running workloads on local cloud infrastructure — the AWS Asia Pacific (Malaysia) Region or Azure Malaysia West — the competitive dynamic between local and offshore providers remains a live procurement consideration, but enterprises should not assume an automatic 8% SST reduction on cloud invoices under Budget 2026.AI Investment IncentivesThe AI-specific measures in Budget 2026 are where things get genuinely interesting for forward-looking enterprises.Additional 50% Tax Deduction for AI and Cybersecurity TrainingTo encourage AI adoption in business operations, Budget 2026 provides an additional 50% tax deduction for micro-, small- and medium-sized enterprises (MSMEs) that incur expenses on AI and cybersecurity training courses recognised by the MyMahir National AI Council for Industry (NAICI) — jointly led by TalentCorp, KESUMA and MyDIGITAL Corporation. The deduction is available once in two years, for applications received by TalentCorp from 1 January 2026 to 31 December 2027.Why this matters: In my experience advising enterprises on AI adoption, the biggest barrier is not technology — it's capability. This deduction directly offsets the upskilling cost that most business cases ignore. For a company building an internal AI team, every ringgit spent on accredited training effectively costs less after tax.Malaysia Digital Acceleration Grant (MDAG)Budget 2026 allocates RM53 million to the Malaysia Digital Acceleration Grant, which catalyses the growth and adoption of emerging technologies such as blockchain, artificial intelligence, and quantum computing. The grant is administered through MDEC, and enterprises evaluating AI projects should check MDEC's current MDAG guidelines for eligible activities, matching ratios, and application windows.The Bigger AI Budget PictureBeyond these enterprise-facing measures, Budget 2026 funds the national AI infrastructure layer that enterprises will build on:RM18 million for the National AI Office (NAIO) to drive AI strategy and governanceRM180 million under the NIMP Industry Development Fund (NIDF) for high-impact sectors including AI and digitalRM2 billion for a proposed Sovereign AI Cloud through MCMC — the foundation for secure, sovereign national AI infrastructureNearly RM5.9 billion for research, development, commercialisation and innovation (R&D&C&I) across ministriesFor enterprise architects, the Sovereign AI Cloud proposal is the one to watch: it signals that Malaysia intends to offer in-country AI infrastructure for workloads where data residency is non-negotiable.Digital Transformation FundingBeyond individual tax deductions, Budget 2026 allocates substantial funding for structured digital transformation programmes.Malaysia Digital Acceleration Grant (MDAG)The flagship enterprise-facing allocation is the RM53 million Malaysia Digital Acceleration Grant, administered through MDEC to catalyse adoption of emerging technologies including AI, blockchain, and quantum computing. Enterprises should check MDEC's published guidelines for eligibility, matching requirements, and application windows.MSME Digital Grant MADANI and SME FinancingFor smaller enterprises, the MSME Digital Grant MADANI remains the primary matching grant for digital adoption, complemented by RM30 billion in government-backed SME financing through SJPP guarantees — a large pool of capital that enterprises can pair with cloud migration and automation projects.What I've seen work: Enterprises that succeed with grant applications typically have three things ready: a documented current-state architecture, a clear target-state vision with measurable KPIs, and an implementation partner with relevant certifications. If you're planning to apply, start your application preparation now — the technical proposal requirements are substantial, and review timelines run to several weeks.Automation Capital AllowanceThe Automation Capital Allowance — administered with applications through MIDA — gives companies undertaking automation a capital allowance on qualifying automation expenditure (an initial allowance followed by accelerated annual allowances), with prior approval required. Under the governing gazette rules (Income Tax (Accelerated Capital Allowance) (Automation Equipment) Rules 2017, as amended in 2020), it applies to companies that have carried on manufacturing activities for at least 36 months, which makes it relevant well beyond the factory floor: process automation, production-line robotics, and AI-augmented quality inspection can all fall within its scope depending on the approved project. Enterprises in the services sector should confirm current eligibility with MIDA, as the scheme's scope has evolved through successive Budget cycles.For cloud architecture professionals, this means that designing cloud-native automation workflows — serverless pipelines, event-driven architectures, AI-augmented document processing — can carry a direct fiscal incentive attached.PDPA Compliance Requirements: The Obligation Behind the OpportunityEvery incentive and grant discussed above operates within a tightening regulatory environment: compliance with Malaysia's Personal Data Protection Act 2010 (PDPA) as amended by the Personal Data Protection (Amendment) Act 2024, which came into force in phases — 1 January 2025, 1 April 2025 and 1 June 2025 — with the mandatory data breach notification obligation (Section 12B of Act 709) effective from 1 June 2025.Budget 2026 reinforces the national data governance agenda alongside its digital investment measures, and government-linked funding programmes increasingly expect applicants to demonstrate sound data protection practices. Treat PDPA compliance as a prerequisite for incentive eligibility, not an afterthought.The direction of travel is visible in two frameworks the Budget endorses. First, the Digital Trust & Data Security Strategy 2026-2030, which sets the roadmap for strengthening cybersecurity and data protection across Malaysia's digital economy. Second, the ASEAN AI Safety Network (ASEAN AI SAFE) — endorsed at the 5th ASEAN Digital Ministers' Meeting in Bangkok and expected to become operational in 2026 — which signals that AI governance expectations will keep rising across the region, not just at home. Both are documented in MyDIGITAL Corporation's Budget 2026 statement. For enterprises, the practical read is simple: the compliance bar you clear today to qualify for incentives is also the bar your customers and regulators will hold you to tomorrow.Key Compliance Requirements for Cloud-Hosted Data:No general data localisation, but residency still matters architecturally. Malaysia does not impose a blanket data localisation requirement — the 2024 amendment actually removed the old whitelist regime for cross-border transfers. Cross-border transfers now rely on mechanisms such as contractual clauses, binding corporate rules, or data subject consent. Even so, regulated sectors (BNM-regulated financial institutions, for example) impose their own data residency and outsourcing controls, and in-country regions such as the AWS Asia Pacific (Malaysia) Region and Azure Malaysia West remain the practical choice for sensitive workloads.Data Protection Impact Assessments are good practice. The PDPA does not mandate DPIAs by statute, but conducting a data protection impact assessment before deploying AI systems that process personal data is the standard risk-management expectation — and many grant applications and regulator conversations will ask for one. Document what data the AI system processes, what decisions it influences, and what safeguards are in place.Data breach notification. Under Section 12B of the amended PDPA and the Data Breach Notification Guideline, a personal data breach must be notified to the Personal Data Protection Commissioner as soon as practicable and no later than 72 hours from the occurrence of the breach, and affected data subjects must be notified within 7 days of the initial notification to the Commissioner where the breach results in, or is likely to result in, significant harm to the data subjects. Your cloud architecture must support audit logging, incident detection, and automated alerting to meet these obligations in practice.Cross-border data transfer governance. Cloud architectures that route data through multiple regions must map every data flow and ensure each cross-border transfer has a valid legal basis under the amended PDPA (Sections 129 and related provisions) and the Commissioner's Cross-Border Personal Data Transfer Guidelines. This is especially relevant for enterprises using global CSPs with multi-region architectures.Practical guidance: In every cloud architecture engagement I lead, data residency mapping is now a Day 1 activity. Before any infrastructure is provisioned, we classify data by sensitivity, map data flows, and identify jurisdictional requirements. This is no longer a compliance checkbox — it's a core architectural constraint.Practical Guidance: Making This Work for Your EnterpriseKnowing the incentives exists is one thing. Executing against them is another. Here's how I'd recommend Malaysian enterprises approach this.Step 1: Conduct a Digital Maturity AssessmentBefore chasing grants and incentives, understand where you are. A digital maturity assessment should cover your current infrastructure, data estate, application portfolio, and workforce capabilities. Several accredited consulting firms offer this as a first step for MSME Digital Grant MADANI and MDAG applications.Step 2: Align Your Cloud Roadmap with Incentive TimelinesThe Accelerated Capital Allowance window closes on 31 December 2026, and the AI training deduction applies once in two years. If your enterprise is planning a cloud migration or AI deployment in 2027, consider accelerating to 2026 to capture the incentive window. But don't rush — a poorly executed migration that misses deadlines is worse than a delayed migration that captures full incentives.Step 3: Engage Early with MDEC and Relevant Sector AgenciesMDAG (Malaysia Digital Acceleration Grant) and AI-focused grant applications require detailed technical proposals. Engaging with MDEC during the scoping phase — not after the project is designed — saves significant time and avoids costly redesigns.Step 4: Build Compliance Into Your ArchitectureDon't treat PDPA compliance as a post-deployment activity. Data localisation, encryption, access controls, audit logging, and breach detection should be architectural requirements, not afterthoughts. Cloud-native services from major CSPs offer compliance-ready building blocks — use them.Step 5: Document Everything for Incentive ClaimsTax deductions and grants require documentation. Keep records of cloud invoices, migration project costs, AI training expenditures, and compliance assessments. In my experience, the enterprises that capture the most value from government incentives are the ones that treated incentive documentation as a project workstream from Day 1.Putting It Into Practice: Compliance-Ready Infrastructure PatternsIncentives reward speed; PDPA punishes haste. The enterprises that capture both are the ones that bake residency, auditability, and breach detection into their infrastructure from Day 1. Here are four patterns I use in almost every Malaysian cloud engagement, with working examples.Pattern 1: Pin workloads to Malaysian regionsThe simplest way to satisfy residency requirements is to make non-compliant deployments impossible rather than discouraged. Azure Policy can restrict allowed locations at the subscription or management group scope: { "policyRule": { "if": { "not": { "field": "location", "in": ["malaysiawest", "global"] } }, "then": { "effect": "deny" } } } The global entry matters — some Azure resources (Entra ID objects, policy assignments) are global-scoped and carry no region. In Bicep, constrain the location parameter so new deployments land in-country by default: @allowed([ 'malaysiawest' ]) param location string = 'malaysiawest' resource storageAccount 'Microsoft.Storage/storageAccounts@2024-01-01' = { name: 'stmyworkload01' location: location sku: { name: 'Standard_LRS' } kind: 'StorageV2' } Pattern 2: Audit residency drift in your existing estateBefore any incentive claim or grant application, audit what you already run. One Azure CLI query surfaces every resource sitting outside Malaysian regions: az resource list \ --query "[?!(contains(location,'malaysia') || location=='global')].{Name:name, RG:resourceGroup, Type:type, Location:location}" \ -o table Run it monthly and treat any output as an architectural debt item. Residency drift is how enterprises end up with offshore data they cannot account for during a regulator conversation.Pattern 3: Centralised audit logging for breach notification readinessThe amended PDPA requires breach notification to the Commissioner as soon as practicable, and no later than 72 hours from the occurrence of the breach. You cannot meet that clock without centralised audit logs. Terraform makes the wiring declarative: resource "azurerm_monitor_diagnostic_setting" "keyvault_audit" { name = "kv-audit-to-loganalytics" target_resource_id = azurerm_key_vault.vault.id log_analytics_workspace_id = azurerm_log_analytics_workspace.security.id enabled_log { category = "AuditEvent" } } Pair the Log Analytics workspace with alert rules for anomalous access and you have the detection layer your breach response process stands on.Pattern 4: Script your DPIA data inventoryData Protection Impact Assessments begin with knowing which services touch personal data. A short Python script using the Azure SDK walks the estate and produces the inventory: from azure.identity import DefaultAzureCredential from azure.mgmt.resource import ResourceManagementClient from azure.mgmt.storage import StorageManagementClient import os credential = DefaultAzureCredential() subscription_id = os.environ["AZURE_SUBSCRIPTION_ID"] rm = ResourceManagementClient(credential, subscription_id) sm = StorageManagementClient(credential, subscription_id) for rg in rm.resource_groups.list(): for sa in sm.storage_accounts.list_by_resource_group(rg.name): print(f"{rg.name}\t{sa.name}\t{sa.location}\t" f"blob_encrypted={sa.encryption.services.blob.enabled}") Export that output to CSV and you have the starting point for the DPIA that grant applications and AI deployments will demand.Each of these patterns does two jobs at once: it hardens your compliance posture, and it generates the documentation trail that incentive claims and grant reviews require. That double duty is exactly why the enterprises winning Budget 2026 incentives treat compliance engineering as part of the migration itself.The Bottom LineBudget 2026 has created a genuine alignment of fiscal incentives, government funding, and regulatory momentum for Malaysian enterprises investing in cloud and AI. The Accelerated Capital Allowance on ICT equipment and software, the additional 50% deduction for AI and cybersecurity training, the Automation Capital Allowance, the RM53 million Malaysia Digital Acceleration Grant, and the RM2 billion Sovereign AI Cloud proposal collectively represent one of the most enterprise-friendly digital transformation environments Malaysia has offered — even though some headline numbers circulating in the market overstate what the Budget actually announced.But these incentives have windows. They require compliance. They demand documentation. And they work best when enterprises approach them with a deliberate, architecturally sound strategy rather than a scramble to meet fiscal year deadlines.If you're a Malaysian enterprise leader considering cloud migration, AI adoption, or broader digital transformation in 2026, the message is simple: the government is meeting you halfway. Make sure you're ready to take that step.Key TakeawaysThe Accelerated Capital Allowance accelerates ICT write-downs — qualifying capital expenditure on ICT equipment, computer software and related costs incurred between 11 October 2025 and 31 December 2026 qualifies for a 20% initial allowance plus 40% annual allowances, fully claimable within 2 years. Confirm qualifying scope with your tax advisor; cloud subscription fees are treated separately.AI training gets a direct tax subsidy — MSMEs receive an additional 50% tax deduction on accredited AI and cybersecurity training (once every two years). Use it to offset the upskilling cost your AI business case is missing.PDPA compliance is the operating licence for incentive eligibility — the amended PDPA (in force in phases from 1 January to 1 June 2025) imposes breach notification to the Commissioner "as soon as practicable and no later than 72 hours" and data subject notification within 7 days for breaches causing significant harm. Build data residency, encryption, and impact assessments into your cloud architecture from Day 1.Grant funding is real but bounded — the RM53 million Malaysia Digital Acceleration Grant and MSME Digital Grant MADANI are the actual vehicles; RM30 billion in SJPP-backed SME financing rounds out the picture. Application processes are rigorous — start preparing your technical proposal now if you plan to apply in 2026.Documentation and incentive capture should be a parallel workstream from project inception — enterprises that systematically track qualifying expenditure and compliance evidence extract significantly more value from government incentives.Have questions about how Budget 2026 incentives apply to your enterprise cloud strategy? Reach out at wenfeng.my/ — I'd be happy to discuss how these changes impact your digital transformation roadmap. --- ## Self-Hosting LLM Inference in 2026: vLLM vs SGLang vs llama.cpp Decision Guide URL: https://wenfeng.my/self-hosting-llm-inference-2026-vllm-sglang-llamacpp-decision-guide/ Author: admin Published: 2026-08-26 06:30:02 Modified: 2026-08-26 06:30:02 The self-hosting vs. API decision for LLM inference has shifted dramatically in 2026. Open-weight models now rival proprietary APIs on many benchmarks, inference engines have matured to production-grade, and GPU capacity in Southeast Asia is finally obtainable without six-month lead times. For Malaysian enterprises with data sovereignty requirements, self-hosting is no longer a compromise — it's often the better architecture.But "let's self-host" is not one decision. It's two: should we self-host, and which engine do we run? This article answers both — with a decision matrix, working deployment patterns on Azure, and the real cost math.Why Teams Are Rethinking the API DefaultThree forces push enterprise teams away from pure-API inference:Data sovereignty. When prompts contain personal data — customer records, employee files, financial information — sending them to an offshore API endpoint creates a PDPA exposure. For financial institutions, BNM's Risk Management in Technology (RMiT) standard raises the bar further with explicit expectations on data residency and third-party risk.Cost unpredictability. API pricing shifts with model generations and vendor decisions. If your unit economics depend on someone else's pricing page, you don't have unit economics — you have a bet.Latency and control. Customer-facing agents need consistent time-to-first-token, and API rate limits become a hard capacity ceiling exactly when traffic spikes.Self-hosting addresses all three — but only if you size it correctly. A GPU VM running at 1% utilization is not cost optimization; it's a donation to Microsoft.The 2026 Engine LandscapeThree inference engines dominate production deployments.vLLM: The Production DefaultvLLM has become the de facto standard for self-hosted LLM inference. Its PagedAttention mechanism — which manages GPU memory the way an operating system manages virtual memory — delivers 2-4x higher throughput than naive implementations.Key capabilities:OpenAI-compatible API server (drop-in replacement for the OpenAI SDK)Continuous batching (dynamic request grouping for maximum GPU utilization)Tensor parallelism (split models across multiple GPUs)Speculative decoding (draft model + verification for 2-3x decode speedup)Prefix caching (automatic KV cache reuse for shared prompt prefixes)Multi-LoRA serving (serve multiple fine-tuned variants on one GPU)Production readiness: Battle-tested at scale; it's the serving layer behind most major open-model deployments. pip install vllm # Serve a 284B open-weight MoE model on 4x A100 80GB # (native FP4 experts + FP8 remainder ≈ 142GB+ of weights — # two 80GB GPUs leave no room for KV cache) vllm serve deepseek-ai/DeepSeek-V4-Flash \ --tensor-parallel-size 4 \ --max-model-len 32768 \ --gpu-memory-utilization 0.9 \ --enable-prefix-caching \ --host 0.0.0.0 --port 8000 SGLang: The Speed ChampionSGLang (Structured Generation Language) is optimized for structured output and complex multi-step programs. Its RadixAttention mechanism provides more aggressive prefix caching than vLLM's implementation.Key capabilities:RadixAttention (automatic KV cache sharing across requests with common prefixes)Structured output generation (JSON, regex, grammar-constrained — native, not post-processed)Multi-turn conversation optimization (session-aware caching)FlashInfer kernels (custom CUDA kernels for maximum throughput)When SGLang wins:Structured output workloads (JSON extraction, function calling)Multi-turn conversations (chat applications, agent loops)Workloads with high prompt overlap (RAG against a shared knowledge base) pip install "sglang[all]" python -m sglang.launch_server \ --model-path deepseek-ai/DeepSeek-V4-Flash \ --tp 4 \ --mem-fraction-static 0.9 \ --host 0.0.0.0 --port 8000 llama.cpp: The Edge and CPU Optionllama.cpp brings LLM inference to commodity hardware — CPUs, Apple Silicon, and edge devices. It's the only production-grade option for deployment without datacenter GPUs.Key capabilities:GGUF quantized models (2-bit through 8-bit — runs on consumer hardware)Metal acceleration (Apple Silicon native)Vulkan/ROCm support (AMD GPUs, older NVIDIA GPUs)Minimal dependencies (single binary, no Python environment)HTTP server with an OpenAI-compatible APIWhen llama.cpp wins:CPU-only inference (no GPU budget)Edge deployment (retail, IoT, on-premises kiosks)Apple Silicon development (M-series Macs)Small model deployment (under 14B parameters) git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp make -j$(nproc) # Start the server with a quantized model ./llama-server \ -m qwen3-14b-q4_k_m.gguf \ --host 0.0.0.0 --port 8000 \ -ngl 99 -c 8192 Decision Matrix: Which Engine for Which Workload FactorvLLMSGLangllama.cpp Best forGeneral production servingStructured output, multi-turnEdge, CPU, development GPU requirementNVIDIA (A100, H100, L40S)NVIDIA (A100, H100)CPU or any GPU Practical max modelMulti-GPU, any sizeMulti-GPU, any size~70B quantized Throughput vs naive2-4x3-5x (structured workloads)0.5-1x (CPU) OpenAI API compatYesYesYes Structured outputGood (guided decoding)Excellent (native)Good (grammars) QuantizationAWQ, GPTQ, FP8AWQ, GPTQ, FP8GGUF (Q2-Q8) Multi-LoRAYesYesLimited Prefix cachingYesYes (RadixAttention)No Standing Up Inference on AzureBefore the software, the hardware. One Azure-specific gotcha first: GPU quota defaults to zero cores for the NC/ND families on most subscriptions. Check and request increases before you commit to any launch date: # Check current GPU quota in Malaysia West az vm list-usage --location malaysiawest \ --query "[?contains(name.value,'NC')].{name:name.value,current:currentValue,max:limit}" \ -o table # Request quota increase (40 vCPUs = one NC40ads_H100_v5) az quota create \ --resource-name StandardNCadsH100v5Family \ --scope subscriptions/<subscription-id>/providers/Microsoft.Compute/locations/malaysiawest \ --limit 40 # Confirm the SKU is actually offered in your target region before committing to a launch date az vm list-skus --location malaysiawest --query "[?contains(name,'NC') || contains(name,'NV')].name" -o table That last command matters more than most teams expect. As of August 2026, az vm list-skus --location malaysiawest returns the NVadsA10_v5 (A10 24GB), NCads/NCadis_H100_v5 (H100), and NCas_T4_v3 families — but not the A100 v4 family. The NC24/NC48/NC96ads_A100_v4 SKUs are offered in southeastasia (Singapore), while the ND96amsr_A100_v4 8-GPU node is only offered in regions such as eastus, westus2, eastus2, westeurope, and uksouth. If in-country residency is non-negotiable, size against H100 v5 or A10 v5 SKUs in Malaysia West; if you need A100-class capacity, plan for a Singapore deployment and check whether your regulator accepts it. Always verify with az vm list-skus --location <region> --all before promising any specific GPU SKU in a specific region — availability changes quietly.Then provision the VM and lock down the network path (example uses southeastasia, where the A100 v4 family is offered): az group create --name rg-inference-prod --location southeastasia az vm create \ --resource-group rg-inference-prod \ --name vm-vllm-01 \ --size Standard_NC48ads_A100_v4 \ --image Ubuntu2204 \ --admin-username azureuser \ --ssh-key-values ~/.ssh/id_rsa.pub \ --os-disk-size-gb 512 # Inference port reachable from the app subnet only — never the internet az network nsg rule create \ --resource-group rg-inference-prod \ --nsg-name vm-vllm-01NSG \ --name Allow-AppSubnet-8000 \ --priority 100 --direction Inbound \ --source-address-prefixes 10.0.2.0/24 \ --destination-port-ranges 8000 \ --access Allow --protocol Tcp For IaC-managed estates, the same VM in Bicep (network and disk resources omitted for brevity): resource vllmVm 'Microsoft.Compute/virtualMachines@2024-07-01' = { name: 'vm-vllm-01' location: 'southeastasia' // A100 v4 family not offered in malaysiawest — verify with az vm list-skus properties: { hardwareProfile: { vmSize: 'Standard_NC48ads_A100_v4' } storageProfile: { imageReference: { publisher: 'Canonical', offer: '0001-com-ubuntu-server-jammy', sku: '22_04-lts-gen2', version: 'latest' } osDisk: { createOption: 'FromImage', diskSizeGB: 512 } } } } Because all three engines speak the OpenAI API, application code is engine-agnostic — switching engines later is a base_url change, not a rewrite: from openai import OpenAI client = OpenAI( base_url="http://10.0.1.10:8000/v1", api_key="your-vllm-api-key", ) resp = client.chat.completions.create( model="deepseek-ai/DeepSeek-V4-Flash", messages=[{"role": "user", "content": "Summarize PDPA obligations for a KL-based insurer."}], max_tokens=512, ) print(resp.choices[0].message.content) Container and Kubernetes PatternsFor production, containerize the engine. Isolation, reproducibility, and rolling updates all come free. docker pull vllm/vllm-openai:latest docker run --gpus all -d \ --name vllm-server \ -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ vllm/vllm-openai:latest \ --model deepseek-ai/DeepSeek-V4-Flash \ --tensor-parallel-size 4 \ --gpu-memory-utilization 0.9 \ --enable-prefix-caching For teams running AKS, a production deployment uses GPU node selectors and resource limits: apiVersion: apps/v1 kind: StatefulSet metadata: name: vllm-inference spec: replicas: 2 selector: matchLabels: app: vllm template: metadata: labels: app: vllm spec: nodeSelector: node.kubernetes.io/instance-type: Standard_NC96ads_A100_v4 containers: - name: vllm image: vllm/vllm-openai:latest args: - "--model" - "deepseek-ai/DeepSeek-V4-Flash" - "--tensor-parallel-size" - "4" - "--gpu-memory-utilization" - "0.9" ports: - containerPort: 8000 resources: limits: nvidia.com/gpu: 4 This pattern gives you self-healing, rolling updates, and horizontal scaling by adding replicas — critical once inference becomes a business-critical dependency.The Real Cost Math: Self-Hosting vs. APINumbers below are rough estimates based on published Azure pay-as-you-go pricing as of mid-2026; reserved capacity cuts GPU compute costs by 30-60%. API baseline: flagship-class pricing at $3 per million tokens.Scenario 1: 10M tokens/month (small workload)API: 10M × $3/1M = $30/month. No infrastructure, no operations.Self-hosted: A 14-32B open-weight model on NC24ads_A100_v4 (1x A100 80GB), ≈ $2,600-2,900/month at pay-as-you-go rates. Capacity is roughly 1-2B tokens/month — utilization under 1%.Verdict: Self-hosting is roughly 90x more expensive here. Use the API.Scenario 2: 500M tokens/month (medium workload)API: 500M × $3/1M = $1,500/month.Self-hosted: Same 1x A100 VM ≈ $2,600-2,900/month, now 25-40% utilized.Verdict: The API is still cheaper in raw dollars. Self-hosting at this volume is a sovereignty decision (PDPA/RMiT), not a cost decision.Scenario 3: 10B tokens/month (large workload)API: 10B × $3/1M = $30,000/month.Self-hosted: DeepSeek-V4-Flash (native FP4/FP8 weights) on ND96amsr_A100_v4 (8x A100 80GB — US/Europe regions only, not Singapore) ≈ $23,000-25,000/month pay-as-you-go — or roughly $10,000-13,000/month with 3-year reserved capacity. With continuous batching, sustained aggregate throughput of 250-350K tokens/minute is realistic — capacity of 10-15B tokens/month.Verdict: Self-hosting saves $5,000-7,000/month (~20%) at pay-as-you-go rates, and $17,000-20,000/month (55-65%) once reserved capacity is committed — with headroom for spikes.The Break-Even Point Break-even tokens/month ≈ GPU VM monthly cost ÷ API price per token $2,600-2,900/month 1x A100 VM vs $3/1M API → ~900M-1B tokens/month$23,000-25,000/month 8x A100 node (pay-as-you-go) vs $3/1M API → ~8B tokens/month, dropping to ~3.5-4B with 3-year reserved pricingTwo caveats. First, if your realistic API baseline is a budget-tier model ($0.10-0.50 per million tokens), break-even shifts 6-30x higher — compare against the capability class you would actually run, not headline prices. Second, self-hosting adds operational cost that never appears on the VM invoice: patching, monitoring, model updates, and on-call.The Malaysian Enterprise DecisionThree factors specific to Malaysian enterprises push toward self-hosting regardless of the raw cost math:1. Data SovereigntyIf your workload processes personal data — customer records, employee data, financial information — PDPA compliance may require that the data never leaves your control. Financial institutions face BNM RMiT on top of that. Self-hosting on Azure VMs in Malaysia West keeps everything in-country and makes the audit story straightforward.2. LatencyMalaysian enterprises serving ASEAN customers need sub-200ms response times. Inference on Azure Malaysia West eliminates the round-trip to Singapore or US API endpoints. For customer-facing agents, that round-trip is often the difference between "feels instant" and "feels sluggish."3. Predictable CostsAPI pricing changes without notice. Reserved capacity helps but doesn't eliminate the exposure. Self-hosted infrastructure has a fixed monthly cost that finance can budget — and that you can optimize over time with quantization and better utilization.Production Deployment ChecklistHardware Sizing Model SizeMin GPURecommended GPUAzure VM SKU 7-8B1x A10 24GB1x A100 80GBNVadsA10_v5 / NC24ads_A100_v4 13-14B1x A100 80GB1x A100 80GBNC24ads_A100_v4 30-34B1x A100 80GB2x A100 80GBNC48ads_A100_v4 70B (INT4)2x A100 80GB4x A100 80GBNC96ads_A100_v4 250-300B MoE (INT4)4x A100 80GB4x-8x A100 80GBNC96ads_A100_v4 / ND96amsr_A100_v4 (US/Europe only) 100B+ dense4x A100 80GB8x A100 80GBND96amsr_A100_v4 (US/Europe only) High Availability # Health check endpoint (vLLM and SGLang both expose it) curl http://localhost:8000/health upstream llm_backend { server 10.0.1.10:8000 weight=1; server 10.0.1.11:8000 weight=1; server 10.0.1.12:8000 weight=1; } server { listen 443 ssl; location /v1/ { proxy_pass http://llm_backend; proxy_read_timeout 300s; } } MonitoringvLLM exposes a Prometheus endpoint at /metrics — scrape it into Azure Monitor managed Prometheus and track:Tokens per second (throughput)Time to first token (latency)GPU utilization (efficiency)Queue depth (capacity planning)Error rate (reliability)Cost per 1M tokens (unit economics)SecurityTLS termination at the load balancer or App Gateway, not the inference serverAPI key authentication (vLLM supports --api-key; SGLang supports --api-key)Network isolation (NSG rules limiting access to known subnets — never expose 8000 to the internet)Model artifact security (private endpoints or encrypted storage for model weights)Prompt injection defense — self-hosted models lack the safety layers of Azure OpenAI. Implement input validation and output filtering at the application layerFive Pitfalls That Sink Self-Hosted InferenceBenchmarking single-stream speed instead of throughput. A demo that generates 60 tokens/second tells you nothing about production capacity. Load-test with realistic concurrency — vLLM ships benchmark_serving.py for exactly this.Defaulting to maximum context length. KV cache memory grows linearly with context. Setting --max-model-len 131072 on a large model can consume more VRAM than the weights themselves. Set it to what your workload actually needs.Shipping an open endpoint. vLLM and SGLang ship without authentication or rate limiting by default. I've seen inference servers indexed by Shodan within days. Use --api-key, NSG restrictions, and TLS termination — all three.Treating quantization as free. INT4/AWQ quality regressions are task-dependent — often negligible for summarization, noticeable for math and code. Evaluate on your own domain data before committing to a quantization level.Discovering GPU quota at deploy time. Azure NC/ND families default to zero cores on most subscriptions, and quota approval can take days. Request increases two to three weeks before your target launch date.ConclusionSelf-hosting LLM inference in 2026 is no longer an experiment — it's an architecture decision with a calculable break-even point. Below roughly 1B tokens/month, the API wins on simplicity and usually on cost; above a few billion tokens/month, self-hosting wins decisively — especially with reserved capacity; and data sovereignty can tip the decision at any volume in between. Start with vLLM, reach for SGLang when structured output dominates your workload, and keep llama.cpp for the edge.Key TakeawaysThe break-even point is ~900M-1B tokens/month for a single-GPU VM against flagship-class API pricing at pay-as-you-go rates — below it, the API wins; above a few billion tokens/month, self-hosting saves real money, and reserved capacity turns that into 50%+ savings.vLLM is the default choice for production — best ecosystem, most features, battle-tested at scale.SGLang wins for structured output — JSON extraction, function calling, and multi-turn agent loops benefit most from RadixAttention and native constrained decoding.llama.cpp is for edge and development — not production inference at scale, but unmatched for CPU-only and on-device deployments.Data sovereignty is the hidden accelerator — for Malaysian enterprises, PDPA and BNM RMiT often justify self-hosting even below the cost break-even point. --- ## Azure Cosmos DB Agent Memory and Agentic Retrieval Toolkits — The Database Layer for Production AI Agents URL: https://wenfeng.my/azure-cosmos-db-agent-memory-agentic-retrieval-toolkits/ Author: admin Published: 2026-08-25 23:30:28 Modified: 2026-10-04 15:26:10 By Law Wen Feng, Principal Solution Architect | wenfeng.my/If you have been building AI agents in the Azure ecosystem over the past year, you have probably hit the same wall I have: agents are stateless by default. The moment you need one to remember context across sessions, pull from a knowledge base, or ground its answers in enterprise data, you are duct-taping a vector store to a chat model and hoping it scales.Most teams I work with in Southeast Asia — Kuala Lumpur, Singapore, Jakarta — start their agent journey the same way: an LLM, a prompt, maybe a tool call or two. It works beautifully for demos. But the moment you ask the agent to recall what happened three conversations ago, or retrieve the right clause from a 50,000-page compliance library, the architecture crumbles.This article is about using Azure Cosmos DB as the database layer for AI agents — not as a generic "use Cosmos for everything" pitch, but as a practical, field-tested approach to agents that remember, retrieve, and act on enterprise data. And it covers the two purpose-built toolkits Microsoft shipped for exactly this problem: the Agent Memory Toolkit and the Agentic Retrieval toolkit.The Agent Memory ProblemAn AI agent has three core capabilities: reasoning (the LLM), acting (tool calls), and — critically — remembering. Without memory, your agent perpetually starts from zero. It cannot learn from past interactions, maintain user preferences, or recall which documents it already retrieved.Enterprise agents really need three kinds of memory:Short-term memory — the conversation context window. The LLM handles this natively, but it is bounded and ephemeral.Long-term memory — persistent facts, user preferences, past decisions, interaction history. This needs a durable store.Episodic memory — the ability to retrieve relevant past episodes or knowledge chunks when facing a new task. This is where retrieval-augmented generation (RAG) comes in.The usual response is to bolt on three separate systems: a cache for conversation history, a relational database for structured state, and a vector database for semantic search. That is three operational burdens, three integration points, three things to monitor, secure, and scale.Cosmos DB can collapse all three into a single, globally distributed database with native vector indexing.Why Cosmos DB for Agent MemoryI have spent enough time with Pinecone, Weaviate, and Milvus to respect what they do. They are purpose-built vector databases, and they do vector search extremely well. But here is what I have learned building production agents for enterprise clients: you rarely need just vector search.You need vector search plus structured queries. Semantic retrieval plus transactional consistency. Global distribution plus single-digit-millisecond latency. And you need all of it in a service your security team already trusts.Cosmos DB for NoSQL gives you:Native vector indexing with DiskANN-based algorithms (the same research behind Microsoft's large-scale search). Three index types: flat for brute-force exact search, quantizedFlat for compressed fast search, and diskANN for high-accuracy approximate search at scale.Hybrid queries that combine vector similarity with SQL-style filtering — tenant, document type, recency, permissions — in a single query.Full-text and hybrid search alongside vector search in the same container.Native TTL to expire ephemeral memories with zero cleanup jobs.Change feed for event-driven reactions when new memories are written.Global distribution with multi-region writes, which matters when your agents serve users across ASEAN from multiple Azure regions.Setting Up a Cosmos DB Container for Agent MemoryFirst, provision the infrastructure. Two details here trip people up, so I will call them out explicitly: the account-create command uses --locations (plural), and the vector embedding policy cannot be modified in place after container creation — you can only add new vector paths, or drop a path and re-add it with new settings, so define it up front.# Account (note: --locations, not --location)az cosmosdb create \ --name agent-memory-prod \ --resource-group rg-ai-agents \ --locations regionName=southeastasia failoverPriority=0 \ --default-consistency-level Session# Databaseaz cosmosdb sql database create \ --account-name agent-memory-prod \ --resource-group rg-ai-agents \ --name AgentMemoryDB \ --throughput 400# Container: indexing policy (with vectorIndexes) and vector embedding policy# are passed as two separate parametersaz cosmosdb sql container create \ --account-name agent-memory-prod \ --resource-group rg-ai-agents \ --database-name AgentMemoryDB \ --name memories \ --partition-key-path "/tenantId" \ --throughput 400 \ --idx @index-policy.json \ --vector-embeddings @embedding-policy.jsonThe two policy files:// index-policy.json{ "indexingMode": "consistent", "automatic": true, "includedPaths": [{ "path": "/*" }], "excludedPaths": [{ "path": "/vector/*" }], "vectorIndexes": [{ "path": "/vector", "type": "quantizedFlat" }]}// embedding-policy.json{ "vectorEmbeddings": [{ "path": "/vector", "dataType": "float32", "dimensions": 1536, "distanceFunction": "cosine" }]}The equivalent Bicep resource (Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers, API version 2025-10-15 or later) takes the same two structures: properties.indexingPolicy.vectorIndexes and properties.vectorEmbeddingPolicy.vectorEmbeddings.One sizing note: quantizedFlat and diskANN indexes require at least 1,000 vectors before the index kicks in — below that threshold Cosmos DB falls back to a brute-force scan. For agent memory stores that grow over time this is fine; for small tests, expect full-scan behaviour.Storing and Retrieving Agent MemoriesA single Cosmos DB document can hold structured metadata, raw text, and the vector embedding together. No separate collections needed.from azure.cosmos import CosmosClientfrom openai import AzureOpenAIfrom datetime import datetime, timezonecosmos = CosmosClient( "https://agent-memory-prod.documents.azure.com:443/", credential="<managed-identity-or-key>")container = cosmos.get_database_client("AgentMemoryDB") \ .get_container_client("memories")oai = AzureOpenAI( azure_endpoint="https://your-openai.openai.azure.com/", api_version="2025-06-01")def get_embedding(text: str) -> list[float]: resp = oai.embeddings.create( model="text-embedding-3-small", input=text) return resp.data[0].embeddingdef store_memory(tenant_id, user_id, session_id, memory_type, content, metadata=None): """Store an agent memory with its vector embedding.""" return container.upsert_item({ "id": f"{session_id}-{datetime.now(timezone.utc).timestamp()}", "tenantId": tenant_id, "userId": user_id, "sessionId": session_id, "memoryType": memory_type, # conversation | fact | preference | document "content": content, "vector": get_embedding(content), "createdAt": datetime.now(timezone.utc).isoformat(), "ttl": 2592000, # 30 days; requires TTL enabled at container level "metadata": metadata or {} })Notice the partition strategy: tenantId as the partition key keeps all of a tenant's memories co-located. For multi-tenant agent deployments you get isolation by design, and tenant-scoped queries hit a single logical partition.Retrieval in an agentic context is not "search and return top-k". The agent reasons about what to retrieve, when, and how to use it. Here is a hybrid query that combines semantic similarity with structured filters in a single round-trip:def agentic_retrieve(tenant_id, query_text, memory_type=None, top_k=10): """Vector similarity + structured filters in one query.""" qv = get_embedding(query_text) where = ["c.tenantId = @tid"] params = [{"name": "@tid", "value": tenant_id}] if memory_type: where.append("c.memoryType = @mt") params.append({"name": "@mt", "value": memory_type}) params.append({"name": "@topk", "value": int(top_k)}) # TOP accepts a parameter in parameterized queries query = f""" SELECT TOP @topk c.id, c.content, c.memoryType, VectorDistance(c.vector, @embedding) AS similarity FROM c WHERE {" AND ".join(where)} ORDER BY VectorDistance(c.vector, @embedding) """ params.append({"name": "@embedding", "value": qv}) return list(container.query_items( query=query, parameters=params, partition_key=tenant_id))Two things that bite teams here. First, always include a TOP clause: without it, the vector search tries to score far more candidates than you need, burning RUs and adding latency. Second, pass TOP as a parameter in parameterized queries (as shown above) rather than f-string interpolation — Cosmos DB supports TOP @param, which keeps the query cacheable and avoids building strings around user-influenced values.The Agent Memory Toolkit: Stop Building This YourselfEverything above is the do-it-yourself path. The reason this topic deserves fresh attention in 2026 is that Microsoft shipped two purpose-built toolkits on top of Cosmos DB for exactly this workload.The Agent Memory Toolkit (AzureCosmosDB/AgentMemoryToolkit on GitHub, public preview) is a Python SDK — pip install azure-cosmos-agent-memory — that gives your agent both raw conversation history and derived memory: thread summaries, extracted facts, and cross-thread user profiles, all searchable semantically. The distillation pipeline can run in-process (zero extra infrastructure) or in a sibling Azure Durable Functions app that watches the Cosmos DB change feed. Sync and async clients mirror each other.from azure.cosmos.agent_memory import CosmosMemoryClientmemory = CosmosMemoryClient( cosmos_endpoint=COSMOS_DB_ENDPOINT, cosmos_database="ai_memory", cosmos_container="memories", ai_foundry_endpoint=AI_FOUNDRY_ENDPOINT, embedding_deployment_name="text-embedding-3-large", chat_deployment_name="gpt-4o-mini", use_default_credential=True, # Entra ID, no API keys)memory.connect_cosmos() # auto-creates database + containersmemory.upsert_memory(user_id="user-001", thread_id=THREAD, role="user", content="I love Cosmos DB.")memory.process_now(user_id="user-001", thread_id=THREAD)# -> thread summary + fact extraction + user profile updatehits = memory.search_cosmos(user_id="user-001", query_text="Cosmos DB preferences", top=5)If you are on the Microsoft Agent Framework, it gets even shorter: the agent-framework-azure-cosmos-memory package (preview) ships a CosmosMemoryContextProvider you attach to your agent once. Its before_run hook searches the user's memory and injects relevant context; its after_run hook stores new turns and lets the toolkit extract facts and summaries in the background. Memory follows the user across threads and sessions via a stable user_id, with no orchestration code on your side.The companion piece is the Agentic Retrieval accelerator (AzureCosmosDB/AgenticRetrieval): a self-correcting, multi-stage RAG pipeline for questions that need multi-hop reasoning. Instead of one search-and-answer pass, it drafts a preliminary answer, identifies what is missing, decomposes the gap into focused sub-questions, retrieves targeted evidence per sub-question across multiple Cosmos DB containers, and synthesises a grounded final answer. It can optionally use the Cosmos DB semantic reranker to reorder results before synthesis. For high-stakes domains — legal, financial, medical — that completeness loop is exactly what a naive single-shot RAG pipeline cannot deliver.And for agents that speak MCP, the Cosmos DB MCP Toolkit (AzureCosmosDB/MCPToolKit) is a production-ready MCP server with Entra ID authentication, document operations, vector search, hybrid search with reciprocal rank fusion, and schema discovery — deployable to Azure Container Apps in minutes.Where You Still Build Your OwnThe toolkits are preview, Python-first, and opinionated. You still want the manual pattern from the previous section when:Your stack is .NET or Java today (the SDKs will catch up; the DIY path works now).You need a partition key or retention policy the toolkit's defaults do not match.Your compliance posture requires you to own every line of data-access code.In those cases the reference architecture is straightforward: documents arrive from SharePoint, Blob Storage, or databases; an ingestion pipeline (Azure Functions works well) chunks and embeds them into Cosmos DB with provenance metadata; the agent's tool-calling loop invokes retrieval only when reasoning requires it; and the change feed fans new memories out to anything else that needs them.Pitfalls I See Repeatedly1. Embedding everything without a filtering strategy. Teams dump every conversation turn into the vector store. After a few weeks, retrieval quality drops because the index is polluted with trivial exchanges. Classify memories at write time. Store only meaningful interactions and assign TTLs per type — conversation turns might expire in 7 days while extracted facts persist. And remember: per-item ttl only takes effect when TTL is enabled at the container level.2. Over-relying on vector similarity. A user asking "what did the finance team approve last quarter?" needs temporal filtering, not just semantic matching. Always combine vector search with structured filters — that hybrid query is Cosmos DB's actual superpower over standalone vector databases.3. Getting the partition key wrong. Partition by document ID instead of tenant ID and your cross-tenant queries become expensive fan-outs, and you lose isolation guarantees. Partition by the highest-level isolation boundary.4. Ignoring the 1,000-vector index threshold. Below it, quantizedFlat and diskANN silently fall back to brute force. It is fine for small stores, but if your latency numbers look odd at low volume, that is why.5. Forgetting cost mechanics. Cosmos DB bills per RU. Vector queries cost more than point reads, so always use TOP, filter by partition key where possible, watch RU consumption in Azure Monitor, and consider serverless capacity mode for agent workloads with spiky traffic — which is most of them.6. Not using the change feed. When a new memory is written, the change feed can trigger downstream updates — search indexes, caches, monitoring. The Agent Memory Toolkit's Durable Functions processor is built exactly this way.Key TakeawaysCosmos DB collapses the agent memory stack. One container handles conversation state, long-term facts, and RAG retrieval — no separate vector and relational databases to stitch together.Hybrid queries are the differentiator. Vector similarity combined with SQL-style filters (tenant, user, time range, type) in a single query is what makes it viable for enterprise agent workloads.Use the toolkits before building your own. The Agent Memory Toolkit, Agentic Retrieval accelerator, and MCP Toolkit are in public preview and encode patterns most teams otherwise rebuild badly.Get the data model right. Partition by tenant, define the vector embedding policy at container creation (it cannot be modified in place), enable TTL at container level, and always include a TOP clause in vector queries.Watch the RU meter. Vector search at scale is real money. Classify memories, expire what does not matter, and alert on cost anomalies early.The enterprise agents that succeed in production will be the ones with a solid data foundation underneath. Cosmos DB now gives you that foundation — with the agent-specific tooling on top — without the operational overhead of stitching together multiple specialised databases.This article is part of my ongoing series on building production AI systems on Azure. If you are working on agent architectures in the Southeast Asian enterprise space, feel free to reach out — I am always happy to compare notes.Follow me on LinkedIn or read more at wenfeng.my/. --- ## Azure Local Disconnected Operations: Sovereign Cloud for Malaysian Regulated Industries URL: https://wenfeng.my/azure-local-disconnected-operations-sovereign-cloud-for-malaysian-regulated-industries/ Author: admin Published: 2026-08-25 06:30:48 Modified: 2026-10-04 15:26:21 Azure Local Disconnected Operations: Sovereign Cloud for Malaysian Regulated Industries For years, Malaysian regulated industries faced an uncomfortable trade-off: you could have Azure's management model, or you could keep every byte of data inside your own walls — but not both. Azure Local (formerly Azure Stack HCI) ran on your hardware, yet its control plane still reached out to Azure for management, updates, and identity. For a BNM-supervised bank's core systems, a hospital's patient records, or a classified government enclave, that outbound dependency was often a non-starter. Disconnected operations for Azure Local removes that dependency entirely. The Azure control plane — portal, ARM, Azure Policy, RBAC, Key Vault — now runs as a virtual appliance inside your own data center, with no ongoing connection to Azure or the internet required. This article explains what actually changed, who in Malaysia should care, what it costs, and where the sharp edges are. What Disconnected Operations Actually Changes Before this capability, "Azure on-premises" still meant "Azure-attached." Your VMs ran locally, but the management plane phoned home. Disconnected operations flips that model: Local Azure portal — the familiar Azure portal experience, served from your infrastructure. Azure Resource Manager (ARM) — subscriptions, resource groups, ARM templates, and CLI, all running locally. RBAC and managed identity — the same access control model you use in public Azure. Azure Policy — enforce compliance guardrails at resource creation, on-premises. Azure Key Vault — secrets and certificates managed locally. Arc-enabled servers and AKS (preview) — manage VM guests and Kubernetes clusters without cloud connectivity. Microsoft positions this as part of a broader sovereign private cloud stack: Azure Local for infrastructure, Microsoft 365 Local for on-premises Exchange, SharePoint, and Skype for Business (Microsoft has committed support for the subscription editions of these server products through at least 2035), and Foundry Local for on-premises AI inference with OpenAI-compatible APIs. For an organization that needs a fully air-gapped environment, all three layers now exist. Who Should Evaluate This in Malaysia Not every enterprise needs disconnected operations — and Microsoft makes that explicit. Procurement requires a valid business need for operating disconnected (connectivity limitations or regulatory restrictions), an eligible agreement (standard MOSA subscriptions do not qualify), and an active support plan. This is a sovereign/regulated play, not a general-purpose on-prem refresh. Banking and financial services (BNM-supervised). Most Malaysian banks run hybrid strategies: core banking and sensitive customer data stay on-premises, less sensitive workloads go to cloud. Disconnected operations gives the on-premises half Azure-native management — same templates, same policies, same skills — while keeping the data boundary physically inside the bank's control. Healthcare (MOH-regulated). Patient data handling requirements plus the latency needs of clinical systems keep hospital workloads on-premises. Add Foundry Local (currently in preview) and you get local AI inference — for example, running imaging or triage models next to the data that must never leave the hospital network. Government, defense, and critical infrastructure. Microsoft's own reference scenarios include remote sites like oil rigs, manufacturing plants, and classified environments. The production control plane requires only three physical servers, which makes this feasible for secure facilities and branch data centers, not just national-scale deployments. The Decision Framework: Malaysia West vs Disconnected Azure Local The real question is never "cloud or on-prem?" — it is "which workloads go where?" Consideration Malaysia West (public cloud) Azure Local disconnected Data location Malaysia West region Your facility Connectivity Internet required None required after setup Management Microsoft-operated You operate the control plane Cost model Pay-as-you-go Flat monthly per physical core, annual term, billed via Azure Identity Entra ID Local AD DS + ADFS Updates Microsoft-managed You control timing (offline/staged workflows) Best for Web, analytics, dev/test, DR target Sovereign workloads, classified data, strict latency For most Malaysian enterprises, the practical architecture is hybrid: public Malaysia West for web apps, analytics, dev/test, and as a DR target; disconnected Azure Local for core banking, clinical systems, classified workloads, and AI inference over sensitive data. The win is operational — both sides use the same ARM templates, CLI, and policy model, so you are not running two unrelated stacks. Practical Examples 1. Deploying the control plane (PowerShell) The control plane runs on a dedicated three-node management cluster (production minimum: 3 nodes, 512 GB RAM, 24 physical cores, and 8 × 2 TB drives per node). Deployment starts on the seed node by staging the appliance files and running the disconnected operations module: # Stage appliance + certificates on the seed node $applianceConfigBasePath = 'C:\AzureLocalDisconnectedOperations' Copy-Item \\fileserver\share\azurelocalfiles\* $applianceConfigBasePath Expand-Archive "$($applianceConfigBasePath)\AzureLocal.DisconnectedOperations.zip" ` -DestinationPath $applianceConfigBasePath Import-Module "$applianceConfigBasePath\OperationsModule\Azure.Local.DisconnectedOperations.psd1" -Force # Ingress network configuration for the appliance $ingressNetworkConfigurationParams = @{ DnsServer = "192.168.200.150" IngressNetworkGateway = "192.168.200.1" IngressIpAddress = "192.168.200.115" IngressNetworkPrefixLength = 24 ExternalDomainSuffix = "contoso.my" } $ingressNetworkConfiguration = New-ApplianceIngressNetworkConfiguration @ingressNetworkConfigurationParams # Install the appliance (identity via ADFS, certificates for 23 ingress endpoints) Install-AzureLocalDisconnectedAppliance @installAzureLocalParams Note the identity wiring: you configure ADFS as the authority (plus LDAPS if you use it), and you need certificates for 23 ingress endpoints plus 2 management endpoints. Plan your PKI before anything else. 2. Managing resources locally (Azure CLI) Once deployed, day-2 operations use the Azure CLI you already know — pointed at the local ARM endpoint. Supported CLI version is 2.81.0, and Azure Local nodes require the 32-bit CLI build (64-bit is for client machines): # Point the CLI at the local ARM endpoint instead of public Azure az login az cloud update --profile latest \ --endpoint-resource-manager "https://management.contoso.my" # Create a resource group and deploy a VM — same verbs as public Azure az group create --name rg-core-banking --location "contoso-dc1" az connectedmachine machine create \ --resource-group rg-core-banking \ --name vm-core-app-01 \ --location "contoso-dc1" 3. ARM/Bicep templates work unchanged Because the local control plane speaks ARM, your existing Bicep templates deploy against it. A Key Vault for local secrets, for example: // keyvault.bicep — deploys against the LOCAL ARM endpoint param location string = resourceGroup().location resource kv 'Microsoft.KeyVault/vaults@2023-07-01' = { name: 'kv-sovereign-01' location: location properties: { tenantId: subscription().tenantId sku: { family: 'A', name: 'standard' } enableRbacAuthorization: true } } az deployment group create \ --resource-group rg-core-banking \ --template-file keyvault.bicep 4. Terraform for hybrid estates For teams standardized on Terraform, the same pattern applies — register the appliance as a custom cloud and manage Azure Local VM resources through the AzAPI provider. A typical pattern for the connected management side (billing registration, update orchestration staging): # main.tf — hybrid: Terraform manages the Azure side; # local resources go through the CLI against the appliance endpoint terraform { required_providers { azurerm = { source = "hashicorp/azurerm", version = "~> 4.0" } azapi = { source = "Azure/azapi", version = "~> 2.0" } } } provider "azurerm" { features {} subscription_id = var.subscription_id } resource "azurerm_resource_group" "hub" { name = "rg-sovereign-hub" location = "malaysiawest" } # Example: track the disconnected cluster as an Arc-connected resource # once you choose to bridge selected telemetry back to Azure. resource "azapi_resource" "local_cluster" { type = "Microsoft.AzureStackHCI/clusters@2024-04-01" name = "cluster-contoso-dc1" parent_id = azurerm_resource_group.hub.id body = jsonencode({ location = "malaysiawest" properties = { cloudConnection = "None" } }) } 5. Local AI inference via Foundry Local (Python) Foundry Local on Azure Local (preview, request-based access) exposes OpenAI-compatible REST endpoints on an Arc-enabled Kubernetes cluster. Application code barely changes: from openai import OpenAI # Endpoint lives inside your network — no internet egress client = OpenAI( base_url="https://foundry.contoso.my/v1", api_key="***", # local API key, not a cloud credential ) response = client.chat.completions.create( model="phi-4", messages=[ {"role": "system", "content": "You are a clinical document summarizer."}, {"role": "user", "content": "Summarize this discharge note..."}, ], ) print(response.choices[0].message.content) The model weights, the prompts, and the responses never leave your cluster — which is precisely the point for patient data or classified material. Pitfalls: Where Deployments Go Wrong Azure Hybrid Benefit does not waive the disconnected core fee. This is the most common misconception. With disconnected operations, you pay the flat per-physical-core fee regardless of Windows Server licensing — AHB is not available for Azure Local in disconnected mode. AHB only applies to the Windows Server VMs running on it (and requires eligible licenses with active Software Assurance). Budget the core fee; do not assume existing Datacenter SA makes it $0. Eligibility is gated. You need an eligible agreement (MOSA is excluded), an active support plan (Standard or higher), and a demonstrable business need for disconnection. Talk to your Microsoft account team before sizing hardware. The control plane is not free real estate. Production requires a dedicated 3-node management cluster, isolated from tenant workloads, sized per Microsoft's minimums (512 GB RAM / 24 cores / 8 × 2 TB drives per node). Don't plan to co-locate workloads on it — Microsoft explicitly forbids that. PKI is the long pole. You need certificates for 23 ingress endpoints and 2 management endpoints, plus ADFS certificate chains. Air-gapped environments also hit a known issue (release 2605) where the Microsoft Code Signing PCA 2011 certificate must be manually downloaded and imported on every node before cloud deployment succeeds. Build your offline certificate workflow early. Updates are your problem now. Microsoft ships update packages for offline/staged application. The control plane can experience downtime during node reboots and updates — plan maintenance windows the way you did before the cloud existed. Preview components have sharp edges. Arc-enabled Kubernetes and AKS on Azure Local in disconnected mode are preview; Foundry Local on Azure Local is preview with request-based access. Keep preview components out of your compliance-critical path until GA. CLI quirks. Supported CLI version is pinned (2.81.0), nodes need the 32-bit build, and clients must trust the appliance root certificate (pip-system-certs or manual PEM import) or every call fails TLS verification. The VMware Migration Angle Many Malaysian enterprises are still on VMware and urgently evaluating post-Broadcom options. Azure Local is worth a spot on that shortlist: Azure Migrate supports migration to Azure Local, and the replication stays on-premises — VM data does not leave your network during migration. Combined with the same-management-plane story, you get a path off VMware without being forced into a public cloud move your regulators won't approve. Conclusion Disconnected operations turns Azure Local from "cloud-attached infrastructure" into a genuine sovereign private cloud. For Malaysian banks, healthcare providers, and government agencies, it is the first time the Azure operating model has been available with zero data egress — portal, ARM, Policy, RBAC, Key Vault, and even local AI inference, all inside your boundary. The honest caveats: it is capacity-billed per physical core (AHB does not zero it out), it demands a dedicated management cluster and serious PKI preparation, and eligibility is gated to organizations with a real sovereignty need. But if your regulator owns your architecture decisions, this is the most Azure-native answer that exists today. Key takeaways: The Azure control plane now runs on your hardware — portal, ARM, RBAC, Azure Policy, and Key Vault operate locally with no ongoing Azure or internet connection. Production needs a dedicated 3-node management cluster (512 GB RAM / 24 cores / 8 × 2 TB per node minimum), isolated from tenant workloads. Budget the per-core fee honestly — disconnected operations is billed monthly per physical core on an annual term via Azure, and Azure Hybrid Benefit for Azure Local is not available in disconnected mode (AHB only covers the Windows Server VMs on top). Eligibility is gated — eligible agreement, active support plan, and a documented business need to operate disconnected; MOSA subscriptions don't qualify. Hybrid remains the practical architecture — Malaysia West for non-sensitive workloads and DR, disconnected Azure Local for sovereign-critical systems, both run with the same tools and templates. Azure Local disconnected operations is not just another on-premises option — it is the first time Azure's management model has been available with zero data egress. For Malaysian regulated industries, that changes what sovereignty can look like. --- ## Azure Functions Serverless Agents Runtime — Build AI Agents as First-Class Serverless Workloads URL: https://wenfeng.my/azure-functions-serverless-agents-runtime/ Author: admin Published: 2026-08-24 23:30:38 Modified: 2026-10-04 15:26:32 If you've been deploying AI agents in production on Azure, you've probably faced the same dilemma I've seen across dozens of enterprise engagements in Southeast Asia: containers feel like overkill for most agent workloads, but you still need reliability, event-driven triggers, and proper tool orchestration. Standing up an AKS cluster just to host a few dozen simple tool-calling agents is like renting a warehouse to store a bicycle.Microsoft's answer is the Azure Functions Agents Runtime — currently in public preview, published on PyPI as azurefunctions-agents-runtime, and open-sourced on GitHub under Azure/azure-functions-agents-runtime. It is a markdown-first programming model for building AI agents on Azure Functions, powered by the Microsoft Agent Framework (MAF). You define agents in .agent.md files, deploy them as event-driven serverless functions, and let the runtime wire up the LLM, tools, and connectors for you.I've been experimenting with it since it appeared, and I want to walk you through what it actually means for enterprise teams — what's real, what's still experimental, and the patterns that make sense in production.The Problem: Agents Don't Fit Existing Hosting Models CleanlyMost enterprise agents fall into one of two buckets. The first is the heavyweight, long-running agent — autonomous research agents, multi-step pipelines, orchestrators coordinating dozens of tools. Those need persistent state, retries, and durable execution, and Container Apps, AKS, or hand-rolled Durable Functions remain reasonable homes for them.The second bucket — where most enterprise agents actually land — is the event-driven agent: it reacts to a trigger, executes a bounded task, and produces a result. An agent that watches a document library, extracts structured data from new files, and writes results to a database. An agent that consumes messages from a queue, classifies intent, and routes each one downstream. An agent that answers a webhook with a retrieval-backed response.These agents don't need a persistent container or a cluster. They need an event source, a runtime, a tool interface, and an LLM endpoint. Until now, you either forced them into Functions as plain code (losing the agent loop, session state, and tool wiring you'd then rebuild by hand) or you over-provisioned compute. The agents runtime closes that gap: the function is your agent, the trigger is its perception layer, and the tools are declared in markdown and config files.The .agent.md Programming ModelThe distinctive piece is the agent file. Each agent is a markdown file with YAML frontmatter — trigger configuration, model overrides, tool bindings — followed by the system prompt as the markdown body. Here is a real, minimal agent from the runtime's own documentation: --- name: Queue Message Processor description: Processes messages from an Azure Storage queue. trigger: type: queue_trigger args: queue_name: agent-input connection: AzureWebJobsStorage --- You process one Azure Storage Queue message at a time. The runtime provides the message as JSON, including `body`, `body_encoding`, and queue metadata such as `id` and `dequeue_count`. When the message body is valid JSON, use `body_json` when it is present. Produce a concise structured summary with the message contents, any implied action, and the relevant queue metadata. That single file is the agent's identity, trigger binding, and behavioral contract. The runtime discovers it, registers it as an Azure Function, and hands every invocation to the LLM with the configured tools. The function app entry point is two lines: # function_app.py from azure_functions_agents import create_function_app app = create_function_app() Two-tier configuration keeps things clean: agents.config.yaml holds app-wide defaults (model, timeout, system tools), while each agent's frontmatter overrides them and filters the capabilities it sees. MCP servers live in mcp.json, custom Python tools in tools/ (decorate functions with @tool and they become callable), and progressive-disclosure prompt modules in skills/. Model providers are pluggable — Microsoft Foundry, Azure OpenAI, or OpenAI — selected via the AZURE_FUNCTIONS_AGENTS_PROVIDER setting, with sensible auto-detection.What I appreciate most is that this makes agents reviewable. In enterprise environments, a human-readable agent definition that security teams, compliance officers, and domain experts can read and approve beats opaque code-only configuration. I've sat through incident reviews in Malaysian financial services projects where the agent's actual behavior was unclear even to the team that shipped it. A markdown contract plus declarative triggers eliminates most of that ambiguity before it ships.The Trigger EcosystemBecause each .agent.md file registers as a real Azure Function, the agent can run on the Functions trigger surface. The supported list includes HTTP, timer, queue, blob, Event Grid, Event Hub, Service Bus (queue and topic), Cosmos DB, Azure SQL, MySQL, Kafka, Dapr, and a generic escape hatch — plus connector_trigger, which plugs into the Azure Functions Connector Extension for connector-triggered apps. One rule to note: one trigger per agent file.The 1,400+ connector story works on both sides of the agent loop. Connector actions (Office 365, Teams, SQL, Salesforce, SAP, and hundreds more) are exposed to agents as connector-backed MCP servers declared in mcp.json, while connector_trigger lets agents fire on connector events. Triggers you'd normally hand-wire become a few lines of YAML in frontmatter.Your First Agent, End to EndThe getting-started flow is refreshingly short. With Python 3.13+ and Azure Functions Core Tools installed: pip install "azurefunctions-agents-runtime[monitor]" Create the project layout: my-agent-app/ ├── function_app.py # create_function_app() entry point ├── agents.config.yaml # app-wide defaults (model, timeout) ├── main.agent.md # the agent definition ├── host.json # extension bundle [4.*, 5.0.0) ├── local.settings.json # provider + model env vars └── requirements.txt agents.config.yaml sets shared defaults: model: $FOUNDRY_MODEL timeout: 900 For local development against Microsoft Foundry, az login and then configure local.settings.json: { "IsEncrypted": false, "Values": { "FUNCTIONS_WORKER_RUNTIME": "python", "AzureWebJobsStorage": "UseDevelopmentStorage=true", "AZURE_FUNCTIONS_AGENTS_PROVIDER": "foundry", "FOUNDRY_PROJECT_ENDPOINT": "https://<project>.<region>.services.ai.azure.com/api/projects/<project>", "FOUNDRY_MODEL": "gpt-5.4" } } Non-HTTP triggers and the MCP endpoint need storage, so run Azurite locally, then func start. Your agent is immediately reachable: a built-in chat UI at /agents/main/, a chat API at /agents/main/chat (plus an SSE chatstream variant), and an MCP tool endpoint at /runtime/webhooks/mcp — all with zero extra code. That built-in surface alone changes how you demo agents to stakeholders.For deployment, the repo's samples use the Azure Developer CLI with Bicep: azd up provisions the resource group, a Flex Consumption function app (FC1), a Microsoft Foundry project with the model deployment, storage, and managed identity in one shot. The sample Bicep pins the region list because Flex Consumption and the default Foundry model deployment must both be available — a realistic constraint worth checking for your own region, especially here in Southeast Asia.Durable Agent Workflows — The Part I'd Been Waiting ForThis is where it gets genuinely interesting for enterprise workloads. The runtime ships an experimental feature called Dynamic Workflows: flip workflows.enabled: true in an agent's frontmatter and the agent gains five built-in tools — start_workflow, get_workflow_status, list_workflows, cancel_workflow, terminate_workflow — that author and run durable DAGs on Azure Durable Functions without you writing orchestration code.The LLM authors a plan of tasks (tool calls, wait durable timers, and sub_agent invocations) with depends_on edges; the runtime validates the DAG, schedules it as a Durable orchestration, and returns a workflow_id immediately. Intermediate results stay inside the orchestration — the agent only ever ingests the final result envelope. That's the token, latency, and context-window discipline of programmatic tool calling, plus durability that survives worker restarts and long sleeps.Workflow-capable tools are ordinary Python with an explicit opt-in marker: # tools/incident_tools.py from typing import Any from azure_functions_agents import workflow_tool @workflow_tool(description="Fetch recent log lines for a service.") def fetch_logs(args: dict[str, Any]) -> dict[str, Any]: return {"service": args["service"], "lines": ["..."]} And a plan the agent might author looks like this: { "tasks": [ { "id": "fetch_logs", "type": "tool", "tool": "fetch_logs", "args": {"service": "payments"} }, { "id": "fetch_metrics", "type": "tool", "tool": "fetch_metrics", "args": {"service": "payments"} }, { "id": "cool_down", "type": "wait", "duration": "PT30S", "depends_on": ["fetch_logs", "fetch_metrics"] }, { "id": "summarize", "type": "tool", "tool": "summarize", "args": {"sources": ["${fetch_logs.result}", "${fetch_metrics.result}"]}, "depends_on": ["cool_down"] } ] } For multi-agent setups, any runnable agent can declare subagents: in its frontmatter — each reference becomes a delegate_<slug> tool on the coordinator, with no handoff builders and no new dependencies. Delegation is single-level by design, and specialists only receive the self-contained task string the coordinator passes them.Decision Framework: Functions vs Containers vs Hand-Rolled DurableAfter deploying agents on Functions, Container Apps, and AKS, here's how I'd frame it:Choose the agents runtime on Functions when: the agent is triggered by discrete events (queue, blob, timer, webhook), execution is bounded, you want scale-to-zero economics, and the workflow shape is "reason, act, respond" — optionally with a durable workflow for the heavy lifting.Choose Container Apps or AKS when: you need persistent in-memory state between invocations, GPU access, long-lived WebSocket connections, custom networking boundaries, or a microservices mesh.Choose hand-written Durable Functions when: you need deterministic, code-authored orchestrations with fine-grained control over retries, entities, and external events — and you're prepared to own that code.One important clarification I'd offer: Dynamic Workflows are not a replacement for hand-written orchestrations. Plans are authored by the LLM, not hand-coded in YAML, and v1 deliberately omits per-task timeout/retry fields (tracked for v2). If your compliance posture demands a deterministic, reviewable orchestration, keep it in code. If you want a durable, observable execution layer that an agent can drive, workflows are the point.For most enterprise agent use cases I encounter in Malaysia and ASEAN — document processing, email triage, scheduled reporting, webhook integrations — the agents runtime is now my default starting point.Pitfalls I've Hit and Ones I'd FlagIt's public preview, moving fast. The package went from 0.0.0 dev releases to 0.1.0 beta in a few months, requires Python 3.13+, and the README explicitly warns features may change before GA. Pin versions, read the changelog before upgrading, and don't bake preview APIs into compliance commitments.One trigger per agent file. If you want the same logic on two event sources, you have two agent files (or one agent plus delegation). Structure your agents/ folder for that from day one.Workflows are fire-and-forget. start_workflow returns immediately with an ID; the agent does not block on completion. Design for the completion envelope coming back later — via the chat UI's auto-notification, or by polling get_workflow_status from an external operator. Trigger-started workflows on non-HTTP triggers get a fresh session ID with no app-wide index, so publish the terminal result somewhere operators can find it.Secure the endpoints deliberately. The chat API defaults to function-key auth (http_auth: function), which is fine for a demo but not for production. Set builtin_endpoints.http_auth: entra for Entra ID enforcement, or put the app behind API Management or private endpoints. "Obscurity through URL complexity" is not a control.Mind the hosting constraints. Flex Consumption works well with queue, timer, and Service Bus triggers, but blob triggers have known Flex limitations (the sample docs steer you to Event Grid-based wiring for blobs). Check trigger support on your chosen SKU before you commit to an architecture.Watch token consumption, not just invocations. Standard Functions metrics don't capture LLM cost. Install the runtime with the [monitor] extra so OpenTelemetry spans and metrics flow to Application Insights automatically, and build your cost dashboards around token usage.ConclusionThe Azure Functions Agents Runtime is the first time Azure has treated an AI agent as a genuine first-class serverless workload: markdown definitions, event triggers, 1,400+ connector-backed tools, built-in chat and MCP endpoints, session persistence, and durable workflows — all on the hosting platform most enterprises already run. It won't replace containers for heavyweight agents or hand-written Durable Functions for deterministic orchestrations, but for the broad middle of enterprise agent patterns, it removes almost everything that used to be glue code.For Southeast Asian teams scaling agent programs in 2026, this meaningfully lowers the barrier: start with a markdown file, a trigger, and a Foundry model. Ship the simple version first. Add complexity only when requirements genuinely demand it — because the simplest architecture is usually the one that actually ships.Key TakeawaysAgents become markdown. .agent.md files combine trigger configuration and system prompt in one reviewable artifact, deployed as ordinary Azure Functions — your existing CI/CD, monitoring, and security tooling apply unchanged.The trigger surface is the perception layer. HTTP, timer, queue, blob, Event Grid, Service Bus, Cosmos DB, SQL, Kafka, Dapr, and connector triggers are declared in frontmatter; 1,400+ connector actions are reachable through connector-backed MCP servers.Dynamic Workflows bring durable execution to agents — one frontmatter flag gives the agent LLM-authored DAGs on Durable Functions that fan out, wait, survive restarts, and keep intermediate results out of the context window. It's experimental; use it with eyes open.Built-in endpoints are the hidden productivity win. Chat UI, chat API, SSE streaming, MCP tool registration, and Blob-backed session persistence come with a flag — no extra code.Start serverless, treat preview pragmatically. Pin versions, verify regional availability of Flex Consumption and your model deployment, use Entra auth for production endpoints, and reserve containers and hand-written orchestrations for cases that truly need them. --- ## Frontier Tuning: Microsoft's Enterprise RL Framework That Teaches AI Your Business — And It Works URL: https://wenfeng.my/frontier-tuning-microsofts-enterprise-rl-framework-that-teaches-ai-your-business-and-it-works/ Author: admin Published: 2026-08-24 06:30:17 Modified: 2026-10-04 15:30:54 There's a persistent frustration with enterprise AI: it knows everything about the world but nothing about how your organization actually works. It can explain transformer architecture in detail but can't figure out that your approval chain requires three sign-offs for purchases over MYR 50,000, or that your engineering team uses "ship" to mean "deploy to staging" while "release" means "deploy to production." At Build 2026, Microsoft introduced Frontier Tuning — a reinforcement learning framework that teaches AI models not just what your data says, but how your business operates. And the early results are genuinely striking: Microsoft's own HR team saw successful task completion jump from 13% to 87%, and EY is tuning an advisory agent for deployment to 75,000 tax professionals globally. This isn't another RAG pipeline or fine-tuning service. It's architecturally different — and for enterprises evaluating their AI strategy, it's worth understanding exactly what it is, what the evidence says, and what the honest limitations are. The Problem: Context Without Behavior For the past two years, enterprise AI has been about feeding models data — wiring them into documents, databases, and internal knowledge systems through RAG and fine-tuning. Microsoft's argument is that this is only half the story. Consider what each approach actually teaches a model: Traditional fine-tuning adjusts weights on a curated dataset. It teaches domain language, style, and task patterns — the WHAT of your business. It does not teach process. RAG retrieves relevant documents at query time. It gives the model context and reduces hallucination for factual queries. It does not teach behavior — the model can read your procurement policy but doesn't know how procurement actually operates in practice. The gap between reading a policy and executing the workflow behind it is exactly where enterprise agents fail. And the failure mode isn't dramatic — it's a slow bleed of mediocre answers that feel like they came from an intern who read the handbook yesterday. How Frontier Tuning Actually Works Frontier Tuning has three interconnected parts that form a continuous improvement loop, as described by Ranveer Chandra, Microsoft's Vice President of Frontier Tuning, in the official announcement. 1. The Reinforcement Learning Environment (RLE) The RLE is the core innovation: a managed reinforcement learning environment used for both post-training and inference. During training, the system learns from real workflows, tool usage, and evaluation signals — without affecting production systems. At inference, it explores multiple frontier and fine-tuned models from Microsoft AI and OpenAI across turns, searching for stronger candidate paths before returning an answer. Continuously, it improves as it learns from each interaction. The key distinction: the RLE doesn't just adjust weights against a static dataset. It learns from the dynamic interplay between agents, tools, and users — capturing the implicit knowledge embedded in how work actually gets done. 2. Your Data, Domain Knowledge, and Workflows You bring your business into the RLE: content, processes, conventions, terminology, and workflows. Microsoft explicitly designed this to require no data science expertise — a guided approach where teams bring data in and start tuning. In Copilot Studio, that will include transcripts, knowledge bases, and Microsoft 365 artifacts. 3. Tuned Outputs Within Your Compliance Boundary The system produces tuned models, embeddings, skills, orchestration logic, and a runtime harness. All of it runs on your data with your controls: Models inherit your access controls — only people who could see the underlying data can access models built from it. Tools are virtualized, so agents can improve without affecting production systems. Nothing leaves your compliance boundary. The Evidence: What the Early Customers Report Frontier Tuning is already in the hands of a focused set of customers: Land O'Lakes, EY, Bristol Myers Squibb, Pearson, McKinsey, McCarthy Tétrault, and the Josh Bersin Company. The reported results, in their words: Microsoft HR (internal). Nathalie D'Hers, CVP Employee Experience: "We're teaching the system how Microsoft HR works — capturing organizational knowledge in one connected environment that learns and improves with every use. We partnered with our product teams until the results were undeniable; successful task completion increased from 13% to 87%. Now we're expanding to more HR workflows." EY. Ben Ambrosino, EY Global Tax CTO: EY is combining a tax-domain-tuned reasoning LLM with its enterprise knowledge and tuning an advisory agent within the RLE, leveraging client context in Microsoft Work IQ, to be deployed to 75,000 tax professionals globally in the coming period — and describes Frontier Tuning as "set to transform the tax practice across the global EY organization." Pearson. Gian Paolo Perrucci, Product & Technology Officer: Frontier Tuning generated "significantly better Copilot outputs for Communication Coach," more closely aligned with Pearson's learning science, giving learners "clearer, more actionable feedback." The Josh Bersin Company. Josh Bersin: it brings "Galileo's research-backed HR intelligence into bespoke agents inside the Copilot experience... one of the most compelling capabilities we have seen for putting deep domain expertise into the daily flow of work." A note of professional caution: these are early engagements reported by Microsoft and its partners, with EY's full rollout explicitly still ahead. But the 13%→87% Microsoft HR number is the kind of concrete metric that separates this from vaporware announcements. The Analyst View: Muscle Memory — and a Warning The sharpest framing comes from Ashish Chaturvedi, leader of executive research at HFS Research, who positions Frontier Tuning alongside Microsoft's IQ context layer (Work IQ, Fabric IQ, Foundry IQ): "The IQ offerings give agents the 'map', comprising organizational knowledge, ontologies, and real-time signals about how the business works. On the other hand, Frontier Tuning gives agents the 'muscle memory', reinforcement learning that trains the model to behave the way your organization actually operates." "An agent that has context (IQ) but generic behavior produces decent answers. An agent that has both context and tuned behavior, including understanding your terminology, your approval chains, your style guides, and your compliance conventions, would produce answers that feel like they came from a seasoned employee. That's the added value." The warning comes from Stephanie Walter, practice lead of the AI stack at HyperFRAME Research: Microsoft's portfolio is getting complex, and there's a real risk of decision paralysis. "CIOs will need clear guidance on when to use Work IQ, Fabric IQ, Foundry IQ, Web IQ, RAG, fine-tuning, and Frontier Tuning. The risk is that more choice becomes more architectural ambiguity unless Microsoft makes the decision path very clear." Her second point is the one architects should internalize: Frontier Tuning does not replace prompt engineering, RAG, or fine-tuning. "It is another layer for higher-value workflows where basic grounding is not enough. Developers should think of it as agent behavior tuning, not just model tuning." What You Can Actually Do Today Here's the honest part: Frontier Tuning is in private preview, available through Microsoft's Forward Deployed Engineering (FDE) program. There is no public CLI, no ARM template, no self-serve portal for it yet — the official entry point (aka.ms/frontiertuning) is a registration form. Copilot Studio and Microsoft Foundry integrations are announced as upcoming. So any code you write today is about preparing the ground. First, the model deployment layer that Frontier Tuning will sit beside — verified against the official Azure CLI reference: # Deploy the foundation model your agents run on (Azure CLI, official syntax) az cognitiveservices account deployment create \ -g rg-ai-foundry \ -n oai-prod \ --deployment-name gpt-4o \ --model-name gpt-4o \ --model-version "2024-11-20" \ --model-format OpenAI \ --sku-name "GlobalStandard" \ --sku-capacity 40 Second — and this is the part your team can genuinely build now — is the evaluation harness. The FDE engagement model starts with defining the scenario and setting eval criteria. Organizations that arrive with measurable task-completion metrics get dramatically more out of tuning, because reinforcement learning needs a reward signal. The same harness also gives you the "before" number to prove ROI against (Microsoft HR measured 13% before they ever measured 87%). A minimal version: import json, os from openai import AzureOpenAI client = AzureOpenAI( azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"], api_key=os.environ["AZURE_OPENAI_KEY"], api_version="2024-10-21", ) def run_eval(task: dict) -> dict: """Score one agent task: did it complete the workflow correctly?""" response = client.chat.completions.create( model=task["model"], # the deployment name, e.g. "gpt-4o" messages=[ {"role": "system", "content": task["system_prompt"]}, {"role": "user", "content": task["input"]}, ], ) output = response.choices[0].message.content judge = client.chat.completions.create( model=task["model"], messages=[ {"role": "system", "content": "You are a strict evaluator. Score the answer 0-100 against " "the rubric. Reply as JSON: {\"score\": int, \"reason\": str}"}, {"role": "user", "content": f"Task: {task['input']}\nExpected: {task['expected']}\n" f"Actual: {output}"}, ], response_format={"type": "json_object"}, ) return json.loads(judge.choices[0].message.content) tasks = json.load(open("eval-tasks.json")) results = [run_eval(t) for t in tasks] completion = sum(1 for r in results if r["score"] >= 80) / len(results) print(f"Task completion: {completion:.0%}") # your baseline number Run that weekly against your highest-value workflow. The trend line it produces is both your tuning justification and your tuning proof. Pitfalls and Honest Limitations 1. It's gated, and that shapes procurement. Private preview via FDE means access is relationship-driven, timeline is not self-serve, and pricing is not published. If Frontier Tuning is strategically important to you, register at aka.ms/frontiertuning now and start the conversation — waiting for GA means waiting for everyone else's learning curve to finish first. 2. Don't skip the foundation layers. Walter's warning applies: if your RAG grounding is weak and your prompts are unstructured, tuning behavior on top of that compounds the mess. Get Work IQ / grounding, retrieval, and prompting solid first. Frontier Tuning is the top layer of the stack, not a substitute for the bottom of it. 3. The naming complexity is a real operational risk. Between Work IQ, Foundry, Copilot Studio, Fabric, and Frontier Tuning, internal teams can burn months in architecture debates. Insist on a written decision framework: which layer owns which problem, for which workflow. 4. Access-control inheritance cuts both ways. Tuned models inheriting your permission model is excellent for governance and PDPA alignment — but it means your Entra ID hygiene directly determines what tuned agents can see. Messy RBAC in, messy agent behavior out. 5. Pick workflows by value, not novelty. The ROI case is strongest for high-value, high-frequency workflows with clear behavioral patterns and measurable outcomes (HR service desks, tax advisory, compliance reviews). For low-frequency or poorly-defined processes, the investment won't justify itself — standard RAG will serve you better. What This Means for Malaysian Enterprises Three things stand out for the organizations I work with: Compliance posture. Tuned models staying inside your compliance boundary, inheriting your access controls, matters enormously for PDPA 2010 and for regulated sectors. The architecture is designed for exactly the "learn from our data without it leaving our environment" requirement Malaysian enterprises consistently raise. The seasoned-employee problem. Institutional knowledge — the shortcuts, the exceptions, the unwritten rules — walks out the door every time a senior employee resigns. Behavioral tuning is the first mainstream mechanism for capturing how your senior people work, not just the documents they leave behind. Region reality check. Frontier Tuning availability will follow Microsoft's rollout sequencing. Malaysian enterprises should engage through their Microsoft account team and FDE program early rather than assuming regional availability at GA. Key Takeaways Frontier Tuning teaches behavior, not just knowledge. The managed RL environment learns from real workflows, tool usage, and feedback — which is why Microsoft HR went from 13% to 87% task completion. It's agent behavior tuning, not model tuning — and it complements rather than replaces RAG and fine-tuning. Ground first, tune second. Everything stays in your compliance boundary — tuned models inherit your access controls and tools are virtualized, which is the architecture PDPA-conscious enterprises have been asking for. The evidence is real but early. Private preview, FDE-led, with EY's 75,000-user rollout still ahead. Treat reported numbers as directionally strong, not as your guaranteed outcome. Prepare with evals, not hype. Build a task-completion harness for your highest-value workflow now. The organization that can measure its baseline is the one that proves tuning ROI fastest. The next frontier in enterprise AI isn't bigger models or more data — it's teaching AI how work actually gets done in your organization. If the Microsoft HR numbers hold at scale, Frontier Tuning is how that happens. --- ## Azure Container Apps Sandboxes: Run Untrusted AI Agents Safely URL: https://wenfeng.my/azure-container-apps-sandboxes-untrusted-ai-agent-code/ Author: admin Published: 2026-08-23 23:30:03 Modified: 2026-10-04 15:33:19 Here's a question that keeps architects up at night: what happens when your AI agent writes Python code, executes it, and that code ends up running next to your production database credentials?If you're building agent systems — the kind where an LLM generates and runs code dynamically — you've faced this exact problem. The agent needs to execute code to do its job. But that code is untrusted. It might be buggy, manipulated by a prompt injection, or just plain weird. Run it inside your application process and you've essentially handed a language model the keys to your infrastructure.In Azure, the capability built for exactly this scenario is called dynamic sessions in Azure Container Apps. It gives you per-execution, Hyper-V-isolated environments with prewarmed pools that allocate sessions in milliseconds. I've been using this pattern with enterprise clients across Malaysia and Southeast Asia, and in this article I'll walk through how it actually works, how to set it up, and the pitfalls I keep seeing in the field.A quick naming note. In June 2026, Microsoft announced a separate public-preview resource type literally called Azure Container Apps Sandboxes (Microsoft.App/SandboxGroups), positioned as the next evolution of dynamic sessions — adding snapshot-based suspend/resume, lifecycle policies, and per-sandbox egress rules. Microsoft has said dynamic sessions will continue to be supported, but new investment is going into Sandboxes. This article focuses on dynamic sessions: they are broadly available across Azure regions today, and the isolation model, security rules, and operational patterns here transfer directly to the newer Sandboxes resource.The Problem: Why the Obvious Approaches Fall ShortBefore the solution, let's be honest about why the alternatives don't cut it for untrusted code execution.Shared-process execution. Running agent-generated code in your application's own process (or a subprocess on the same host). Fast, simple, and catastrophic from a security standpoint: a prompt-injected script can read environment variables, grab connection strings, and pivot to internal services.VM-per-execution. Bulletproof isolation, but cold starts of 30–90 seconds. Users will not wait that long for an agent to "think," and you'll pay for the whole VM while it idles.Kubernetes pods. Better cold start than VMs, but now you're operating a cluster, node pools, and a scheduling layer just to sandbox snippets of Python. Overkill for most teams.Shared serverless runtimes. Functions platforms give you resource limits, but they were not designed as hard isolation boundaries for hostile code. You want a dedicated, disposable boundary per execution — not a multi-tenant worker.What you actually need is the combination of four properties: strong isolation, millisecond allocation, disposable state, and no infrastructure to babysit. That's precisely the gap dynamic sessions were built to fill.What Dynamic Sessions Actually AreAzure Container Apps dynamic sessions provide secure, sandboxed execution environments that are ideal for running code requiring strong isolation from other workloads. The official documentation lists "safely execute AI-generated code" as the headline scenario.The key properties:Hyper-V isolation. Each session is isolated by a Hyper-V boundary — hardware-level separation, not just Linux namespaces. Sessions are isolated from each other and from the host.Prewarmed pools, subsecond allocation. A session pool keeps ready-but-unallocated sessions warm. When a request arrives, the pool allocates an existing session instead of creating one from scratch — allocation in milliseconds, not seconds.Ephemeral by design. Sessions are short-lived. After a configurable cooldown with no activity, the session is destroyed and resources are cleaned up automatically.Network egress disabled by default. Sessions cannot reach the internet or your internal services unless you explicitly enable egress on the pool.Managed lifecycle. No container orchestration, no node pools, no scaling rules to tune for the sandbox itself.There are two pool types: Code interpreter poolCustom container pool ImageNone — platform-built runtimes (PythonLTS, NodeLTS, Shell)Your own container image Best forLLM-generated Python/JS/shell execution, fastest setupCustom runtimes, libraries, binaries, any TCP protocol EnvironmentRequires nothing extraRequires a workload-profiles Container Apps environment BillingPer allocated session duration, in 1-hour incrementsDedicated plan, based on E16 instances backing the pool For most AI agent code-execution scenarios, the PythonLTS code interpreter pool is where you start. It comes preloaded with the packages agents typically reach for — NumPy, pandas, scikit-learn — so generated data-analysis code just runs.Hands-On: Building the SandboxStep 1 — Create a code interpreter session poolUpdate the CLI and Container Apps extension first (sessions support lives in the preview extension track): az upgrade az extension add --name containerapp --upgrade --allow-preview true -y Then create the pool. Note the network setting — leave egress disabled for untrusted workloads: az containerapp sessionpool create \ --name agent-sandbox-pool \ --resource-group rg-ai-agents \ --location southeastasia \ --container-type PythonLTS \ --max-sessions 100 \ --cooldown-period 300 \ --network-status EgressDisabled The knobs that matter:--max-sessions: maximum concurrent allocated sessions (up to 600).--cooldown-period: idle seconds before a session is terminated, 300–3600. Every request resets the timer.--network-status: EgressDisabled (default) or EgressEnabled.For teams serving Malaysia, southeastasia (Singapore) is a supported region and the closest one — dynamic sessions are available across a long region list including East Asia, Japan, Korea, Central India, and Australia.Grab the pool management endpoint — this is what your agent calls: az containerapp sessionpool show \ --name agent-sandbox-pool \ --resource-group rg-ai-agents \ --query "properties.poolManagementEndpoint" \ --output tsv Step 2 — Grant your agent's identity accessAuthentication uses Microsoft Entra tokens and a purpose-built RBAC role: az role assignment create \ --role "Azure ContainerApps Session Executor" \ --assignee <AGENT_APP_PRINCIPAL_ID> \ --scope <SESSION_POOL_RESOURCE_ID> For direct REST calls, the token must carry an audience (aud) claim of https://dynamicsessions.io: from azure.identity import DefaultAzureCredential credential = DefaultAzureCredential() token = credential.get_token("https://dynamicsessions.io/.default") If you use an LLM framework integration (LangChain, LlamaIndex, Semantic Kernel, AutoGen all have official tutorials), the framework handles token generation for you — you just give the app a managed identity with the role assignment above.Step 3 — Execute code over the REST APIEach request carries an identifier query parameter — a session ID you define (4–128 characters). Same identifier means the same session is reused; a new identifier allocates a fresh one. import requests POOL_ENDPOINT = "https://southeastasia.dynamicsessions.io/subscriptions/<SUB_ID>/resourceGroups/rg-ai-agents/sessionPools/agent-sandbox-pool" def execute_code(code: str, session_id: str) -> dict: resp = requests.post( f"{POOL_ENDPOINT}/executions", params={"api-version": "2025-10-02-preview", "identifier": session_id}, headers={"Authorization": f"Bearer {token.token}"}, json={ "properties": { "codeInputType": "inline", "executionType": "synchronous", "code": code, } }, timeout=240, ) resp.raise_for_status() return resp.json() result = execute_code( "import pandas as pd\nprint(pd.DataFrame({'x': [1, 2, 3]}).describe())", session_id="conversation-4f8a2c", ) Each individual execution is capped at 220 seconds — dynamic sessions are built for short-lived, interactive code, not hour-long batch jobs. Files you upload land in /mnt/data inside the session, and you can upload, list, and download them through the files endpoints (128 MB upload limit).For a LangChain agent, the entire integration is this small: from langchain_azure_dynamic_sessions.tools.sessions import SessionsPythonREPLTool repl = SessionsPythonREPLTool( pool_management_endpoint=POOL_ENDPOINT, session_id="conversation-4f8a2c", ) tools = [repl] The agent now has a code interpreter tool; every snippet it writes executes inside an isolated session instead of your server.Step 4 — When you need more than Python: custom container poolsIf your agents need specific libraries, binaries, or a non-Python runtime, bring your own container: az containerapp sessionpool create \ --name custom-sandbox-pool \ --resource-group rg-ai-agents \ --environment aca-env-workloadprofiles \ --container-type CustomContainer \ --image myregistry.azurecr.io/agent-sandbox:1.0 \ --registry-server myregistry.azurecr.io \ --cpu 0.5 --memory 1Gi \ --target-port 8080 \ --max-sessions 10 \ --ready-sessions 5 \ --cooldown-period 300 \ --network-status EgressDisabled \ --location southeastasia --ready-sessions keeps a warm pool so allocation stays instant even under burst. Custom container pools also support an OnContainerExit lifecycle (the session lives until the container exits or a max alive period) and liveness/startup probes so the pool can replace unhealthy sessions.If you prefer infrastructure-as-code, session pools are ARM resources (Microsoft.App/sessionPools). A trimmed Bicep example for a code interpreter pool: resource sessionPool 'Microsoft.App/sessionPools@2024-08-02-preview' = { name: 'agent-sandbox-pool' location: 'southeastasia' properties: { poolManagementType: 'Dynamic' containerType: 'PythonLTS' scaleConfiguration: { maxConcurrentSessions: 100 } dynamicPoolConfiguration: { executionType: 'Timed' cooldownPeriodInSeconds: 300 } sessionNetworkConfiguration: { status: 'EgressDisabled' } } } The schema is still on preview API versions and evolves — always check the SessionPools REST API reference before pinning this in production. In Terraform, the same body maps directly onto an azapi_resource block.Hardening: The Security Model in PracticeDynamic sessions are explicitly designed to run untrusted code, and the defaults are sane. But a few rules make or break the design:1. Keep egress disabled unless you have a concrete reason. The documentation is blunt about this: if you enable egress, code in the session can reach the internet, which a hostile script can use for exfiltration or denial-of-service. For pure code-interpretation agents, EgressDisabled is the right answer.2. Never put secrets in the session. Microsoft's own guidance is to assume the code is malicious and has full access to the container — including environment variables, files, and anything you upload. Don't inject connection strings into a custom container sandbox "just for this one call." If the agent needs data, fetch it in your trusted orchestration layer and pass only the necessary values into the session.3. Treat session identifiers as credentials. A valid Entra token can create and access any session in the pool; the identifier is what separates one user's session from another's. Generate identifiers cryptographically (never sequential IDs), never expose them in URLs or logs, and ensure each user or conversation can only ever reach its own identifier. If you build one session per conversation, make sure the end user cannot modify the identifier value.4. Leave managed identity access off. Custom container pools can expose the pool's managed identity inside the session, but this is disabled by default for good reason — any code in the session could then mint Entra tokens as your identity. Only enable it with eyes open.5. Remember isolation is per-session, not per-request. Anything inside a single session — files, variables, state — is visible to every subsequent request to that same session. One session per end user or per conversation, not one shared session for everyone.What This CostsTwo billing models, depending on pool type:Code interpreter sessions are billed by allocated session duration, in increments of one hour, from allocation to deallocation. This is the number that surprises people: a session that runs for four minutes and then cools down still bills for one hour. The lever is your cooldown period — keep it as short as your interaction pattern allows (minimum 300 seconds), and reuse sessions aggressively per conversation so you're not allocating a fresh session per message.Custom container sessions run on dedicated E16 compute instances under the Dedicated plan; the pool's nodeCount grows with active and ready sessions. This is a heavier cost model — right for specialized runtimes, overkill for basic Python execution.Check the Azure Container Apps pricing page for current rates in your currency; for a typical interactive agent workload (code interpreter pool, modest concurrency), the sandbox itself is usually a small line item next to the LLM inference costs.Observability: Know What Your Agents RanOne asymmetry to be aware of: code interpreter sessions don't emit session logs to Log Analytics. Execution outputs (stdout and stderr) are returned in the API response, and usage metrics come back as HTTP response headers — so your orchestration layer must capture them at the app boundary. Log the request, the code submitted (sanitized), and the result in your own application telemetry.Custom container sessions do emit AppEnvSession Log Analytics tables when the container writes to stdout/stderr, plus platform logs for pool lifecycle events. Either way, wire alerting on execution error rates and unusual patterns — a sudden spike in failed executions or network-related errors is often the first sign someone is probing your sandbox.Pitfalls I Keep Seeing in the FieldTreating it like a shared scratch space. Teams reuse one session identifier across all users to "save sessions," then discover that files and state leak between tenants. One session per user or conversation, always.Enabling egress for convenience. "The agent needs to pip-install things" is the usual justification. It doesn't — the built-in Python image is preloaded with the common data packages; pre-bake anything else into a custom container instead of opening egress.Ignoring the 1-hour billing increment. Short-lived, throwaway sessions per request multiply your bill. Batch executions into the same session and tune cooldown.Expecting long-running jobs. The 220-second per-execution cap means dynamic sessions are the wrong tool for training runs or long ETL. Use them for the interactive, LLM-in-the-loop execution loop.Skipping adversarial testing. Have someone craft prompt-injection payloads that steer the agent into generating malicious code, then verify the sandbox contains the blast radius. If it escapes, you want to find that in UAT, not production.Forgetting the orchestration layer is your first line of defense. The sandbox is the last line. Input validation, permission scoping on your agent's tools, and review gates for high-risk operations all sit in front of it. Don't skip them just because the sandbox exists.Getting StartedIf you're building agents that execute code in production, here's the path I recommend:Create a PythonLTS code interpreter pool with EgressDisabled in your nearest supported region.Wire it into your agent with the official framework integration for your stack (LangChain, Semantic Kernel, LlamaIndex, AutoGen) or the REST API directly.Use one session identifier per user/conversation, generated cryptographically.Capture stdout/stderr and execution metrics in your own telemetry.Load-test with adversarial prompts before going live.Move to a custom container pool only when the built-in runtimes genuinely can't cover your dependencies.Dynamic sessions give you the pragmatic middle ground between "run everything in one process" and "a VM per execution": hardware-level isolation, millisecond allocation, and nothing to operate. For any team putting code-execution tools in front of an LLM — which, in 2026, is most agent teams — it should be your default answer on Azure.Key TakeawaysAzure Container Apps dynamic sessions are the purpose-built sandbox for untrusted AI-agent code — Hyper-V isolated, prewarmed for millisecond allocation, ephemeral by design.The defaults are secure: egress is disabled out of the box. Only enable it with a concrete, reviewed justification — it's the single biggest risk lever.Never place secrets inside a session and treat session identifiers as credentials — one cryptographically generated identifier per user or conversation, never exposed in URLs or logs.Understand the billing model before you scale: code interpreter sessions bill in 1-hour increments per allocated session, so reuse sessions and keep cooldown tight.The sandbox is your last line of defense, not your only one. Validation, tool permissioning, and adversarial testing in your orchestration layer are where most attacks get stopped first. Related Reading Migrating Legacy .NET Apps to Azure Container Apps: A Step-by-Step Guide &#8211; Microsoft Agent Framework 1.0 GA — The AutoGen Successor Goes Production-Grade & AKS at Build 2026 — Blue-Green Agent Pools, Cross-Cluster Networking, and GPU Mo --- ## Microsoft Scout: The OpenClaw-Based Always-On Enterprise Agent That Changes Everything URL: https://wenfeng.my/microsoft-scout-the-openclaw-based-always-on-enterprise-agent-that-changes-everything/ Author: admin Published: 2026-08-23 06:30:38 Modified: 2026-08-23 06:30:38 Microsoft Scout: The OpenClaw-Based Always-On Enterprise Agent That Changes EverythingWhen Microsoft announced Scout at Build 2026 on June 2, I didn't need to read the press release to understand what it meant. I already run OpenClaw in production — as the backbone of a multi-agent pipeline that researches, writes, edits, and distributes content across my blog and social channels every single day. Scout is Microsoft's enterprise product built on the same open-source platform I've been operating, and it changes the calculus for every enterprise evaluating autonomous AI agents.Let me break down what Scout actually is, what Microsoft announced around it, and what it means if you're already running OpenClaw — or thinking about it.The Problem: Agents That Only Work When You AskMost AI assistants today follow a prompt-response pattern. You type a question, you get an answer, the session ends. The moment you look away, the system stops. Any follow-through — chasing a stalled decision, preparing for tomorrow's meeting, blocking calendar time before a deadline — falls back on you.That is the gap Microsoft is targeting. In Microsoft's own words: "Most systems still stop at answering the question. The real unlock is in the follow-through, where systems hold your priorities and act on them for you, under your control."For enterprises, there is a second problem: trust. Letting an agent act autonomously raises immediate questions about identity, credentials, permissions, audit trails, and data protection. Custom-built agent systems struggle to answer all of these credibly.What Was Announced: Three Layers, Not One ProductThe Build 2026 Day 1 keynote announced three things together, and it's worth separating them because each serves a different purpose:1. Windows platform integration. OpenClaw's node and gateway now run natively on Windows inside MXC (Microsoft Execution Containers) — a policy-driven, OS-level sandbox runtime spanning Windows and WSL. IT departments can declare and enforce agent permissions (filesystem, network, API access) through Intune policies. According to press coverage, OpenAI and NVIDIA have both announced MXC support, which suggests it may become the de facto standard for agent sandboxing on Windows.2. Microsoft Scout. The first product in a new category Microsoft calls "Autopilots" — always-on agents that work autonomously under their own identity and act on your behalf without being prompted each time.3. Agent 365. The governance layer, generally available since May 1, 2026. Microsoft Defender and Intune gain a "Shadow AI" dashboard that discovers and manages local AI agents running on Windows endpoints. OpenClaw is the first supported agent; GitHub Copilot CLI and Claude Code are scheduled next.OpenClaw itself is the open-source local agent framework published by Peter Steinberger in January 2026, which reached roughly 180,000 GitHub stars within months. Microsoft is betting its enterprise agent strategy on that codebase.What Scout Actually DoesScout operates across cloud, desktop, and web. It connects to Teams, Outlook, OneDrive, and SharePoint, and to the data that powers your working day: chats, email, calendar, and contacts. You interact with it in Teams, and extend its reach through the desktop app to your browser, local resources, and Model Context Protocol (MCP) servers.The capabilities Microsoft shipped in preview:Proactive scheduling. Scout coordinates meeting times across time zones, flags important meetings, and generates preparation materials — while keeping you in the loop.Calendar management. It identifies upcoming deliverables and blocks calendar time automatically. If a report is due Friday and it's Wednesday, Scout protects Thursday afternoon.Risk identification. It spots stalled decisions — a critical email unanswered for days, a decision waiting on one person — and surfaces them before they become blockers.Work IQ. Over time, Scout builds context powered by Work IQ: it learns how you work, what you care about, and what needs to happen next, and gets more aligned with your actual priorities.The key shift is proactive versus reactive. Traditional assistants answer; Scout follows through.Enterprise Security: The Real DifferentiatorThe reason Scout matters to enterprises — not just power users — is the governance architecture:Governed identity. Every agent operates under its own governed Microsoft Entra identity, not a shared anonymous service account. The work it does is attributable to a known actor your directory already understands.Credential protection. Credentials are scoped to the task at hand, redacted from logs and diagnostics, and managed with first-party Microsoft rigor. You know precisely whose authority the agent carried and that nothing sensitive leaked along the way.Access control with human sign-off. Agents can only reach resources you approve. Sensitive actions can require a human to sign off before they proceed.Purview enforcement in the moment. Microsoft Purview data protection policies — sensitivity labels and loss prevention — are enforced before anything is sent or written. Scout doesn't bypass these controls; it operates within them.For Malaysian enterprises concerned about PDPA compliance, this matters: agent actions that are auditable, identity-scoped, and policy-enforced are three requirements that are genuinely difficult to achieve with custom-built agent systems.Agent 365 completes the picture on the endpoint side. If your employees are already running OpenClaw instances on company Windows machines — which happens more often than IT knows — Agent 365's Shadow AI detection will surface them, and you can classify each agent as blocked, allowed, or monitored via Intune.Scout vs Self-Managed OpenClaw: The Honest ComparisonHaving run OpenClaw in production for months, here's my honest read.Where Scout wins:Zero infrastructure management. Microsoft hosts the service, manages updates, handles scaling.M365 integration depth. Native access to Teams, Outlook, OneDrive, SharePoint internals is deeper than anything you can build with connectors.Enterprise governance out of the box. Entra identity, Purview enforcement, Intune management, Agent 365 visibility.Work IQ learning, which requires the telemetry and model infrastructure Microsoft provides.Where self-managed OpenClaw wins:Full control over models, data flow, tool access, and governance policy. Nothing opaque.Data sovereignty. Your data stays on your infrastructure. For PDPA-sensitive workloads, this matters.Cost predictability. Self-hosted OpenClaw costs are compute plus model inference. Scout's final pricing is unannounced — it currently requires a GitHub Copilot license and Frontier enrollment, and whether it will be bundled into Microsoft 365 Copilot or billed separately is still unknown.Customization and non-Microsoft workloads. If your agents need AWS, GCP, or non-Microsoft SaaS, self-managed OpenClaw is more flexible.Community velocity. The open-source ecosystem ships new capabilities weekly.The hybrid pattern will be the practical answer for most organizations: Scout for M365-centric workflows (meetings, email, calendar, document discovery), self-managed OpenClaw for custom pipelines, non-Microsoft integrations, and anything touching sensitive data outside M365. Same pattern as cloud architecture — managed services where they fit perfectly, self-managed where you need control.What Running OpenClaw in Production Actually Looks LikeSince the Scout announcement, several clients have asked me what a real OpenClaw deployment involves. Here's a realistic picture from my own environment.A production OpenClaw installation runs as a systemd service with a gateway on a local port. Health checking is straightforward:# Gateway health check curl -s http://127.0.0.1:18789/health # Check configured agents and their model bindings openclaw agents list --json # Channel status (WhatsApp, Discord, Telegram, etc.) openclaw channel status Agent behavior is configured in openclaw.json. A typical production pattern is a primary model with a fallback chain — this keeps agents alive when a provider has intermittent timeouts:{ "agents": { "defaults": { "model": { "primary": "xiaomi-mimo/mimo-v2.5", "fallbacks": ["opencode-go/deepseek-v4-flash"] } } } } Auditing what your agents actually did — the self-hosted equivalent of Scout's "attributable to a known actor" property — means reading the structured gateway logs:# Audit agent actions over the last hour journalctl --user -u openclaw-gateway.service --since '1 hour ago' --no-pager \ | grep -iE 'message send|toolCall|channel exited' And if you're evaluating Scout's enterprise identity model, the equivalent Azure-side hygiene checks look like this:# Find agent-related service principals in Entra ID az ad sp list --query "[?contains(displayName, 'agent') || contains(displayName, 'scout')].{name:displayName, id:appId}" -o table # List managed devices via Microsoft Graph (Intune scope) az rest --method GET \ --uri "https://graph.microsoft.com/v1.0/deviceManagement/managedDevices?\$select=deviceName,managedDeviceName,complianceState" This is the honest trade: everything Scout gives you as a managed service, you can build on self-managed OpenClaw — but identity governance, credential scoping, and policy enforcement become architecture work that you own.Pitfalls and Watch-Outs1. Scout is preview, and access is restricted. Availability today is an experimental release for Frontier organizations — private preview for a select group of customers. Access requires Frontier enrollment, Intune policy configuration, an opt-in attestation, and a GitHub Copilot license. Regional availability timelines beyond the Frontier program have not been announced.2. Pricing is unconfirmed. Neither Scout's final pricing nor Agent 365's official pricing has been announced. Third-party observations suggest Agent 365 may land around $15/user/month, but Microsoft has not confirmed this. Don't build a business case on unconfirmed numbers.3. MXC requirements are not published. Supported Windows versions and licensing requirements for Microsoft Execution Containers are not documented yet. Confirm requirements before planning any MXC-dependent rollout.4. Data residency needs verification. Scout includes cloud-based processing. Organizations in financial services, healthcare, or the public sector with strict data location requirements should verify alignment with Microsoft's data residency policies before deployment — the same due diligence you'd apply to any M365 workload.5. Shadow AI detection will surface existing agents. If employees are already running OpenClaw on Windows endpoints, Agent 365 will discover those instances. Establish IT policies and communicate with employees before deploying Agent 365, or you'll be handling surprises and policy exceptions reactively.6. Self-managed OpenClaw moves fast — including regressions. I've personally hit a gateway version regression that spiked CPU to 100% and had to pin to a known-good build. The open-source velocity that makes OpenClaw exciting also means version pinning, monitoring, and a tested rollback path are mandatory for production. Scout offloads that operational burden to Microsoft; self-hosting keeps it on you.Practical GuidanceNot using agents yet? If you're an M365 shop, evaluate Scout through Frontier. The governance and integration are turnkey. Learn what always-on agents can do for your team before building anything custom.Already on OpenClaw? Don't abandon your investment. Scout and self-managed OpenClaw are complementary. Use Scout for M365 workflows when it reaches your region; keep custom OpenClaw agents for non-Microsoft integrations; and watch the policy conformance capabilities Microsoft is contributing upstream — they'll raise the security baseline for everyone.Evaluating platforms? A simple decision matrix:M365-centric, standard governance needs: Microsoft Scout.Custom workloads, data sovereignty: self-managed OpenClaw.Multi-cloud, mixed SaaS: self-managed OpenClaw with MCP integrations.Maximum governance, minimum operations: Scout plus Agent 365, with self-managed OpenClaw only where control requirements demand it.Key TakeawaysScout is Microsoft's enterprise product built on OpenClaw — announced at Build 2026 alongside Windows-native MXC sandboxing and Agent 365 governance. Microsoft is also contributing policy conformance capabilities back upstream, which raises the security floor for every OpenClaw deployment.Autopilots are a new interaction model — always-on, proactive, identity-scoped agents that follow through without being prompted each time. That's a structural shift from chat-based assistants.Governance is the real differentiator — governed Entra identity, task-scoped credentials redacted from logs, human sign-off for sensitive actions, and Purview enforcement in the moment. Self-managed OpenClaw can reach similar governance, but you're designing and operating it yourself.Hybrid is the practical pattern — Scout for M365-centric workflows, self-managed OpenClaw for custom, multi-cloud, and data-sovereignty workloads. Don't force one tool to do everything.Do your homework before committing — Scout pricing, Agent 365 pricing, MXC Windows requirements, and regional availability are all unconfirmed. Verify data residency for regulated industries, and if employees already run OpenClaw, get ahead of Agent 365's Shadow AI detection with clear policies.Running OpenClaw in production has taught me that the architecture works. Microsoft Scout shows where the enterprise layer is heading. The question for enterprises isn't whether to adopt AI agents — it's which deployment model fits your governance, data, and cost requirements. That answer, increasingly, is both. --- ## Fine-Tuning Local LLMs in 2026 — Now Realistic on a Single Consumer GPU URL: https://wenfeng.my/fine-tuning-local-llms-2026-single-consumer-gpu/ Author: admin Published: 2026-08-22 23:30:24 Modified: 2026-08-22 23:30:24 Two years ago, if someone told you to fine-tune a large language model on a single consumer GPU, you would have laughed. Fine-tuning required hundreds of gigabytes of VRAM, enterprise-grade hardware, and engineering teams with deep ML expertise. The only practical option for most organizations was to use API-based models — OpenAI, Azure OpenAI, Anthropic — and hope the base model was good enough.That has changed. Dramatically. In 2026, fine-tuning a capable LLM on a single NVIDIA RTX 4090 or RTX 5090 is not just possible — it is practical, repeatable, and increasingly the right architectural choice for enterprise teams that need domain-specific AI capabilities without shipping proprietary data to third-party APIs.If you are part of an Azure-first team in Southeast Asia — dealing with multilingual requirements, local regulatory constraints, or domain-specific terminology that general-purpose models consistently get wrong — this shift changes the economics and architecture of your AI strategy.Let me walk you through what changed, when fine-tuning makes sense, and how to actually do it.The Real Question: Fine-Tuning or RAG?Before we get into techniques and hardware, let us address the decision that trips up most teams: when should you fine-tune, and when should you use Retrieval-Augmented Generation?The answer in 2026 is clearer than it was two years ago:Use RAG when:Your knowledge base changes frequently (daily, weekly)You need citations and source attributionThe information is factual and retrievable from documentsYou want low maintenance overheadUse fine-tuning when:You need a specific output format or style consistentlyYou want the model to internalize domain terminology and reasoning patternsYou need to reduce prompt engineering complexity in productionYou need to reduce latency by eliminating retrieval overheadYou are operating under data sovereignty constraints that prohibit sending data to external APIsUse both together when:You need domain-specific reasoning (fine-tuned) combined with access to current documents (RAG)The model needs to understand your domain's language but also reference specific, frequently updated contentI have seen teams waste months building RAG systems when the real problem was that the base model did not understand their domain. No amount of retrieved context can compensate for a model that fundamentally misinterprets your industry terminology. Conversely, I have seen teams fine-tune models for knowledge retrieval when a well-structured RAG pipeline would have been simpler and more maintainable.The key insight: RAG tells the model what to think about. Fine-tuning changes how the model thinks. Most production systems in 2026 benefit from both.LoRA and QLoRA: The Techniques That Made This PossibleThe technical breakthrough that made single-GPU fine-tuning practical is Low-Rank Adaptation, commonly known as LoRA and its quantized variant QLoRA.Here is the core idea, without the academic jargon:Instead of updating every parameter in a 7B or 13B parameter model (which requires enormous memory), LoRA injects small trainable matrices into the model's attention layers. These matrices — called low-rank adapters — capture the delta between the base model and your fine-tuned version. You are training maybe 0.1% to 1% of the total parameters while getting performance close to full fine-tuning.LoRA (Low-Rank Adaptation):Freezes the base model weights entirelyAdds small trainable adapter matrices to attention layersTypical rank (r) values: 8, 16, 32, 64Results in adapters that are a few megabytes to tens of megabytesRequires the base model to fit in VRAMQLoRA (Quantized LoRA):Combines LoRA with 4-bit quantization of the base modelLoads the base model in 4-bit precision (NF4 format)Keeps the adapter weights in higher precision (BF16)Reduces VRAM requirements by roughly 4x compared to full-precision LoRAIntroduces minimal quality degradation for most tasksFor a Llama 3 8B model, here is what the VRAM picture looks like: MethodVRAM RequiredQuality Impact Full fine-tuning~60 GBBaseline LoRA (BF16)~18 GBNegligible QLoRA (4-bit)~6 GBMinimal Inference only (4-bit)~5 GBN/A That 6 GB figure for QLoRA is the magic number. It means a single RTX 4090 with 24 GB of VRAM can comfortably fine-tune a Llama 3 8B model with room to spare for larger batch sizes. The newer RTX 5090 with 32 GB makes the math even more comfortable.For Southeast Asian teams working with multilingual models — say, fine-tuning a Llama variant for Bahasa Malaysia, Thai, or Vietnamese — QLoRA on a consumer GPU is the practical path to production-quality results.Hardware Requirements: What You Actually NeedLet me be specific about hardware, because the vague "you need a good GPU" advice helps nobody.Minimum viable setup (QLoRA on 7B-8B models):GPU: NVIDIA RTX 4060 Ti 16GB or betterRAM: 32 GB system RAMStorage: 100 GB SSD (for datasets, checkpoints, and the base model)Training time: 4-12 hours depending on dataset size and hyperparametersRecommended setup (QLoRA on 13B models):GPU: NVIDIA RTX 4090 (24 GB) or RTX 5090 (32 GB)RAM: 64 GB system RAMStorage: 200 GB NVMe SSDTraining time: 8-24 hoursProduction-capable setup (LoRA on 13B-34B models):GPU: NVIDIA RTX 5090 (32 GB) or dual GPUsRAM: 128 GB system RAMStorage: 500 GB NVMe SSDTraining time: 12-48 hoursA critical note for teams in the region: if your organization already has Azure infrastructure, consider using Azure VMs with NC-series or ND-series VMs for training runs. An NC96ads_A100_v4 gives you 4x A100 80GB PCIe GPUs (320GB total), which handles 70B parameter models with QLoRA. On-demand pricing is roughly $14-15 per hour in most regions (spot instances run significantly cheaper, around $2.70/hr where available); reserved instances reduce this further. For one-off training runs, this often makes more sense than purchasing consumer hardware.Two practical warnings before you commit to the cloud path. First, GPU availability is region-limited — the NCads A100 v4 series is not offered in every Azure region, so verify availability before you design around it. Second, most subscriptions start with zero quota for GPU families; request a quota increase (Subscription → Usage + quotas) before your first run, because approvals can take a day or more. # Confirm the SKU is available in your target region az vm list-skus --location eastus \ --size Standard_NC96ads_A100_v4 \ --resource-type virtualMachines \ --all --output table # Provision the training VM on-demand az group create --name rg-llm-finetune --location eastus az vm create \ --resource-group rg-llm-finetune \ --name vm-finetune-a100 \ --size Standard_NC96ads_A100_v4 \ --image Ubuntu2204 \ --admin-username azureuser \ --ssh-key-values ~/.ssh/training_key.pub \ --os-disk-size-gb 512 # Tear everything down when the run finishes — billing stops immediately az group delete --name rg-llm-finetune --yes --no-wait If your team standardizes on infrastructure as code, the same training VM is a short block in either Bicep or Terraform: // main.bicep — GPU training VM (NIC and VNet resources omitted for brevity) param location string = resourceGroup().location param sshPublicKey string resource vm 'Microsoft.Compute/virtualMachines@2024-07-01' = { name: 'vm-finetune-a100' location: location properties: { hardwareProfile: { vmSize: 'Standard_NC96ads_A100_v4' } osProfile: { computerName: 'vm-finetune-a100' adminUsername: 'azureuser' linuxConfiguration: { disablePasswordAuthentication: true ssh: { publicKeys: [ { path: '/home/azureuser/.ssh/authorized_keys', keyData: sshPublicKey } ] } } } storageProfile: { imageReference: { publisher: 'Canonical', offer: '0001-com-ubuntu-server-jammy', sku: '22_04-lts-gen2', version: 'latest' } osDisk: { createOption: 'FromImage', diskSizeGB: 512, managedDisk: { storageAccountType: 'Premium_LRS' } } } networkProfile: { networkInterfaces: [ { id: nic.id } ] } } } # main.tf — the same GPU training VM in Terraform resource "azurerm_linux_virtual_machine" "finetune" { name = "vm-finetune-a100" resource_group_name = azurerm_resource_group.rg.name location = azurerm_resource_group.rg.location size = "Standard_NC96ads_A100_v4" admin_username = "azureuser" admin_ssh_key { username = "azureuser" public_key = file("~/.ssh/training_key.pub") } os_disk { caching = "ReadWrite" storage_account_type = "Premium_LRS" disk_size_gb = 512 } source_image_reference { publisher = "Canonical" offer = "0001-com-ubuntu-server-jammy" sku = "22_04-lts-gen2" version = "latest" } network_interface_ids = [azurerm_network_interface.nic.id] } Whichever provisioning path you choose, the operational pattern is the same: treat the GPU VM as disposable. Spin it up, train, export the adapter to durable storage, delete the VM.That said, for teams that want to iterate quickly — running multiple experiments per day, testing different hyperparameters and datasets — having a local workstation with an RTX 4090 or RTX 5090 removes the friction of spinning up cloud instances for every experiment.A Practical Fine-Tuning WorkflowHere is the actual workflow I recommend for enterprise teams. This is not theory — this is the process I have seen work in production environments.Step 1: Dataset PreparationThis is where 80% of your effort should go. The quality of your dataset determines the quality of your fine-tuned model. Full stop.Data format: For instruction fine-tuning, structure your data as instruction-input-output triplets: { "instruction": "Classify the following customer complaint by department and urgency level.", "input": "My invoice has been overdue for 60 days and I still haven't received the correct amount. This is affecting our cash flow reporting.", "output": "Department: Finance/Accounts Receivable | Urgency: High | Reason: Payment discrepancy persisting beyond standard terms, impacting customer's financial reporting." } Dataset size guidelines:Minimum viable: 500-1,000 high-quality examplesGood results: 2,000-5,000 examplesExcellent results: 5,000-10,000+ examplesDiminishing returns beyond 50,000 examples for most tasksData quality checklist:Every example should represent the actual distribution of tasks the model will faceInclude edge cases and difficult examples, not just straightforward onesEnsure output format is consistent across all examplesValidate that domain terminology is used correctlyRemove duplicates and near-duplicatesSplit 90% train, 10% validation — never touch the test set until final evaluationA common mistake I see in Southeast Asian enterprise teams: using GPT-4 or Claude to generate synthetic training data without validating it against real production examples. Synthetic data is useful for augmentation, but it must be grounded in actual domain expertise. Have your subject matter experts review and correct at least 20% of any synthetic dataset.Step 2: Choose Your Base Model and Training FrameworkFor most enterprise use cases in 2026, these are the practical choices:Base models:Llama 3.1 8B — the standard starting point for single-GPU fine-tuning (best community support)Llama 4 Scout (109B total, 17B active) — strong multimodal open model with 10M-token contextQwen 2.5 7B / 14B — strong multilingual performance, excellent for CJK and Southeast Asian languages (Qwen 3.5, released February 2026, extends this further)Mistral 7B / Mixtral 8x7B — efficient, well-documentedTraining frameworks:Unsloth — the fastest path to a working fine-tune. Optimized LoRA/QLoRA, 2x faster training, 60% less memory. If you are starting today, start here.Hugging Face TRL — the standard. Well-documented, huge community, integrates with everything in the HF ecosystem.Axolotl — excellent configuration-driven approach. Define your training in a YAML file and let it handle the rest.Here is a minimal Unsloth script to fine-tune Llama 3.1 8B with QLoRA: from unsloth import FastLanguageModel import torch # Load base model in 4-bit model, tokenizer = FastLanguageModel.from_pretrained( model_name="unsloth/Meta-Llama-3.1-8B", max_seq_length=2048, dtype=None, # Auto-detect load_in_4bit=True, ) # Add LoRA adapters model = FastLanguageModel.get_peft_model( model, r=16, # LoRA rank target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], lora_alpha=16, lora_dropout=0, bias="none", use_gradient_checkpointing="unsloth", ) # Training with TRL's SFTTrainer from trl import SFTTrainer from transformers import TrainingArguments trainer = SFTTrainer( model=model, tokenizer=tokenizer, train_dataset=dataset, max_seq_length=2048, args=TrainingArguments( per_device_train_batch_size=4, gradient_accumulation_steps=4, warmup_steps=50, num_train_epochs=3, learning_rate=2e-4, fp16=not torch.cuda.is_bf16_supported(), bf16=torch.cuda.is_bf16_supported(), logging_steps=10, output_dir="outputs", optim="adamw_8bit", ), ) trainer.train() model.save_pretrained("lora_model") On an RTX 4090, this will train a 3,000-example dataset in roughly 2-4 hours. On an RTX 5090, expect roughly 30-40% faster training.Step 3: EvaluationEvaluation is where most fine-tuning projects fall apart. People train a model, run a few test prompts, see that it "looks good," and ship it. Then production users discover the model fails on edge cases.Quantitative evaluation:Run your held-out test set through the fine-tuned modelCalculate task-specific metrics (accuracy, F1, BLEU, ROUGE — depends on your task)Compare against the base model on the same test setIf your metric does not improve by at least 5-10%, the fine-tuning did not add enough valueQualitative evaluation:Have domain experts manually evaluate 50-100 outputsTest on known failure cases from the base modelTest on adversarial inputs and edge casesVerify the model has not "forgotten" general capabilities it still needsRed flags to watch for:Catastrophic forgetting: the model gets worse at general tasksOverfitting: training loss is low but validation loss is climbingMemorization: the model reproduces training examples verbatim instead of generalizingFormat collapse: the model always produces the same structure regardless of inputStep 4: DeploymentOnce you have a validated fine-tuned model, deployment options in 2026 are straightforward:Option 1: Merge and serve with vLLM Merge the LoRA adapter into the base model, export as a single GGUF or SafeTensors file, and serve with vLLM or Ollama. This is the simplest path for internal tools and prototypes.Option 2: LoRA hot-swapping with vLLM Keep the base model in memory and load different LoRA adapters on demand. This is powerful for serving multiple fine-tuned variants from a single GPU — say, one adapter for finance queries, another for legal queries.Option 3: Azure deployment For production enterprise workloads, deploy to Azure ML or Azure AI Foundry with GPU endpoints. Fine-tuned models served through Azure give you enterprise-grade availability targets, Entra ID integration, and the ability to keep data within your chosen region.Option 4: On-premises air-gapped deployment For organizations with strict data sovereignty requirements — common in government, healthcare, and financial services across Southeast Asia — a fine-tuned model running on local hardware eliminates the need to send any data to external services. This is one of the most compelling use cases for local fine-tuning in the region.What I Have Learned From Real ProjectsA few hard-won observations from fine-tuning projects I have been involved with:Start smaller than you think. A well-prepared 7B model often outperforms a poorly prepared 70B model for specific tasks. The quality of your training data matters more than the size of your base model.Multilingual fine-tuning is trickier than it looks. If you are training a model for Bahasa Malaysia mixed with English (code-switching is natural in Malaysian business), make sure your training data reflects actual code-switching patterns. Training on Malay-only and English-only data separately will not give you a model that handles natural Malaysian business communication.The feedback loop is everything. Deploy the model internally, collect failure cases, add them to your training set, retrain, and repeat. The first version of your fine-tuned model is never the last version. Budget for at least 3-5 iteration cycles.Documentation is your friend. Document every experiment — hyperparameters, dataset versions, evaluation results. Six months from now, you will not remember why you chose rank 16 over rank 32. Your future self will thank you.The Bottom LineFine-tuning local LLMs in 2026 is no longer a research exercise. It is a practical, accessible capability that enterprise teams can and should consider as part of their AI architecture — especially in Southeast Asia, where data sovereignty, multilingual requirements, and domain-specific needs often make general-purpose API models insufficient.The combination of QLoRA, optimized training frameworks like Unsloth, and capable open-weight models like Llama 4 and Qwen 2.5 means that a single engineer with a consumer GPU can produce a fine-tuned model that would have required a research team and enterprise hardware just two years ago.The question is not whether your team can do this. The question is whether your team should.Key TakeawaysFine-tuning and RAG serve different purposes. RAG provides current knowledge; fine-tuning changes how the model thinks. Most production systems benefit from both — use fine-tuning for domain-specific reasoning and RAG for document access.QLoRA makes single-GPU fine-tuning practical. You can fine-tune a 7B-8B parameter model on a single RTX 4090 with 24GB VRAM, or on Azure GPU VMs for one-off runs. The barrier to entry has dropped from hundreds of thousands of dollars to a few thousand.Dataset quality is the single biggest factor in success. Invest 80% of your effort in preparing 2,000-10,000 high-quality, validated training examples. Synthetic data generation is useful but must be grounded in real domain expertise.Evaluation is not optional. Run quantitative metrics on a held-out test set and qualitative evaluation with domain experts before deployment. Watch for catastrophic forgetting, overfitting, and format collapse.Start with a small model and iterate. A well-fine-tuned 7B model often beats a poorly prepared 70B model. Plan for 3-5 iteration cycles and document every experiment. Multilingual code-switching (common in Malaysian enterprise contexts) requires careful dataset construction. --- ## AI Agent Governance: Bounded Autonomy, Governance Agents, and the Enterprise Production Scaling Gap URL: https://wenfeng.my/ai-agent-governance-bounded-autonomy-governance-agents-and-the-enterprise-production-scaling-gap/ Author: admin Published: 2026-08-22 06:30:00 Modified: 2026-10-04 15:31:04 AI Agent Governance: Bounded Autonomy, Governance Agents, and the Enterprise Production Scaling Gap Every enterprise I talk to in Malaysia is excited about AI agents. They've seen the demos, read the Gartner predictions — 40% of enterprise applications will incorporate task-specific AI agents by 2026 — and they want in. But here's the uncomfortable truth that most vendor presentations skip: the vast majority of AI pilots never reach production. MIT's NANDA report found that roughly 95% of organizational generative AI pilots fail to deliver measurable value, and Gartner predicts over 40% of agentic AI projects will be canceled outright by the end of 2027. The common cause is rarely the model — it's that organizations can't govern what they deploy. I've spent the past year building and operating multi-agent content pipelines in production. Not demos. Not POCs. Real agents writing, publishing, and distributing content on a schedule. That experience has taught me something the marketing materials won't: governance is not a compliance checkbox — it's the architecture that determines whether your agents scale or collapse. The Production Scaling Gap Is Real Let's start with the numbers that matter. Gartner's latest research paints a sobering picture: 40% of enterprise applications will incorporate task-specific AI agents by 2026, up from less than 5% in 2024 (Gartner) 33% of enterprise software applications will include agentic AI by 2028, up from less than 1% in 2024 (Gartner) Over 40% of agentic AI projects will be canceled by the end of 2027 — driven by cost, unclear business value, and inadequate risk controls (Gartner) The gap between experimentation and production isn't about model quality or tool availability. It's about three things most teams underestimate: governance architecture, operational cost control, and human-agent interface design. When I run my own multi-agent pipeline — a Research Scout, Content Strategist, Technical Writer, SEO Optimizer, and Social Writer working in sequence — I don't worry about whether the LLM can write a paragraph. I worry about what happens when the SEO Optimizer decides to change a title that conflicts with the Strategic Writer's intent, or when a scheduled job fires twice due to a race condition and publishes duplicate content. Those are governance failures, not model failures. Bounded Autonomy: The Architecture That Actually Works The concept of "bounded autonomy" is gaining traction as the practical answer to the governance question. Instead of asking "how much autonomy should agents have?", bounded autonomy asks "what are the operational limits within which agents can act freely, and what triggers human escalation?" Think of it as the agent equivalent of least-privilege access control in cybersecurity. A database administrator gets read access to production logs but not write access to the customer table. Similarly, an AI agent should have clear boundaries on what it can do, what it can access, and what decisions it can make independently. Three Levels of Agent Autonomy In practice, I've found that most enterprise agent deployments fall into three tiers: Level 1: Full Automation (Low-Stakes) Agent handles routine, reversible decisions with minimal impact Examples: Categorizing incoming support tickets, generating standard report sections, routing internal requests Governance: Audit trail logging, periodic review Risk: Low — mistakes are easily corrected Level 2: Supervised Autonomy (Medium-Risk) Agent operates within defined parameters but flags exceptions for human review Examples: Code review suggestions, draft content creation, data pipeline monitoring Governance: Approval gates for edge cases, escalation thresholds, rollback capability Risk: Medium — requires clear fallback paths Level 3: Human-Led with Agent Assistance (High-Stakes) Agent provides analysis and recommendations; human makes the final decision Examples: Financial approvals, customer-facing commitments, regulatory submissions Governance: Full audit trail, mandatory human sign-off, time-bounded review windows Risk: High — agent influence must be transparent and traceable The critical insight is that these levels are not determined by the agent's capability but by the task's risk profile. A powerful frontier model might be Level 1 for categorizing emails but Level 3 for drafting a customer contract. The governance architecture must separate capability from authority. Implementing Bounded Autonomy in Practice Here's what bounded autonomy looks like in a real multi-agent system: # Example: Agent governance configuration agent: content-writer autonomy_level: 2 # Supervised boundaries: max_tokens_per_request: 4000 allowed_tools: [ghost_api, markdown_writer] blocked_tools: [publish_directly, delete_post, modify_tags] approval_required: - action: "publish" threshold: "status changes to 'published'" - action: "modify_existing_post" threshold: "post older than 7 days" escalation: trigger: "content_quality_score < 0.6" target: "human_reviewer" timeout: "24h" audit: log_all_tool_calls: true log_reasoning_trace: false # Privacy: don't log prompts retention_days: 90 The key elements are: Explicit tool permissions — agents can only use tools they're authorized for Action-level approval gates — certain actions always require human review Quality thresholds — automatic escalation when output quality drops Audit logging — complete traceability of what the agent did and why Governance Agents: When Agents Watch Agents The most interesting development in enterprise agent governance is the emergence of governance agents — dedicated AI agents whose job is to monitor other agents for policy violations, anomalous behavior, and compliance gaps. This sounds like adding complexity on top of complexity, but it's actually a natural evolution. In traditional IT operations, you don't rely on developers to monitor their own production systems. You have separate SRE teams, SIEM platforms, and automated alerting. The same principle applies to agent fleets. What Governance Agents Monitor A well-designed governance agent tracks: Policy compliance: Is the working agent adhering to its defined autonomy level? Is it accessing only authorized tools? Behavioral anomalies: Is the agent suddenly making unusual API calls, accessing unexpected data, or producing outputs that deviate from its normal patterns? Cost boundaries: Is the agent exceeding its token budget? Is it making redundant calls that could be cached? Temporal boundaries: Is the agent operating within its allowed schedule? Are there unauthorized escalation attempts? Content safety: For content-generating agents, is the output within brand guidelines, factual accuracy thresholds, and compliance requirements? The Architecture Pattern The governance agent pattern in practice: ┌─────────────────┐ │ Working Agent │ ← Normal agent doing its job │ (e.g., Writer) │ └────────┬────────┘ │ actions/logs ▼ ┌─────────────────┐ │ Governance Agent │ ← Monitors for violations │ (e.g., Auditor) │ └────────┬────────┘ │ alerts ▼ ┌─────────────────┐ │ Human Dashboard │ ← Reviews flagged events │ (e.g., Slack/Bot)│ └─────────────────┘ The governance agent doesn't slow down the working agent — it operates asynchronously, analyzing logs and outputs after the fact. When it detects a policy violation or anomalous pattern, it escalates to the human dashboard with a structured alert. In my own pipeline, I've implemented a lightweight version of this: after each content generation step, an automated review checks word count, topic relevance, link safety, and brand voice alignment. When any check fails, the content is flagged for human review rather than published automatically. It's not a full governance agent, but it demonstrates the principle. The EU AI Act and Regulatory Reality The governance conversation isn't just philosophical — it's becoming a legal requirement. The EU AI Act was originally scheduled to make high-risk AI system obligations enforceable from 2 August 2026, but the Digital Omnibus on AI (Regulation (EU) 2026/1744, published in the Official Journal on 24 July 2026) deferred those deadlines. High-risk obligations now apply from 2 December 2027 for systems under Annex III (use-case-based high risk) and 2 August 2028 for Annex I systems (those covered by EU harmonisation legislation such as machinery and medical devices). Requirements that do apply to multi-agent orchestration in the meantime include: Human-in-the-loop oversight for critical decisions Immutable audit trails for all agent actions Scenario testing before deployment Persistent identity management for non-human entities In May 2026, Colorado repealed and replaced its 2024 AI Act with SB26-189, the "Automated Decision-Making Technology" law, which shifts accountability from system-level compliance to decision-by-decision accountability and targets consequential decisions (employment, financial services, housing, healthcare, insurance, legal services, and government services). The new law imposes developer and deployer duties with risk assessments and consumer notice requirements. For Malaysian enterprises, while these specific regulations don't apply directly, the trend is clear: PDPA compliance will increasingly intersect with AI governance requirements. BNM (Bank Negara Malaysia) and sector-specific regulators will follow the EU's lead on AI oversight. Enterprises that build governance into their agent architecture now will be ahead of the compliance curve. The practical cost impact is significant: industry analyses consistently report that compliance, governance, and integration overhead dominate agent deployment budgets. Planning for this upfront prevents the common failure mode where governance costs blow through an unplanned budget and the project gets canceled. Lessons from Production: What Actually Matters After running multi-agent systems in production for over a year, here are the governance principles that have proven most valuable: 1. Start with Task-Level Risk Assessment Before assigning autonomy levels, assess each task: Reversibility: Can the action be undone? If yes, lower governance bar. Impact radius: How many people/systems are affected? Wider = more governance. Data sensitivity: Does the agent access PII, financial data, or proprietary information? Higher sensitivity = stricter controls. Frequency: High-frequency tasks need automated governance; low-frequency ones can rely on periodic review. 2. Design for Failure, Not Success Every agent will make mistakes. The governance architecture should answer: What happens when the agent's output is wrong? How quickly can a human detect and correct the error? What's the blast radius of an uncaught error? Is there an automatic rollback mechanism? 3. Make Audit Trails Useful, Not Just Complete Logging everything is easy. Logging what matters is hard. Your audit trails should answer: What did the agent decide? What inputs did it use? What alternatives did it consider? What would a human have done differently? 4. Cost Governance Is Agent Governance Agent fleets make thousands of daily LLM calls. Without cost governance: A runaway loop can burn through thousands of dollars in hours Redundant API calls compound across agent chains Model selection becomes a financial architecture decision The "Plan-and-Execute" pattern — where a capable frontier model plans and cheaper models execute — can reduce costs by up to 90%. This isn't just cost optimization; it's sustainable governance. A Practical Governance Checklist for Malaysian Enterprises If you're deploying AI agents in production, here's what to verify before going live: [ ] Each agent has a defined autonomy level (1, 2, or 3) [ ] Tool permissions are explicitly listed and audited [ ] Approval gates exist for high-impact actions [ ] Escalation paths are defined with timeout thresholds [ ] Audit trails capture actions, inputs, and reasoning [ ] Cost budgets are set per agent and per pipeline [ ] A governance review happens at least monthly [ ] Rollback procedures are tested, not just documented [ ] PDPA compliance is verified for data-touching agents [ ] Human override capability exists at every autonomy level Enforcement in Practice: Azure CLI, Bicep, Terraform, and Python Theory matters less than enforcement. Here are patterns I use to make agent governance concrete on Azure. Scoped Identities for Agents (Azure CLI) Every agent gets its own service principal with least-privilege access — never shared credentials: #!/bin/bash # Create a dedicated identity for a single agent AGENT_SP=$(az ad sp create-for-rbac \ --name "agent-content-writer-prod" \ --role "Contributor" \ --scopes "/subscriptions/$SUB_ID/resourceGroups/rg-agents-prod" \ --query appId -o tsv) # Deny all role assignment changes — agents cannot escalate their own privileges az role assignment create \ --assignee "$AGENT_SP" \ --role "Role Based Access Control Administrator" \ --scope "/subscriptions/$SUB_ID/resourceGroups/rg-agents-prod" \ --condition "((!(ActionMatches{'Microsoft.Authorization/roleAssignments/write'})))" # Enable audit logging for all agent actions az monitor diagnostic-settings create \ --name agent-audit \ --resource "/subscriptions/$SUB_ID/resourceGroups/rg-agents-prod" \ --logs '[{"categoryGroup":"audit","enabled":true}]' \ --workspace "$LOG_ANALYTICS_WS_ID" # Cost boundary: hard spending cap on the agent's subscription az consumption budget create \ --budget-name agent-budget-prod \ --amount 5000 \ --time-grain Monthly \ --scope "/subscriptions/$AGENT_SUB_ID" Guardrails via Azure Policy (Bicep) Prevent agents from deploying resources outside approved configurations: // agentPolicyAssignment.bicep param policyDefinitionId string param enforcementMode string = 'Default' // 'DoNotEnforce' for dry-run resource agentPolicy 'Microsoft.Authorization/policyAssignments@2024-04-01' = { name: 'restrict-agent-vm-skus' scope: resourceGroup() properties: { displayName: 'Restrict agent-deployed VM SKUs' policyDefinitionId: policyDefinitionId enforcementMode: enforcementMode parameters: { listOfAllowedSKUs: { value: [ 'Standard_B2s_v2', 'Standard_D2s_v5' ] } } } } Same Guardrails Across a Terraform Estate resource "azurerm_resource_policy_assignment" "agent_governance" { name = "agent-governance-prod" resource_id = azurerm_resource_group.agents.id policy_definition_id = var.agent_policy_definition_id enforce = true parameters = jsonencode({ listOfAllowedSKUs = { value = ["Standard_B2s_v2", "Standard_D2s_v5"] } }) } Cost Governance in Python Every agent call passes through a budget gate — the software equivalent of a credit limit: from functools import wraps from azure.monitor.opentelemetry import configure_azure_monitor AGENT_BUDGETS = {"content-writer": 150.00} # USD per month agent_spend = {} def budget_guard(agent_name): def decorator(fn): @wraps(fn) def wrapper(*args, **kwargs): spent = agent_spend.get(agent_name, 0.0) if spent >= AGENT_BUDGETS[agent_name]: raise PermissionError( f"{agent_name} exceeded monthly budget ${AGENT_BUDGETS[agent_name]}" ) result = fn(*args, **kwargs) agent_spend[agent_name] = spent + result.token_cost return result return wrapper return decorator Combine this with Azure Monitor OpenTelemetry export, and every agent action lands in an immutable Log Analytics audit trail — the exact evidence trail regulators are starting to demand. Key Takeaways The enterprise scaling gap is a governance gap, not a technology gap. MIT reports roughly 95% of generative AI pilots fail to deliver value, and Gartner expects over 40% of agentic AI projects to be canceled by 2027 — because organizations can't govern agents, not because models are incapable. Bounded autonomy is the practical architecture. Separate agent capability from agent authority. Match autonomy levels to task risk profiles, not model capability. Governance agents watching working agents is the emerging pattern. Dedicated monitoring agents that track policy compliance, behavioral anomalies, and cost boundaries — operating asynchronously to avoid latency impact. Regulatory pressure is real and accelerating. The EU AI Act's high-risk obligations were deferred via the Digital Omnibus on AI (Regulation (EU) 2026/1744) to December 2027 and August 2028, and Colorado replaced its 2024 AI Act with SB26-189 in May 2026. Malaysian enterprises should build governance now rather than retrofit it later. Start with your riskiest agent, not your most impressive one. Governance architecture is proven under stress. If your governance can handle the agent that touches financial data, it can handle everything else. Building production agent systems is as much about governance architecture as it is about model selection. The enterprises that get this right in 2026 will have a durable advantage — not because their agents are smarter, but because their agents are trustworthy. --- ## Bicep vs Terraform for Azure in 2026: An Honest Decision Framework After Two Years of Maturation URL: https://wenfeng.my/bicep-vs-terraform-for-azure-in-2026-an-honest-decision-framework-after-two-years-of-maturation/ Author: admin Published: 2026-08-22 02:00:17 Modified: 2026-10-04 15:26:54 Bicep vs Terraform for Azure in 2026: An Honest Decision Framework After Two Years of Maturation Every Azure team I work with asks the same question: "Should we use Bicep or Terraform?" After two years of watching both tools mature — Bicep closing tooling gaps, Terraform's Business Source License saga settling under IBM ownership, and OpenTofu emerging as a legitimate fork — I can give a more honest answer than the vendor-sponsored comparisons. The short version: for Azure-only work, Bicep is now the pragmatic default. For multi-cloud or teams with significant Terraform investment, stay the course. And whatever you do, don't migrate between tools unless you have a compelling reason. Let me explain why. The State of Play in 2026 Bicep Has Closed the Gap Two years ago, Bicep's main weakness was tooling. The module ecosystem was thin, testing was primitive, and the CI/CD story was underdeveloped. In 2026: Azure Verified Modules (AVM) — Microsoft's curated module library — has matured significantly. Most Azure patterns have production-ready modules, versioned and maintained by Microsoft. Syntax and developer experience — Bicep's type inference, parameter validation, and VS Code extension are genuinely better than Terraform's HCL for Azure resources. Day-zero resource coverage — Bicep gets new Azure resources and API versions on day one, since it compiles to ARM templates. Terraform's AzureRM provider can lag for newly announced resources. No state management — Azure is the source of truth. No state files to store, secure, lock, or corrupt. For small-to-midsize teams, this is a significant operational win. Terraform Remains Strong Where It Matters Terraform's strengths haven't diminished: Multi-cloud — 3,000+ providers. If you manage AWS, GCP, or non-Azure resources, Terraform is the only realistic option. Drift detection — terraform plan gives a full dependency-graph diff before applying changes. Bicep's what-if is useful but less comprehensive. Testing ecosystem — Terratest, terraform-compliance, tflint, tfsec — the testing tooling is more mature and battle-tested. State-based operations — Explicit state enables targeted operations like terraform import, terraform state mv, and precise dependency tracking. OpenTofu Changed the Licensing Math The licensing story is stable but permanent. IBM kept HashiCorp's Business Source License in place after completing its acquisition in early 2025, so the governance question isn't going away — but it's now solved: Teams concerned about BSL: OpenTofu is wire-compatible with Terraform and adds its own features (native state encryption, removed blocks, early variable evaluation, override files) Migration is trivial: swap the binary, keep the same .tf files, run opentofu init Enterprise story: Spacelift, Atlantis, and env0 all support OpenTofu natively State management options: S3+DynamoDB, Azure Storage, or hosted platforms like Spacelift and env0 — all viable If BSL licensing is a governance concern for your organization, OpenTofu eliminates it without changing your workflows. The Honest Decision Matrix Here's what I actually tell teams, compressed into one table: Situation Choice Why Azure-only, new project Bicep No state, day-zero support, AVM modules Azure + AWS/GCP Terraform/OpenTofu Multi-provider is Terraform's home turf Heavy third-party integration Terraform/OpenTofu Cloudflare, GitHub, SaaS providers exist Existing Terraform estate Stay on Terraform Migration cost exceeds benefit Existing ARM templates Bicep az bicep decompile is a clean upgrade path License governance concern OpenTofu MPL-licensed, wire-compatible Choose Bicep When: Azure-only workloads — no multi-cloud plans, no third-party resources Small-to-midsize platform teams — state management overhead isn't worth it Greenfield projects — starting fresh with no existing investment ARM template heritage — you have existing ARM templates; az bicep decompile is a clean upgrade path Fastest IaC adoption — lower learning curve, cleaner syntax, no state files Choose Terraform (or OpenTofu) When: Multi-cloud environments — Azure + AWS, GCP, or other providers Heavy third-party integration — Cloudflare DNS, GitHub repos, SaaS providers, Kubernetes clusters alongside Azure Significant existing Terraform investment — rewriting hundreds of .tf files is rarely justified Strong drift detection needs — terraform plan is genuinely superior for change preview Enterprise governance with Sentinel — if you're invested in Terraform Cloud + Sentinel policies Stay on What You Have When: You have a working pipeline — migration cost almost always exceeds the theoretical benefit The team is productive — developer velocity matters more than tool choice Neither tool is blocking a requirement — if Bicep can do what you need, or Terraform can do what you need, optimize elsewhere Syntax Comparison: The Practical Difference The readability difference is real but overstated in vendor comparisons. Here's the same storage account in both: // Bicep — cleaner, typed, Azure-aware resource storage 'Microsoft.Storage/storageAccounts@2023-05-01' = { name: 'st${uniqueString(resourceGroup().id)}' location: location sku: { name: 'Standard_LRS' } kind: 'StorageV2' properties: { minimumTlsVersion: 'TLS1_2' allowBlobPublicAccess: false } } # Terraform — more verbose, but explicit and clear resource "azurerm_storage_account" "this" { name = "st${random_string.suffix.result}" resource_group_name = azurerm_resource_group.this.name location = var.location account_tier = "Standard" account_replication_type = "LRS" min_tls_version = "TLS1_2" allow_nested_items_to_be_public = false } Bicep is more concise. Terraform is more explicit about dependencies and references. For teams already comfortable with HCL, the Terraform version is perfectly readable. For teams new to IaC, Bicep's learning curve is lower. CI/CD Integration: Both Work, Differently Bicep CI/CD Pattern # GitHub Actions — Bicep - name: What-If run: az deployment group what-if -g myRG -f main.bicep -p params.json - name: Deploy run: az deployment group create -g myRG -f main.bicep -p params.json No state backend to configure. No locking to manage. The pipeline is simpler. Terraform CI/CD Pattern # GitHub Actions — Terraform - name: Plan run: terraform plan -out=tfplan - name: Apply run: terraform apply tfplan Requires a state backend (Azure Storage, Terraform Cloud, or S3). Requires state locking. More setup, but more operational control. Cost Implication Terraform with a hosted state backend (Terraform Cloud, Spacelift) adds per-user or per-workspace cost. Bicep has no backend cost. For a 10-person team, this could be $50-200/month depending on the backend. Day-2 Operations: Where the Real Differences Live Syntax debates dominate these comparisons, but day-2 operations is where the tools actually feel different. Here's the practical side. Bicep: deploy from the CLI, preview first # Bicep: preview changes without touching anything az deployment group what-if \ --resource-group rg-app-prod \ --template-file main.bicep \ --parameters environment=production # Bicep: deploy (incremental by default) az deployment group create \ --resource-group rg-app-prod \ --template-file main.bicep \ --parameters environment=production That's the whole operational loop. No state to initialize, no backend to configure. Terraform: the backend is real work # Terraform: one-time backend setup — the part Bicep doesn't need terraform init \ -backend-config="storage_account_name=sttfstate001" \ -backend-config="container_name=tfstate" \ -backend-config="key=app-prod.terraform.tfstate" terraform plan -out=tfplan terraform apply tfplan The storage account needs RBAC lockdown (state files contain secrets in plaintext — see the pitfalls section), and locking relies on Azure Blob leases. It's well-understood, but it's extra surface area your team owns forever. A Cheap Drift Check with Python Neither tool watches your estate for you. Here's a lightweight drift check I use for tag baselines — the kind of thing that quietly rots when people click around the portal: # drift_check.py — flag resources whose tags drifted from the baseline from azure.identity import DefaultAzureCredential from azure.mgmt.resource import ResourceManagementClient SUBSCRIPTION_ID = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" EXPECTED_TAGS = {"environment": "production", "owner": "platform-team"} client = ResourceManagementClient(DefaultAzureCredential(), SUBSCRIPTION_ID) drifted = [] for resource in client.resources.list(): actual = resource.tags or {} for key, expected in EXPECTED_TAGS.items(): if actual.get(key) != expected: drifted.append((resource.name, key, actual.get(key))) for name, key, actual in drifted: print(f"DRIFT: {name} tag '{key}' = {actual!r} (expected {EXPECTED_TAGS[key]!r})") print(f"{len(drifted)} drifted tag(s) found" if drifted else "No tag drift detected") This only catches tag drift, not full configuration drift — terraform plan and az deployment group what-if remain the authoritative previews. But a 30-line script running nightly in CI catches portal edits before they become mysteries. The Hybrid Pattern That Actually Works Many Azure-first enterprises aren't purely Azure-only. They have: Azure workloads (primary) Cloudflare DNS (third-party) GitHub repositories and actions (SaaS) Maybe a small AWS footprint for specific services The practical pattern: Bicep: Azure infrastructure (VNets, AKS, databases, storage) Terraform/OpenTofu: Third-party resources (Cloudflare, GitHub, SaaS) This isn't clean, but it's honest. Each tool handles what it does best. The team uses Bicep for Azure because it's simpler and faster, and Terraform for everything else because it has the providers. The risk: two tools means two skill sets, two CI/CD pipelines, and two sets of operational knowledge. For a small team, this overhead might not be worth it. For a larger platform team with clear ownership boundaries, it works. The "Don't Migrate" Rule I've seen teams spend months migrating from Terraform to Bicep (or vice versa) with no functional improvement. The migration cost is almost never justified unless: The current tool is genuinely blocking a requirement — e.g., you need day-zero Azure resource support and Terraform can't deliver it The operational cost of the current tool is unsustainable — e.g., Terraform state corruption is a recurring problem Organizational mandate — leadership has decided on a standard Otherwise: standardize on one tool for new projects, let old projects attrit, and invest the migration effort in something that actually delivers value. What About Testing? Terraform has the more mature testing ecosystem: Terratest — integration testing with real infrastructure terraform-compliance — policy-as-code testing tflint — static analysis tfsec/trivy — security scanning Bicep testing is thinner but improving: az bicep build — compilation validation az deployment group what-if — preview changes PSRule for Azure — policy compliance checking AVM module tests — Microsoft ships AVM modules with test coverage If testing maturity is a hard requirement, Terraform wins today. If you're comfortable with Azure-native validation tools, Bicep is sufficient for most teams. Pitfalls: What Goes Wrong in Practice Five mistakes I see repeatedly, and how to avoid them. 1. Treating Bicep what-if like terraform plan. What-if has blind spots — some property changes show as "no change" when they actually modify the resource. Fix: for security-critical properties (NSG rules, RBAC, firewall config), verify after deployment with az resource show or audit through Azure Policy compliance, not just the what-if output. 2. Leaving Terraform state storage wide open. The state file contains every secret you've ever passed through a variable — in plaintext. If the storage account has permissive RBAC or public access, that's a data breach waiting. Fix: dedicated resource group, private endpoint, least-privilege role: az role assignment create \ --assignee <pipeline-sp-object-id> \ --role "Storage Blob Data Contributor" \ --scope /subscriptions/<sub>/resourceGroups/rg-tfstate/providers/Microsoft.Storage/storageAccounts/sttfstate001 3. Running the hybrid pattern without ownership boundaries. If both tools can touch the same resources, you get two sources of truth fighting each other. Fix: write the boundary down — Bicep owns all Azure resources, Terraform owns all third-party resources — and enforce it with CODEOWNERS or a pipeline path filter. 4. Migrating on licensing FUD. If your Terraform estate works and the only concern is BSL, OpenTofu already solves it. Migrating Terraform→Bicep over license anxiety wastes months. Fix: evaluate migration on operational merit — blocked requirements or unsustainable state operations — not headlines. 5. Unpinned module and provider versions. Floating AVM module references or loose provider constraints break deployments on an ordinary Tuesday. Fix: pin everything: terraform { required_providers { azurerm = { source = "hashicorp/azurerm" version = "~> 4.30" } } } module storage 'br/public:avm/res/storage/storage-account:0.35.0' = { name: 'storageDeploy' params: { name: storageAccountName location: location } } Key Takeaways Bicep is the pragmatic default for Azure-only work — cleaner syntax, no state management, day-zero resource coverage, and Azure Verified Modules have closed the ecosystem gap. Terraform (or OpenTofu) wins for multi-cloud and third-party resources — 3,000+ providers, superior drift detection, and mature testing tooling remain unmatched. OpenTofu eliminates the BSL licensing concern — wire-compatible with Terraform, adds its own features, and removes the governance risk of the source-available license. The hybrid pattern works for Azure-first teams — Bicep for Azure, Terraform for third-party resources, with ownership boundaries written down. Don't migrate between tools unless you have a compelling reason — the operational cost of migration almost always exceeds the theoretical benefit. Standardize on one tool for new projects and let old ones attrit. The best IaC tool is the one your team will actually use correctly. A well-run Bicep estate outperforms a neglected Terraform one, and the reverse is also true. Pick the one that fits your team, your cloud strategy, and your operational maturity — then invest in making it work well. --- ## Agentic Observability: The Missing Layer in Enterprise AI Agent Deployments URL: https://wenfeng.my/agentic-observability-missing-layer-enterprise-ai-agent-deployments/ Author: admin Published: 2026-08-21 23:30:42 Modified: 2026-10-04 15:31:04 If you have ever deployed an AI agent into production and then watched it do something inexplicable — burn through tokens on a loop, call the wrong tool at the wrong time, or silently fail while the user sees a generic error — you already know the pain this article addresses. Traditional application monitoring tells you that a request arrived and a response left. It does not tell you why the agent chose one tool over another, where it spent three seconds deciding, or how much it cost to produce a single answer.Agentic observability fills that gap. It is the practice of instrumenting, tracing, and analysing every decision an AI agent makes during a request lifecycle. For enterprise teams running multi-agent systems on Azure — particularly those of us building for regulated markets in Southeast Asia — it is not optional. It is the missing layer that determines whether your agent deployment is a controlled capability or an expensive black box.This article walks through why observability matters for multi-agent systems, the specific signals you need to capture, how the emerging tooling landscape fits together, and how to implement the whole stack using Azure Monitor and Application Insights with minimal overhead.Why Traditional Observability Falls ShortA standard microservices architecture emits three categories of telemetry: logs, metrics, and traces. Application Insights captures all three out of the box. You get request duration, dependency call success rates, exception stacks, and distributed trace correlation. That is excellent for HTTP APIs, message consumers, and database-backed services.AI agents break this model in three ways.First, agent execution is non-deterministic. The same user prompt can lead to different tool chains, different reasoning paths, and different resource consumption. A fixed trace template does not capture the branching logic that agents actually follow.Second, agents make external calls that are invisible to standard dependency tracking. When an agent invokes a search API, a vector database, a code interpreter, or a third-party plugin, those calls may not go through standard HTTP clients that Application Insights auto-instruments. Without explicit custom dependency logging, you see gaps in the trace — the agent spent 200 milliseconds on "something," but you do not know what.Third, cost is per-decision, not per-request. In a traditional API, cost correlates roughly with compute time. In an agent system, a single user question might trigger five LLM calls, three tool invocations, and two retry loops. The cost is additive and variable. Without per-step cost attribution, your monthly Azure bill becomes a mystery.For enterprise teams — especially those handling sensitive data under PDPA or preparing for ISO 42001 compliance — these gaps are not just operational inconveniences. They are governance risks.The Signals That Matter: What to CaptureAgentic observability is not about collecting everything. It is about collecting the right things. Here are the five signal categories that give you complete visibility into agent behaviour.1. Agent Decision TracesEvery reasoning step an agent takes should emit a trace event. This includes the model call (which model, what prompt was sent, what tokens were used), the reasoning output (the agent's plan or decision), and the confidence or routing score if applicable.In a multi-agent system, you also need to capture inter-agent communication. When Agent A delegates a subtask to Agent B, that handoff should appear as a parent-child span in your distributed trace. Application Insights supports this natively through Operation ID correlation.2. Tool Call MonitoringEach tool invocation — whether it is a database query, an API call, a file operation, or a code execution — should be logged as a custom dependency. You need:Tool name and versionInput parameters (sanitised for PII)Output summary (not full payload — store that in blob storage if needed)DurationSuccess or failure statusToken consumption if the tool itself calls an LLMThis gives you a complete map of what your agent actually did, not just what it intended to do.3. Cost AttributionToken usage is the primary cost driver. You need to track input tokens, output tokens, and the model tier for every LLM call. Multiply by Azure OpenAI pricing tiers and you have real-time cost attribution per request, per user, per agent, and per tool.4. Latency BreakdownEnd-to-end latency is misleading for agents. A 10-second response might be 800ms of network time, 200ms of tool execution, and 9 seconds of LLM inference. Breaking latency into these components tells you where to optimise. If 90% of your latency is LLM inference, caching or using a faster model for simple queries will have more impact than optimising your database queries.5. Safety and Compliance EventsFor enterprise deployments, you need to track every content filter trigger, every guardrail intervention, every PII detection event, and every human-in-the-loop escalation. These are not just operational metrics — they are audit evidence.Where the Tooling Landscape StandsBefore building on Azure Monitor, it is worth knowing what the specialised market offers, because several dedicated LLM observability platforms have matured quickly.Langfuse is the most prominent open-source option. It provides trace capture at the prompt and completion level, evaluation scoring, dataset management, and prompt versioning. Its recent v4 release moved to a ClickHouse backend for substantially faster ingestion, which matters once you are tracing thousands of agent turns a day.LangSmith, from the LangChain team, positions itself as an end-to-end agent platform: observability, evaluation suites, and deployment tooling in one product. Arize Phoenix is the open-source tracing and evaluation layer from Arize AI, designed to pair with their production monitoring platform for teams that want ML-style observability over LLM workloads.What these specialised tools add over a general APM is LLM-native depth: full prompt and response capture, evaluation scores computed with an LLM-as-a-judge, and side-by-side trace comparison across prompt versions.The Azure-native path is different. Application Insights plus OpenTelemetry keeps every trace inside your existing governance, retention, and cost perimeter. The OpenTelemetry GenAI semantic conventions now standardise the span and attribute names for LLM calls — including dedicated agent spans and an OpenAI instrumentation — so your instrumentation stays vendor-neutral and portable.The trade-off is straightforward. Specialised tools give you a faster developer loop and richer evaluation features, but they are another vendor and another data perimeter — and remember that agent traces contain prompts, which in regulated industries may carry customer data. Azure-native gives you unified alerting, workbooks, KQL, and role-based access with no extra data egress, but you build the LLM-specific views yourself.My recommendation for enterprise teams already running Azure Monitor: start there. Add a specialised tool later if evaluation-heavy workflows justify the additional data perimeter.Practical Implementation with Azure Monitor and Application InsightsLet me walk through a concrete implementation. This is the pattern I use with enterprise clients running agents on Azure Container Apps or Azure Kubernetes Service.Step 1: Instrument Your Agent FrameworkWhether you are using Semantic Kernel, AutoGen, LangChain, or a custom framework, you need a telemetry wrapper around every agent action. Here is a Python example using the Application Insights SDK: from azure.monitor.opentelemetry import configure_azure_monitor from opentelemetry import trace # Configure Application Insights configure_azure_monitor( connection_string="InstrumentationKey=your-key;IngestionEndpoint=https://your-region.in.applicationinsights.azure.com/" ) tracer = trace.get_tracer("agent-service") def execute_agent_turn(user_prompt: str, agent_id: str): with tracer.start_as_current_span("agent_turn") as span: span.set_attribute("agent.id", agent_id) span.set_attribute("user.prompt_length", len(user_prompt)) # Log the reasoning step with tracer.start_as_current_span("reasoning") as reasoning_span: model_response = call_llm(agent_id, user_prompt) reasoning_span.set_attribute("model.name", model_response.model) reasoning_span.set_attribute("tokens.input", model_response.usage.prompt_tokens) reasoning_span.set_attribute("tokens.output", model_response.usage.completion_tokens) reasoning_span.set_attribute("cost.usd", calculate_cost(model_response)) # Log the tool call as a custom dependency if model_response.tool_call: with tracer.start_as_current_span("tool_call") as tool_span: tool_span.set_attribute("tool.name", model_response.tool_call.name) tool_result = execute_tool(model_response.tool_call) tool_span.set_attribute("tool.duration_ms", tool_result.duration_ms) tool_span.set_attribute("tool.success", tool_result.success) return build_response(model_response) To provision the Application Insights workspace itself as infrastructure-as-code, the Bicep looks like this: param location string = resourceGroup().location param workspaceName string = 'law-agent-observability' param appInsightsName string = 'appi-agent-observability' resource logAnalytics 'Microsoft.OperationalInsights/workspaces@2025-07-01' = { name: workspaceName location: location properties: { sku: { name: 'PerGB2018' } retentionInDays: 90 } } resource appInsights 'Microsoft.Insights/components@2020-02-02' = { name: appInsightsName location: location kind: 'web' properties: { Application_Type: 'web' WorkspaceResourceId: logAnalytics.id IngestionMode: 'LogAnalytics' } } output connectionString string = appInsights.properties.ConnectionString Two details matter here. IngestionMode: 'LogAnalytics' routes telemetry through Log Analytics, which is what unlocks KQL queries and log-based metrics across your whole platform. And the retentionInDays setting is your first cost lever — agent traces are verbose, so 90 days is a reasonable starting point for interactive queries, with longer retention exported to cheap blob storage if compliance requires it.Step 2: Set Up Custom Metrics in Application InsightsStandard metrics (request rate, duration, failures) appear automatically. For agent-specific metrics, emit them through OpenTelemetry (the same exporter already configured for traces): from opentelemetry import metrics meter = metrics.get_meter("agent-service") tokens_counter = meter.create_counter( "agent.tokens.total", description="Total tokens consumed per agent", ) cost_counter = meter.create_counter( "agent.cost.usd", description="Estimated cost per agent turn", ) llm_latency = meter.create_histogram( "agent.latency.llm_ms", unit="ms", ) # Record per LLM call, with agent/model dimensions tokens_counter.add(total_tokens, { "agent_id": agent_id, "model": model_name, }) cost_counter.add(cost_usd, { "agent_id": agent_id, }) llm_latency.record(llm_latency_ms, {"agent_id": agent_id}) These custom metrics land in Application Insights alongside your standard metrics and can be queried with KQL.Step 3: Build a Workbooks DashboardAzure Monitor Workbooks give you rich, shareable dashboards. Create a workbook with these sections:Agent Health: Request rate, success rate, and p95 latency by agent IDCost Overview: Daily token consumption and estimated cost, broken down by model tier and agentTool Usage: Heatmap of tool invocations, success rates, and average durationError Analysis: Top errors by agent, with drill-down to trace detailsSafety Events: Content filter triggers, guardrail interventions, and PII detections over timeStep 4: Set Up AlertsConfigure alert rules for the metrics that matter: # Alert on high token consumption (potential runaway agent) az monitor metrics alert create \ --name "agent-high-token-usage" \ --resource-group my-rg \ --scopes "/subscriptions/sub-id/resourceGroups/my-rg/providers/microsoft.insights/components/my-app-insights" \ --condition "total agent.tokens.total > 50000" \ --window-size 5m \ --evaluation-frequency 1m \ --action my-action-group # Alert on elevated failed-request volume (requests/failed is the standard # Application Insights failed-request metric — it is a count, so use a # total aggregation, not a percentage) az monitor metrics alert create \ --name "agent-error-rate" \ --resource-group my-rg \ --scopes "/subscriptions/sub-id/resourceGroups/my-rg/providers/microsoft.insights/components/my-app-insights" \ --condition "total requests/failed > 25" \ --window-size 10m \ --action my-action-group Step 5: Implement Distributed Tracing Across Multi-Agent SystemsWhen multiple agents collaborate, you need trace context propagation. The OpenTelemetry standard handles this. Pass the trace context through your agent messaging layer: from opentelemetry.propagate import inject def delegate_to_specialist(task, parent_context): headers = {} inject(headers, context=parent_context) # Send task with trace headers to specialist agent response = requests.post( "https://specialist-agent.internal/task", json=task, headers=headers ) return response This ensures that when you view a trace in Application Insights, you see the complete journey across all agents as a single correlated operation.Cost Tracking: Making AI Spend VisibleCost management deserves special attention because it is often the first executive concern. Here is the pattern that works.Every LLM call emits a trace event with token counts and model tier. A background function aggregates these into hourly cost summaries written to Log Analytics. A Power BI dashboard — or an Azure Monitor Workbook — surfaces the data.The key metrics to track:Cost per agent: Which agents are the most expensive?Cost per user: Are power users driving disproportionate spend?Cost per task type: Is the research agent spending 10x more than the triage agent? Is that justified?Cost trend: Are costs growing linearly with usage, or is there a non-linear component (e.g., retry loops) driving costs up?For Southeast Asian enterprises billing in MYR, SGD, or THB, you can extend the aggregation to apply real-time exchange rates. This makes cost discussions with finance teams much more productive.Latency Analysis: Where Does the Time Go?Agent latency is a user experience killer. A 15-second response feels broken, even if it is producing a high-quality answer. Here is how to diagnose and fix latency.Instrument every step with start and end timestamps. In Application Insights, this gives you a waterfall view of each agent turn. Common findings:LLM inference dominates (70-90% of total latency): Use streaming responses to show partial results. Consider a smaller, faster model tier — gpt-5-nano or gpt-5-mini on Azure OpenAI — for simpler subtasks within your agent pipeline.Tool calls are slow (databases, external APIs): Add caching layers. Pre-fetch common data. Use semantic caching for repeated queries.Agent-to-agent communication adds up (multi-agent systems): Reduce the number of agent hops. Use direct tool calls instead of delegating to specialist agents when the task is simple.Retry loops inflate latency: Set explicit timeout and retry budgets. Log retry counts so you can identify problematic patterns.A Note on Compliance and GovernanceFor enterprise deployments in regulated industries — financial services, healthcare, government — agentic observability is also a compliance enabler. PDPA in Malaysia, PDPA in Singapore, and emerging AI governance frameworks across ASEAN all point toward the same direction: you need to demonstrate control over automated decision-making systems.Detailed trace logs showing exactly what data an agent accessed, what decisions it made, and what outputs it produced are the foundation of an AI audit trail. Store traces in immutable storage (Azure Blob with WORM policy or Azure Immutable Blob Storage) alongside your Application Insights data for long-term retention.Pitfalls I See RepeatedlyA few failure patterns show up in almost every agent observability implementation I review.Pitfall 1: Logging full prompts and responses by default. It feels thorough, but agent traces containing prompts can carry customer data straight into your telemetry pipeline. That creates a second copy of personal data with its own access surface, retention clock, and compliance exposure. Sanitise or truncate by default, and gate full-payload capture behind an explicit flag with its own access control.Pitfall 2: Telemetry volume surprises. Agent traces are an order of magnitude more verbose than HTTP service traces — every LLM call, every tool invocation, every handoff emits spans and custom dimensions. At Log Analytics ingestion pricing, an uninstrumented-for-volume rollout can quietly double your monitoring bill. Set daily caps, sample verbose attributes, and review ingestion volume after the first week, not the first month.Pitfall 3: Alerting on averages. Average token consumption and average latency hide the exact behaviour you care about — the runaway loop, the retry storm, the single agent turn that cost ten times normal. Alert on totals and p95/p99 percentiles, and use per-agent dimensions so one misbehaving agent does not get masked by nine healthy ones.Pitfall 4: Treating observability as an afterthought. Teams that add tracing after their first production incident end up retrofitting instrumentation into framework internals. The cost of instrumenting from day one is a few wrapper functions per agent action. The cost of not having traces during your first incident is measured in hours of blind debugging.Pitfall 5: Cost attribution without a denominator. Tracking total spend is easy; making it actionable requires dividing by something — cost per resolved ticket, per completed task, per active user. Without a business denominator, cost conversations stay abstract and budgets get cut instead of steered.Key TakeawaysAgentic observability is non-negotiable for production AI agents. Traditional monitoring captures the request-response cycle but misses the reasoning, tool calls, and cost dynamics that define agent behaviour.Instrument every decision, tool call, and LLM invocation. Use OpenTelemetry and Application Insights custom spans and dependencies to build a complete picture of what your agent does and why.Cost tracking must be per-step, not per-request. Token consumption varies wildly between agent turns. Per-step attribution is the only way to manage AI spend responsibly.Latency analysis reveals where to optimise. Waterfall traces that break down LLM inference time, tool execution, and inter-agent communication tell you exactly which component to tune.Start with Azure Monitor and Application Insights. They are already in your Azure subscription, they scale with your agent deployment, and they integrate with the rest of your Azure governance stack. No additional vendor needed.Have questions about implementing agentic observability in your Azure environment? Reach out at wenfeng.my/ or find me on LinkedIn. I work with enterprise teams across Southeast Asia on production AI agent architectures. --- ## Azure Landing Zone in a Weekend — The 48-Hour Enterprise Setup Guide for 2026 URL: https://wenfeng.my/azure-landing-zone-in-a-weekend-the-48-hour-enterprise-setup-guide-for-2026/ Author: admin Published: 2026-08-21 06:30:09 Modified: 2026-10-04 15:27:04 I remember my first Azure project. We spun up a subscription, deployed a few VMs, pointed a DNS record at them, and called it "the cloud." Six months later we had security alerts nobody read, cost overruns nobody tracked, and a network topology that looked like spaghetti thrown at a whiteboard. Sound familiar?That experience is exactly why Azure Landing Zones exist — and why I want to show you how to deploy one from scratch in a single weekend. Not a theoretical whitepaper exercise. A real, working, production-ready environment you can stand up on Saturday morning and have governance, security, networking, and identity wired in by Sunday evening.If you have been putting off landing zone adoption because it feels like a three-month consulting engagement, this guide is for you. I am going to walk through the entire process using Azure CLI and Bicep, covering every major component. By the end, you will have a management group hierarchy, hub-spoke network, Azure Policy enforcement, Defender for Cloud, Sentinel SIEM, and Entra ID integration — all automated and reproducible.Let us get started.Why Landing Zones Matter (and Why 2026 Is the Right Time)Microsoft Azure Landing Zone is not new — the Cloud Adoption Framework has been around since 2019. But two things have changed in 2026 that make this the perfect moment to adopt.First, the Malaysia West (Kuala Lumpur) Azure region is fully operational, with three availability zones and in-country data residency for customer data. For Southeast Asian enterprises — especially those answering to PDPA or Bank Negara requirements — this removes the last major blocker for in-region deployment.Second, the landing zone accelerators — ALZ-Bicep, ALZ-Terraform, and the portal-based Azure Landing Zone Accelerator — have matured significantly. They are faster, more modular, and far easier to customize than they were even a year ago. You no longer need to wade through nested ARM templates manually; the heavy lifting is done by well-maintained, community-reviewed modules.The business case is straightforward: without a landing zone, every team provisions resources their own way. Network isolation is inconsistent. Policies are either nonexistent or bolted on after a breach. Cost management is an afterthought. A landing zone gives you guardrails that are automated, auditable, and scalable.The 48-Hour RoadmapHere is how I break down the weekend:Saturday Morning (Hours 0-4): Management group hierarchy and subscription governanceSaturday Afternoon (Hours 4-8): Hub-spoke networking and connectivitySunday Morning (Hours 8-14): Identity, Azure Policy, and Defender for CloudSunday Afternoon (Hours 14-20): Sentinel, monitoring, and landing zone subscriptionsThis is aggressive but achievable, especially with the code I am sharing below. Let me walk through each phase.Phase 1: Management Groups and Subscription StructureEverything starts with governance hierarchy. Azure Management Groups let you apply policies and RBAC at scale, above individual subscriptions. The default hierarchy gives you "Tenant Root Group" — but you should not leave it empty.First, log in and set your defaults: # Login and set your subscription context az login az account set --subscription "your-platform-subscription-id" az account show --query id --output tsv # Set default values for all commands az configure --defaults location=malaysiawest Now create the management group structure. I recommend four child groups under the tenant root, following the Cloud Adoption Framework pattern: # Create management group hierarchy MG_ROOT="your-tenant-id" az account management-group create --name "Contoso" az account management-group create --name "Contoso-Platform" --parent "Contoso" az account management-group create --name "Contoso-Online" --parent "Contoso" az account management-group create --name "Contoso-Sandbox" --parent "Contoso" az account management-group create --name "Contoso-Management" --parent "Contoso" This gives you a clean separation: Platform for shared services, Online for internet-facing workloads, Sandbox for experimentation, and Management for monitoring and logging.Next, create the subscriptions themselves: # Create platform subscriptions using alias az account alias create \ --name "contoso-connectivity-alias" \ --display-name "contoso-connectivity" \ --billing-scope "/providers/Microsoft.Billing/billingAccounts/{your-billing-account-id}" \ --workload "Production" az account alias create \ --name "contoso-management-alias" \ --display-name "contoso-management" \ --billing-scope "/providers/Microsoft.Billing/billingAccounts/{your-billing-account-id}" \ --workload "Production" # Move subscriptions under the right management group az account management-group subscription add \ --name "Contoso-Platform" \ --subscription "contoso-connectivity" Weekend Tip: Run this in parallel with creating your Bicep parameter files. While the subscriptions are provisioning (it takes a few minutes), start drafting your network topology.Phase 2: Hub-Spoke Networking with BicepThe hub-spoke topology is the backbone of any landing zone. The hub subscription hosts shared services — VPN/ExpressRoute gateways, Azure Firewall, DNS, and jump boxes — while spoke subscriptions connect to the hub via Virtual Network Peering.Here is a simplified Bicep template for the hub network. Note that every subnet that needs protection gets an NSG — define them explicitly rather than relying on defaults: // hub-network.bicep param location string = resourceGroup().location param hubVnetName string = 'vnet-hub' param hubVnetAddressPrefix string = '10.0.0.0/16' resource nsgBastion 'Microsoft.Network/networkSecurityGroups@2025-01-01' = { name: 'nsg-bastion' location: location properties: {} // Add Bastion service tag rules before production use } resource nsgManagement 'Microsoft.Network/networkSecurityGroups@2025-01-01' = { name: 'nsg-management' location: location properties: {} // Deny internet inbound; allow management traffic only } resource hubVnet 'Microsoft.Network/virtualNetworks@2025-01-01' = { name: hubVnetName location: location properties: { addressSpace: { addressPrefixes: [hubVnetAddressPrefix] } subnets: [ { name: 'GatewaySubnet' properties: { addressPrefix: '10.0.1.0/24' } } { name: 'AzureFirewallSubnet' properties: { addressPrefix: '10.0.2.0/24' } } { name: 'AzureBastionSubnet' properties: { addressPrefix: '10.0.3.0/24' networkSecurityGroup: { id: nsgBastion.id } } } { name: 'Management' properties: { addressPrefix: '10.0.4.0/24' networkSecurityGroup: { id: nsgManagement.id } routeTable: { id: rtHub.id } } } ] } } resource rtHub 'Microsoft.Network/routeTables@2025-01-01' = { name: 'rt-hub' location: location properties: { routes: [ { name: 'DefaultToFirewall' properties: { addressPrefix: '0.0.0.0/0' nextHopType: 'VirtualAppliance' nextHopIpAddress: '10.0.2.4' } } ] } } output hubVnetId string = hubVnet.id For a spoke subscription, create a separate Bicep module that deploys a virtual network and peers it to the hub: // spoke-network.bicep param location string = resourceGroup().location param spokeVnetName string = 'vnet-workload' param spokeVnetAddressPrefix string = '10.1.0.0/16' param hubVnetId string resource spokeVnet 'Microsoft.Network/virtualNetworks@2025-01-01' = { name: spokeVnetName location: location properties: { addressSpace: { addressPrefixes: [spokeVnetAddressPrefix] } subnets: [ { name: 'WorkloadSubnet' properties: { addressPrefix: '10.1.1.0/24' } } { name: 'DataSubnet' properties: { addressPrefix: '10.1.2.0/24' } } ] } } resource spokeToHubPeer 'Microsoft.Network/virtualNetworks/peerings@2025-01-01' = { parent: spokeVnet name: 'peering-to-hub' properties: { allowVirtualNetworkAccess: true allowForwardedTraffic: true allowGatewayTransit: false remoteVirtualNetwork: { id: hubVnetId } } } Deploy the hub first, then the spokes. The key addresses you need to plan upfront:Hub VNet: 10.0.0.0/16 (gateways, firewall, bastion, management)Spoke VNet 1: 10.1.0.0/16 (production workloads)Spoke VNet 2: 10.2.0.0/16 (non-production / DevTest)Spoke VNet 3: 10.3.0.0/16 (shared services like AKS)Do not forget to enable "Use Remote Gateway Transit" on the hub-side peering when you add VPN or ExpressRoute gateways later. I have seen this forgotten more times than I can count.Weekend Tip: Use a parameter file to separate environment-specific values. Create hub.bicepparam and spoke-prod.bicepparam with different address spaces. This keeps your templates clean and reusable.Phase 3: Identity with Entra IDNo landing zone is complete without proper identity governance. Entra ID (formerly Azure AD) is the backbone of your zero-trust architecture. Here is what you need to wire up on Sunday morning.First, ensure your subscriptions are governed by Entra ID groups, not individual user accounts. Create groups for RBAC using the Azure CLI: # Create an Entra ID group for production workloads az ad group create \ --display-name "Contoso-Prod-Contributors" \ --mail-nickname "contoso-prod-contributors" # Assign Contributor role at the landing zone subscription scope az role assignment create \ --assignee-object-id "$(az ad group show \ --group 'Contoso-Prod-Contributors' \ --query id -o tsv)" \ --role "Contributor" \ --scope "/subscriptions/your-prod-subscription-id" For privileged access, deploy Azure Privileged Identity Management (PIM) and Conditional Access policies. This is one area where I recommend using the Microsoft Graph PowerShell SDK alongside the Azure CLI: # Require MFA and compliant device for admin access # (run in PowerShell with Microsoft.Graph module) New-MgIdentityConditionalAccessPolicy -DisplayName "Require MFA for Admins" -State "enabled" -Conditions @{ Applications = @{IncludeApplications = @("All")} Users = @{IncludeGroups = @("your-admin-group-id")} Platforms = @{IncludePlatforms = @("windows", "macOS", "linux")} } -GrantControls = @{ Operator = "OR" BuiltInControls = @("mfa") } Weekend Tip: Use Microsoft Entra Connect Cloud Sync if you have an on-premises AD. If you are cloud-native, configure Entra ID as your primary identity provider and enable passwordless authentication (FIDO2 or Windows Hello) from day one.Phase 4: Azure Policy for GovernanceAzure Policy is your automated compliance engine. Without it, teams will inevitably create resources in the wrong regions, use unapproved VM sizes, or deploy public endpoints that should be private.Start with Azure Policy built-ins — every Azure tenant ships with them. A good first guardrail is the built-in "Allowed locations" policy, which restricts where resources can be created: # Assign the built-in "Allowed locations" policy at the root management group # (definition ID: e56962a6-4747-49cd-b67b-bf8b01975c4c) az policy assignment create \ --name "AllowedLocations" \ --display-name "Allowed locations for Contoso" \ --policy "e56962a6-4747-49cd-b67b-bf8b01975c4c" \ --scope "/providers/Microsoft.Management/managementGroups/Contoso" \ --params '{ "listOfAllowedLocations": { "value": ["malaysiawest", "southeastasia"] } }' # Browse more built-ins you may want az policy definition list \ --query "[?contains(displayName, 'Landing Zone') || contains(displayName, 'Azure Security')].{name:name, displayName:displayName} | [0:10]" \ --output table For the full enterprise governance stack, you do not need to author these yourself: the ALZ-Bicep modules and the Azure Landing Zone Accelerator assign the complete Azure Landing Zone policy initiatives for you. For a weekend build, start with a handful of built-ins.For more granular control, create custom policies. For example, a policy that requires tags on all resources. Note the quoting: the rule references a tag name with square brackets, so keep the JSON in a file or escape carefully — this trips people up constantly: cat > /tmp/require-costcenter.json <<'EOF' { "if": { "not": { "field": "tags['CostCenter']", "exists": true } }, "then": { "effect": "audit" } } EOF az policy definition create \ --name "RequireCostCenterTag" \ --display-name "Require CostCenter tag on resources" \ --rules @/tmp/require-costcenter.json \ --mode All I cannot stress this enough: start with "audit" effect and switch to "deny" once you are confident in your policy definitions. Deploying a deny policy on a Saturday night without testing will generate angry Slack messages on Sunday morning.Phase 5: Defender for Cloud and SentinelSecurity is not optional, and in 2026 it should be the first thing you turn on, not the last.Enable Defender for Cloud across your subscriptions: # Enable Defender for Cloud plans az security pricing create --name "VirtualMachines" --tier "Standard" az security pricing create --name "AppServices" --tier "Standard" az security pricing create --name "SqlServers" --tier "Standard" az security pricing create --name "StorageAccounts" --tier "Standard" az security pricing create --name "KeyVaults" --tier "Standard" # Set security contact (current CLI parameter set) az security contact create \ --name "default" \ --emails "security@contoso.com" \ --notifications-by-role '{"state":"On","roles":["Owner"]}' \ --alert-notifications '{"state":"On","minimalSeverity":"High"}' For Sentinel, deploy a Log Analytics Workspace and enable the solution: # Create Log Analytics Workspace az monitor log-analytics workspace create \ --resource-group "rg-security" \ --workspace-name "law-contoso-security" \ --location "malaysiaweast" # Onboard Sentinel (Microsoft.SecurityInsights onboardingStates, stable API) az rest --method put \ --uri "https://management.azure.com/subscriptions/{subscription-id}/resourceGroups/rg-security/providers/Microsoft.OperationalInsights/workspaces/law-contoso-security/providers/Microsoft.SecurityInsights/onboardingStates/default?api-version=2024-03-01" \ --body '{}' Connect your diagnostic logs from the hub firewall, NSGs, and key resources to the workspace. Then enable the Microsoft Sentinel solution for built-in analytics rules and workbooks.Weekend Tip: Use the Sentinel content hub to deploy ready-made detection rules. The "Microsoft Network Session Essentials" solution and the "Microsoft Defender for Cloud" connector give you immediate visibility without writing custom KQL queries.Common Pitfalls I Have SeenAfter helping organizations across Southeast Asia deploy landing zones, here are the mistakes that cost the most time:Address space overlap. Plan your IP addressing before you write a single line of code. Overlapping address spaces between on-prem and Azure will block VPN/ExpressRoute integration and force you to rebuild subnets.Forgetting DNS. Private DNS zones for Azure services (Blob, SQL, Key Vault) are not optional. Without them, service endpoints and private endpoints do not resolve correctly. Include privatelink.blob.core.windows.net and peers in your hub deployment.Over-engineering on day one. You do not need Azure Arc, Arc-enabled Kubernetes, and a full DevOps pipeline in the first weekend. Get the foundation solid — networking, identity, policy — and layer advanced capabilities on top incrementally.Ignoring cost. A landing zone subscription with Azure Firewall Premium, Bastion, and Defender for every resource tier can easily run $2,000+/month before you deploy a single workload. Right-size for your actual needs and use the Azure Advisor cost recommendations from day one.No automation for subscription provisioning. If your team needs a new landing zone subscription and the process is "file a ticket and wait two weeks," your governance model is already broken. Automate subscription creation through ServiceNow, GitHub Actions, or Azure DevOps pipelines.Wrapping Up: Your Monday Morning ChecklistIf you followed this guide over the weekend, here is what you should have on Monday:Management group hierarchy under your tenant rootAt least three subscriptions: connectivity, management, and one workload zoneHub-spoke VNet peering with routing through Azure FirewallAzure Policy assigned at the management group level (start with audit)Defender for Cloud enabled on all subscriptionsSentinel deployed and receiving logsEntra ID RBAC groups mapped to subscription scopesAll infrastructure defined in Bicep and committed to a repoThat is a production-ready foundation. Not perfect — no landing zone is on day one — but governed, secure, and scalable.Key TakeawaysStart with governance, not resources. Management groups and Azure Policy give you guardrails before any workload lands. Design the hierarchy first, then provision subscriptions into it.Automate everything with Bicep. Manual portal deployments are not repeatable and not auditable. Every networking component, policy assignment, and security configuration should be defined as code.Plan your IP addressing like your career depends on it. Overlapping address spaces between on-premises and Azure are the number one cause of rework in landing zone deployments. Document your scheme before you start.Enable Defender and Sentinel early, not late. Security visibility is not a Phase 2 activity. Turn on Defender for Cloud and Sentinel on day one so you can catch misconfigurations while they are cheap to fix.Iterate, do not boil the ocean. A weekend landing zone is a foundation. Layer advanced capabilities — private endpoints, Azure Arc, FinOps dashboards, CI/CD pipelines — in subsequent sprints as your team matures.This guide is part of my ongoing series on Azure architecture for Southeast Asian enterprises. If you are deploying in the Malaysia West region or need help with PDPA compliance in Azure, feel free to reach out. More articles at [wenfeng.my/](https://wenfeng.my/). --- ## Git-Ape: Microsoft's Agentic Migration Framework That Converts AWS to Azure URL: https://wenfeng.my/git-ape-microsoft-agentic-migration-framework-aws-to-azure/ Author: admin Published: 2026-08-20 23:30:51 Modified: 2026-10-04 15:27:15 By Law Wen Feng, Principal Solution Architect — MalaysiaIf you have ever migrated a workload from AWS to Azure, you know the pain. You stare at a Terraform repo full of EC2 instances, IAM roles, S3 buckets, and load balancers, and you begin the painstaking process of manually translating each resource into its Azure equivalent. The networking is different. The identity model is different. The deployment patterns are different. And buried inside your user_data scripts are tribal knowledge assumptions that no documentation ever captured.Microsoft recently released an open-source framework that attacks this exact problem — not with a simple find-and-replace, but with AI agents that understand what your infrastructure is actually trying to do and propose Azure-native replacements. The framework is called Git-Ape, and it represents a new category of tooling: agentic platform engineering.I have been following Git-Ape since it appeared on GitHub under the Azure organisation. After spending time with the source code, the documentation, and Microsoft's published migration walkthrough, I want to share what this tool is, how it works, and whether it is something your team should consider for your next cloud migration project.What Is Git-Ape?Git-Ape is a platform engineering framework built on top of GitHub Copilot. It is a multi-agent system — meaning it uses multiple specialised AI agents working together — that plans, validates, and deploys Azure workloads with security gates, cost analysis, and CI/CD pipeline integration baked in.The project is open source under the MIT license and lives at github.com/Azure/git-ape. It ships as a VS Code agent plugin and a GitHub Copilot CLI plugin.The name is a deliberate play on "monkey work" — the tedious, repetitive tasks that engineers waste hours on during migration and deployment. Git-Ape aims to eliminate that monkey work.Here is the critical thing to understand: Git-Ape is not a simple syntax converter. It does not translate Terraform HCL into Bicep line by line. Instead, it performs what Microsoft calls "intent extraction and architecture remapping." The agents read your existing deployment code, understand what it is trying to accomplish, and then propose an Azure-native equivalent that follows Azure best practices.How Git-Ape Automates AWS-to-Azure MigrationThe migration workflow follows a structured pipeline managed by a central orchestrator agent called @git-ape. Here is the step-by-step process.Step 1: Prerequisite ValidationBefore anything happens, Git-Ape checks that your tooling is in place. It validates Azure CLI (az), GitHub CLI (gh), jq, and git — along with your active authentication sessions (az login, gh auth login). For AWS-to-Azure migrations, it additionally confirms your AWS credentials are valid (read-only access is enough for repo analysis). You can trigger the tooling check manually with the /prereq-check command. # Verify all tools are installed and authenticated /prereq-check This catches the most common failure mode: discovering midway through a migration that your Azure subscription token expired three days ago.Step 2: Intent Extraction from the Source RepositoryThis is where the AI agents earn their keep. Git-Ape reads your source AWS repository — every Terraform file, every script, every README — via the GitHub API. It then extracts deployment intent:What runtime is being used (e.g., Next.js on Node.js 20)What compute resources are provisioned (e.g., EC2 t3.micro)What networking exists (VPCs, subnets, load balancers, security groups)What storage and identity patterns are in playWhat monitoring is configured (or, more commonly, what is missing)In Microsoft's published walkthrough, Git-Ape analysed a Contoso Outdoors web application running on AWS with EC2, an Application Load Balancer, S3 for artifact storage, IAM for identity, and PM2 for process management. It extracted every detail including the fact that the app built itself on startup from a tarball downloaded from S3 — a pattern that was not documented anywhere.Step 3: Azure Architecture RemappingBased on the extracted intent, Git-Ape proposes an Azure-native mapping. The agents choose the most appropriate Azure services, not just the closest 1:1 equivalent. In the Contoso example: AWS SourceAzure TargetWhy EC2 + user_data.sh + PM2App Service (B1, Linux, Node 20)No server patching; platform-managed runtime Application Load BalancerApp Service built-in load balancer + HTTPSSimpler ingress with improved security defaults IAM roles/policiesSystem-assigned Managed IdentityNo stored credentials; Azure RBAC S3 artifact storageDropped entirelyCI builds and deploys artifacts directly Terraform apply from laptopGitHub Actions (OIDC) + BicepRepeatable deployments with audit trail No monitoringApplication Insights + Log AnalyticsFirst-class telemetry from day one The key insight here is that Git-Ape does not just copy your architecture across clouds. It recommends the Azure-native pattern. The S3 bucket disappeared because the agents recognised that deploying via CI makes artifact storage in a separate bucket redundant.Step 4: Design CritiqueBefore generating any code, Git-Ape runs its output through an independent design review agent — a "rubber duck critique" step. In the published example, this caught two blocking issues:Build-on-startup anti-pattern: The original AWS setup ran npm ci && npm run build on every instance boot. The critique agent flagged this as a production anti-pattern causing slow cold starts, nondeterministic deployments, and broken scale-out. The fix: build in CI, deploy a ready-to-run artifact via zip deploy.Unnecessary storage mirroring: The initial plan included Azure Blob Storage to mirror S3. The critique pointed out this added cost and complexity with zero benefit when deploying via CI. The fix: drop Blob Storage entirely.This critique step is what separates Git-Ape from a glorified code generator. It applies architectural reasoning before writing a single line of infrastructure code.Step 5: Infrastructure Code GenerationGit-Ape generates deployment-ready artifacts. In the migration walkthrough, it produced:Bicep template (infra/main.bicep): An approximately 80-line template defining the App Service Plan, Web App, Application Insights, and Log Analytics workspace. Notably, it chose Bicep over Terraform — Azure's native IaC format — which resulted in significantly cleaner output.GitHub Actions workflow (.github/workflows/deploy.yml): A complete CI/CD pipeline with checkout, Node.js setup, npm build, Azure login via OIDC, resource group creation, Bicep deployment, zip deploy, and health checks.Documentation: Updated README.md and a DEPLOYMENT_GUIDE.md tailored to Azure.The generated Azure deployment reduced the estimated monthly cost from approximately $34 (AWS) to $13 (Azure) while adding security features like HTTPS-only, TLS 1.2 enforcement, FTP disabled, and Managed Identity.Step 6: Optional Quality Gates and DeploymentGit-Ape's full pipeline includes additional stages that were not exercised in the migration walkthrough but are available:Security gate: Automated security analysis of generated templates with blocking/pass verdictsPreflight validation: ARM what-if analysis before any deploymentCost estimation: Real-time Azure Pricing API queries for per-resource cost breakdownsDeployment execution: az deployment create with progress monitoringIntegration testing: Post-deployment health checksDrift detection: Ongoing reconciliation between live Azure state and stored deployment artifactsSupported Services and Output FormatsGit-Ape supports any Azure resource deployable via Azure Resource Manager. The agents back this up with a dedicated reference-lookup skill that fetches official Azure REST API and ARM template documentation — exact property schemas, required fields, and latest stable API versions — before generating or modifying templates, so they are not limited to a hardcoded list of resource types.The framework's Template Generator natively produces ARM templates in JSON, and in the AWS migration walkthrough the output landed on Bicep (Azure's domain-specific language for ARM) — not by default, but after human-in-the-loop steering. Git-Ape initially started generating Terraform with the AzureRM provider; the reviewer changed the instruction to "write the most efficient code — doesn't have to be Terraform," and the result was a single ~80-line Bicep template instead of 200+ lines of Terraform.For CI/CD, it generates GitHub Actions workflows. The deployment model uses OIDC federation for authentication — no stored secrets, with full audit trails.The framework ships with 8 specialised agents and 15 skills (as of writing):Agents include the Requirements Gatherer, Template Generator, Resource Deployer, Principal Architect (WAF review), Policy Advisor, and IaC Exporter, all orchestrated by the central @git-ape agent.Skills cover areas like ARM API reference lookup, Cloud Adoption Framework naming, resource availability checks, security analysis, Azure Policy compliance (CIS, NIST), cost estimation, drift detection, and integration testing.Current LimitationsGit-Ape is marked as EXPERIMENTAL and is not production-ready. Microsoft is explicit about this. Here is what you need to know:Cloud coverage is Azure-only. Git-Ape generates Azure deployments. If you need multi-cloud output or are targeting GCP, this is not the tool. The AWS analysis capability reads AWS repos as input, but the output is always Azure-native.GitHub-centric. The CI/CD pipeline integration is built around GitHub Actions. If your team uses Azure DevOps, GitLab CI, or another platform, you will need to adapt the generated workflows.ARM and Bicep, not Terraform. Git-Ape's Template Generator defaults to ARM JSON templates, and in the migration walkthrough it settled on Bicep — either way, Terraform is not a first-class output format. Teams standardised on Terraform will find the generated output requires translation. In the Contoso example, that trade paid off: roughly 80 lines of Bicep replaced 200+ lines of Terraform — but adoption reality in the field is more nuanced, and many enterprises have Terraform skills and pipelines that Bicep cannot simply replace.Requires GitHub Copilot. The entire framework runs on GitHub Copilot's agent infrastructure. You need a Copilot subscription, and the agents run within Copilot's context limits. This is not a standalone CLI tool you can run independently.Not production-grade. The warning banner is there for a reason. The agents can generate incorrect resource configurations, misidentify service mappings, or produce templates that fail validation. Every output requires human review.Limited to deployment code analysis. Git-Ape reads your IaC repositories, not your running infrastructure directly (though the IaC Exporter skill can import live resources). If your actual deployment diverges from what your Terraform code describes, Git-Ape will migrate the code, not reality.Practical Adoption Guidance for Teams in Southeast AsiaFor Azure-first enterprise teams in Malaysia and across the region, here is my practical take on when and how to use Git-Ape.Use it for discovery, not just migration. Even if you are not migrating from AWS, Git-Ape's intent extraction and architecture critique capabilities are valuable. Point it at your existing Azure deployments to get an independent architectural review. The security gate and WAF assessment skills work on any Azure workload.Start with a sandbox. Do not run Git-Ape against your production subscription. Use a dedicated development subscription and treat the first run as a learning exercise. The framework's safety model is designed around this — it requires explicit confirmation before any deployment.Pair it with Azure Migrate. Git-Ape handles the infrastructure-as-code conversion. Azure Migrate handles discovery, assessment, and server migration. They are complementary tools, not competing ones. Use Azure Migrate to understand your estate, then use Git-Ape to generate the deployment patterns for target workloads.Contribute back. The project is MIT-licensed and actively accepting feedback. If you find gaps in service mapping for Azure resources commonly used in the region — Azure Government, sovereign cloud configurations, regional availability zone patterns — the GitHub Issues page is the place to flag them.Watch the Copilot licensing cost. Git-Ape requires GitHub Copilot, which means per-developer licensing. For teams evaluating this tool, factor the Copilot subscription cost into your migration budget alongside the Azure consumption costs.Getting StartedIf you want to experiment with Git-Ape today:Install the VS Code extension from the VS Code MarketplaceSign in with az login and configure the Azure MCP serverOpen a repository containing Azure or AWS deployment codeIn Copilot Chat, try: @git-ape deploy a Python function app or @git-ape analyse this AWS repo for Azure migrationReview the output critically — treat it as a draft, not a final answerThe framework is evolving rapidly. Microsoft has published a detailed walkthrough of the AWS-to-Azure migration workflow on the Azure DevBlogs that is worth reading alongside this article.Key TakeawaysGit-Ape is not a syntax translator. It performs intent extraction from existing infrastructure code and proposes Azure-native replacements that follow best practices — a fundamentally different approach from simple code conversion.The design critique step is the most valuable feature. Before generating any code, an independent agent reviews the proposed architecture and catches anti-patterns. In Microsoft's published example, it identified two blocking design issues that would have degraded production reliability.It is experimental — use it for learning and discovery. Git-Ape is explicitly not production-ready. Treat generated output as a well-informed starting draft that requires human review, security validation, and adaptation to your organisation's standards.The agent model reduces maintenance burden. Instead of maintaining 50+ Bicep or Terraform modules, you update the agent's context. It reads live Azure API specifications at generation time, so it stays current without manual module updates.Factor in the GitHub Copilot dependency. Git-Ape runs on Copilot's infrastructure. Evaluate the per-developer licensing cost alongside the migration efficiency gains before committing to adoption.Have questions about Git-Ape or cloud migration in general? Reach out on wenfeng.my/ or connect with me on LinkedIn. --- ## Malaysia Cloud Migration in 2026 — From Lift-and-Shift to Strategic Transformation URL: https://wenfeng.my/malaysia-cloud-migration-in-2026-from-lift-and-shift-to-strategic-transformation/ Author: admin Published: 2026-08-20 06:30:30 Modified: 2026-08-20 06:30:30 Malaysia Cloud Migration in 2026 — From Lift-and-Shift to Strategic TransformationIf you had told me three years ago that half of Malaysian enterprises would be running production workloads in the cloud by 2025, I would have raised an eyebrow. Yet here we are. Industry estimates peg cloud adoption among mid-to-large enterprises at 50–60% as of late 2025, and the trajectory is accelerating. The Malaysian government's MyDIGITAL blueprint — launched in February 2021, with its cloud-first direction for public sector systems — has given the entire ecosystem a directional nudge that private sector CIOs cannot ignore.But here is the uncomfortable truth I see repeated across client engagements in Kuala Lumpur, Penang, and Johor Bahru: most organisations that "went to cloud" in 2023–2024 did so by lifting and shifting virtual machines from on-premises data centres into Azure VMs or AWS EC2 instances. The servers changed address; the architecture did not change at all. And the bill? Often higher than the data centre it replaced.The question for 2026 is no longer "should we migrate?" It is "how do we migrate well — and how do we turn cloud from a cost centre into a genuine competitive advantage?" That is what this article is about.The Current State of PlayLet me set the scene with what I observe on the ground.Adoption is broad but shallow. Many Malaysian enterprises have some workloads in the cloud — typically email, file storage, or a subset of development environments. But mission-critical systems, databases with regulatory sensitivity, and core ERP workloads often remain on-premises. The "cloud-first" policy exists on paper; practice is more cautious.MyDIGITAL is shaping procurement. The blueprint's emphasis on shared services, open data, and digital government has a downstream effect on enterprise procurement. Vendors and system integrators are aligning their proposals to MyDIGITAL pillars. If you are pitching a transformation project to a GLC or government agency, your architecture review panel will ask how your design aligns with the blueprint.Hyperscaler presence is real. Microsoft's Azure Southeast Asia region (Singapore) and the Azure Malaysia West region (generally available since May 2025) have made Azure a default consideration for Malaysian enterprises. AWS and Google Cloud have similarly expanded local partnerships. The infrastructure is there. The gap is in strategic adoption, not connectivity.Talent remains the bottleneck. This is the part nobody likes to talk about. The demand for cloud architects, platform engineers, and FinOps practitioners outstrips supply. This is exactly why a clear migration strategy — one your team can execute without hiring twenty consultants — matters so much.A Practical Migration Strategy FrameworkOver the past two years, I have refined a framework that works for mid-sized Malaysian enterprises (500–5,000 employees, RM 50M–500M annual IT spend). It has five phases. I will keep it practical — no consulting-babble, just what actually works.Phase 1: Discover and ClassifyBefore you write a single ARM template, you need to know what you have. Use Azure Migrate or Cloudamize to inventory your on-premises estate. The output should be a classified workload map:Tier 1 (Migrate first): Stateless web applications, development/test environments, file servers, collaboration tools.Tier 2 (Modernize as you migrate): Line-of-business applications with moderate customisation, internal APIs, reporting workloads.Tier 3 (Evaluate carefully): Databases with regulatory constraints, legacy ERP, systems with hard-coded IP dependencies.This classification is not just technical — it involves your compliance officer (for PDPA considerations), your finance team (for budget impact), and your business unit leads (for dependency mapping).Phase 2: Landing Zone ArchitectureDo not skip this. I have seen organisations jump straight to migration and end up with a cloud environment that looks like a data centre disaster. A landing zone establishes your governance guardrails: subscription structure, networking, identity, policy, and monitoring.Here is a simplified Bicep template that provisions an Azure landing zone management group and subscription structure. This is the kind of IaC that should be version-controlled and peer-reviewed before anyone touches production:// landing-zone.bicep — Azure Landing Zone Foundations // Deploy with: az deployment mg create --template-file landing-zone.bicep --management-group-id <root-mg-id> targetScope = 'managementGroup' @description('Root management group for Malaysia enterprise landing zone') param orgName string = 'wenfeng' // Deployed at management group scope: with no explicit parent, // this management group is created under the tenant root group. resource mgFoundation 'Microsoft.Management/managementGroups@2023-04-01' = { name: '${orgName}-foundation' properties: { displayName: '${orgName} Foundation' } } resource mgProd 'Microsoft.Management/managementGroups@2023-04-01' = { name: '${orgName}-prod' properties: { displayName: 'Production Workloads' details: { parent: { id: mgFoundation.id } } } } resource mgNonProd 'Microsoft.Management/managementGroups@2023-04-01' = { name: '${orgName}-nonprod' properties: { displayName: 'Non-Production (Dev/Test/Staging)' details: { parent: { id: mgFoundation.id } } } } resource mgSandbox 'Microsoft.Management/managementGroups@2023-04-01' = { name: '${orgName}-sandbox' properties: { displayName: 'Sandbox and Experimentation' details: { parent: { id: mgFoundation.id } } } } The principle is simple: governance before workloads. Your sandbox subscription lets teams experiment without risk. Your non-production subscription mirrors production networking. Your production subscription has the strictest policies.Phase 3: Migrate in Waves, Not in a Big BangThis is where discipline matters. Group workloads into migration waves of 5–15 VMs each. Each wave should be:Self-contained — migrating a batch of web servers and their database backend together, not separating them.Testable — include rollback criteria. If the application does not pass smoke tests within the cutover window, you roll back.Time-boxed — every wave gets a two-week sprint: one week for preparation and pre-staging, one week for cutover and validation.Azure Migrate project creation and appliance registration are handled through the Azure portal or the Az.Migrate PowerShell module — the Azure CLI's az migrate extension does not yet cover project creation or appliance setup. Here is the PowerShell sequence for project creation, plus the networking setup in Azure CLI:#!/usr/bin/env pwsh # Phase 3: Azure Migrate — Project Creation (PowerShell, Az.Migrate module) # Step 1: Create the resource group and Azure Migrate project New-AzResourceGroup -Name rg-azure-migrate-my -Location southeastasia New-AzMigrateProject ` -Name "MY-Enterprise-Migration-2026" ` -ResourceGroupName rg-azure-migrate-my ` -Location southeastasia # Step 2: Download and register the Azure Migrate appliance # The appliance (OVA/VHD) and its registration key are generated in the # Azure portal: Azure Migrate hub → Servers → Discover → select the # appliance type, then download and configure on-premises. #!/bin/bash # Phase 3: Target resource group and VNet (Azure CLI) az group create --name rg-prod-workloads-my --location southeastasia az network vnet create \ --resource-group rg-prod-workloads-my \ --name vnet-prod-malaysia \ --address-prefixes 10.10.0.0/16 \ --subnet-name subnet-app \ --subnet-prefixes 10.10.1.0/24 az network vnet subnet create \ --resource-group rg-prod-workloads-my \ --vnet-name vnet-prod-malaysia \ --name subnet-data \ --address-prefixes 10.10.2.0/24 Phase 4: Optimise and ModerniseOnce workloads are running in Azure, the lift-and-shift is done — but the transformation is not. This is where you start right-sizing (shutting down those oversized VMs that followed you from the data centre), moving to PaaS where it makes sense, and introducing containers or serverless for appropriate workloads.I typically see 30–40% cost reduction in the first six months post-migration just from right-sizing alone. Azure Advisor is your friend here, but a structured FinOps practice — with monthly cost reviews and automated alerts — is essential.Phase 5: Operationalise and GovernThis phase never really ends. Establish monitoring with Azure Monitor and Log Analytics, implement Azure Policy for compliance guardrails, and create runbooks for incident response. If you are operating in regulated industries, set up Azure Confidential Computing for sensitive workloads and ensure your logging meets audit requirements.Which Workloads to Migrate FirstBased on my experience with Malaysian enterprises, here is my prioritised list:Start with these: - Development and test environments (lowest risk, immediate cost savings) - Web applications behind load balancers (stateless, easy to migrate) - File servers and collaboration platforms (move to Azure Files / SharePoint) - Internal APIs and microservices (containerise if already Dockerised)Migrate next: - SQL Server databases (use Azure Database Migration Service for minimal downtime) - Line-of-business applications with moderate customisation - Reporting and analytics workloads (great candidates for Microsoft Fabric — the successor and consolidation of Azure Synapse Analytics)Evaluate carefully: - Core ERP systems (SAP, Oracle) — consider hyperscaler-certified offerings - Workloads with hard-coded IP or MAC address dependencies - Systems processing classified or highly sensitive personal dataMulti-Cloud vs Azure-First: The Malaysian ContextI get this question at every client workshop. Here is my honest take.For most Malaysian enterprises, an Azure-first strategy makes more sense than an aggressive multi-cloud approach. The reasons are practical:PDPA and data residency. Azure's Southeast Asia region and the Malaysia West region give you clear data residency within the region. While AWS and Google Cloud have similar offerings, Azure's existing enterprise agreements and government cloud alignment in Malaysia are ahead.Enterprise agreement consolidation. Most Malaysian enterprises already have Microsoft EA agreements covering Office 365, Teams, and Windows licensing. Adding Azure to an existing EA unlocks significant cost advantages — Azure Hybrid Benefit alone can save up to 80% on Windows Server and up to 85% on SQL Server compared to pay-as-you-go rates.Talent availability. The Malaysian market has more Azure-certified professionals than AWS or GCP certified ones. This is not a permanent state, but it is the current reality and affects your delivery timelines.MyDIGITAL alignment. Microsoft's partnership with the Malaysian government on digital transformation initiatives means Azure tooling and compliance frameworks are often pre-aligned with government procurement requirements.When multi-cloud makes sense: If you have specific SaaS workloads that run best on GCP (BigQuery, Vertex AI), or if you are in an industry where regulatory requirements mandate infrastructure diversity, then a deliberate multi-cloud architecture — not accidental multi-cloud — is warranted. The key word is deliberate.Data Sovereignty and PDPA: What You Actually Need to KnowThe Personal Data Protection Act 2010 (PDPA) and its amendments require that personal data processing complies with specific consent, storage limitation, and security principles. Here is what matters for cloud migration:Where does your data live? Azure's Southeast Asia region (Singapore) and the Malaysia West region (GA May 2025) ensure data stays within the region. For PDPA compliance, you need to know exactly which Azure regions your data is stored in. Use Azure Policy to enforce resource creation in approved regions only:# Enforce data residency: only allow specific Azure regions az policy assignment create \ --name "enforce-my-region" \ --display-name "Restrict to Southeast Asia Regions" \ --policy "/providers/Microsoft.Authorization/policyDefinitions/e56962a6-4747-49cd-b67b-bf8b01975c4c" \ --params '{"listOfAllowedLocations":{"value":["southeastasia","asia"]}}' \ --scope "/subscriptions/{your-subscription-id}" Who has access? Azure RBAC with PIM (Privileged Identity Management) ensures that access to personal data is auditable and time-bound. For organisations processing sensitive personal data, Azure Confidential Computing adds an extra layer — data is encrypted even during processing.How do you handle cross-border transfers? If your Azure resources span regions, or if your development team accesses production data from outside Malaysia, you need a documented cross-border transfer assessment. The PDPA amendments are tightening around this — get ahead of it.Common Pitfalls (and How to Avoid Them)Pitfall 1: Migrating without optimisation. Lift-and-shift is Phase 1, not Phase 3. If you stop at rehosting, you will pay cloud prices for data centre architecture. Budget for the optimisation phase.Pitfall 2: Skipping the landing zone. I have repaired more cloud environments that were built without governance than I care to count. Two weeks of landing zone setup saves months of rework.Pitfall 3: Ignoring FinOps. Cloud costs are not fixed. Without ongoing cost governance, your Azure bill will surprise you every month. Implement Azure Cost Management with budget alerts from Day 1.Pitfall 4: Underestimating networking. Malaysian enterprises often underestimate the complexity of cloud networking — VPN connectivity, hybrid DNS, ExpressRoute circuits. Plan your network architecture as carefully as your compute.Pitfall 5: No rollback plan. Every migration wave needs a rollback criteria document. If the application does not meet defined performance and functionality thresholds within the cutover window, you roll back. Period.ConclusionCloud migration in Malaysia in 2026 is not about whether to move — it is about how to move strategically. The MyDIGITAL blueprint has set the direction. The hyperscaler infrastructure is mature. The gap is in execution: disciplined landing zones, phased migration waves, post-migration optimisation, and ongoing governance.The organisations that will win are not the ones that migrate the fastest. They are the ones that migrate with purpose — aligning cloud strategy to business outcomes, respecting data sovereignty requirements, and building operational excellence from the start.Key TakeawaysClassify before you migrate. Use the Tier 1/2/3 framework to prioritise workloads by business criticality, technical complexity, and regulatory sensitivity. Not everything moves at once, and that is by design.Landing zone first, workloads second. Invest in governance guardrails — subscription structure, networking, identity, policy — before migrating a single workload. Two weeks of architecture saves months of rework.Azure-first is pragmatic for Malaysia. Enterprise agreement consolidation, regional data residency, PDPA alignment, and talent availability all favour Azure as the primary platform. Multi-cloud can be deliberate and secondary — not accidental.Optimisation is where the ROI lives. Lift-and-shift is the starting line, not the finish. Right-sizing, PaaS migration, and FinOps practices typically deliver 30–40% cost reduction in the first six months post-migration.PDPA compliance is not optional. Enforce data residency with Azure Policy, implement RBAC with PIM, and document cross-border data transfers. Get ahead of the tightening regulatory environment now, not after an audit. --- ## Open-Source LLMs 2026: DeepSeek-V4, Kimi-K2.6, and the Enterprise Self-Hosting Decision URL: https://wenfeng.my/open-source-llms-2026-deepseek-v4-kimi-k26-enterprise-self-hosting/ Author: admin Published: 2026-08-19 23:30:17 Modified: 2026-08-19 23:30:17 Open-Source LLMs 2026: DeepSeek-V4, Kimi-K2.6, and the Enterprise Self-Hosting DecisionBy Law Wen Feng, Principal Solution Architect — wenfeng.my/If you've been watching the open-source LLM space over the past twelve months, you know we've entered a new era. DeepSeek's V4 family openly competes with frontier closed-source models on reasoning and coding, and Moonshot AI's Kimi K2.6 has redefined what an agentic, code-first open-weight model can do. Both are open-weight. Both are serious. Both are MIT-licensed — with one caveat on Kimi that I'll cover.For enterprise teams in Southeast Asia — particularly those already running on Azure — the question is no longer whether open-source LLMs are viable. The question is: should we self-host, on what hardware, and at what real cost?This article breaks down both models with verified specifications, does the GPU memory math that most blog posts skip, lays out actual Azure Southeast Asia pricing, and walks through a deployment. If you're an Azure-first team in Malaysia, Singapore, or the broader region, this is for you.The Two Models: What You're Actually ComparingDeepSeek-V4DeepSeek-V4 ships in two variants, both designed for long-context reasoning, coding, and agentic workflows, with a 1M-token context window:DeepSeek-V4-Pro: 1.6 trillion total parameters, 49 billion active per token. This is the flagship — a sparse Mixture-of-Experts model built for maximum reasoning, coding, and agentic performance.DeepSeek-V4-Flash: 284 billion total parameters, 13 billion active per token. The cost-efficient option. It trails Pro on knowledge-heavy tasks due to its smaller scale, but reaches comparable quality on many production workloads at a fraction of the serving cost.The MoE architecture is the key point. Only a fraction of the total parameters activate on any given token, so inference compute per token is dramatically lower than a dense model of equivalent capability. But — and this is where most write-ups go wrong — all the weights still need to sit in GPU memory. Sparsity makes inference cheap; it does nothing for your memory footprint. I'll do that math properly below.DeepSeek-V4 is released under the MIT license, supporting commercial use, modification, and distribution with minimal restrictions.Kimi K2.6Moonshot AI's Kimi K2.6 is a roughly 1 trillion parameter MoE model with 32 billion active parameters per token, built around Multi-head Latent Attention (MLA) for efficient long-context handling, with a 256K-token context window. It is natively multimodal — a MoonViT vision encoder supports image and video input, though video understanding is still experimental and currently limited to the official API.Where K2.6 stands out:Long-horizon coding: Benchmark results are competitive with top closed-source models, and it sustains long autonomous coding sessions across frontend, backend, DevOps, and performance tuning — including niche languages like Zig.Agent swarm orchestration: K2.6 can decompose complex tasks into up to 300 parallel sub-agents across 4,000 coordinated steps — a major expansion from K2.5's 100 sub-agents and 1,500 steps.Proactive autonomous agents: In internal tests, a K2.6-backed agent operated autonomously for five days managing monitoring, incident response, and system operations without human oversight.A preserve_thinking mode that keeps full reasoning traces across turns, improving reliability in multi-turn agent workflows. Moonshot's Kimi Code CLI is the purpose-built framework for it.On licensing: K2.6 uses a modified MIT license. The sole modification — if you use it commercially with over 100 million monthly active users or over $20 million monthly revenue, you must prominently display "Kimi K2.6" in your product UI. For nearly every enterprise reading this, that clause is irrelevant.My short version: K2.6 is the specialist for code-centric and agentic workloads. DeepSeek-V4 is the generalist for reasoning, document processing, and multilingual enterprise AI.When Self-Hosting Actually Makes SenseBefore infrastructure and costs, let's establish when self-hosting is the right call. From client engagements across the region, self-hosting makes sense when at least one of these is true and cost is not your only lens:Data sovereignty requirements: Regulated data — financial records, healthcare information, government contracts under PDPA 2010 — that compliance will not allow to leave your infrastructure or flow to third-party API endpoints.Deep customization: You need to fine-tune on domain data (legal documents, technical manuals, proprietary codebases) and need full control of the weights.Extreme volume: You are processing tens of billions of tokens per month. API costs scale linearly; self-hosting amortizes fixed infrastructure.Vendor risk management: You've been burned by API deprecations, pricing changes, or rate limits.If none of these apply, API access is almost certainly cheaper and simpler. Don't romanticize self-hosting — it's an operational commitment. The break-even math below will show you exactly why.The GPU Memory Math Most Articles SkipMoE sparsity reduces compute per token, but every parameter still needs memory. Here's the honest arithmetic at different precisions: Model Total Params BF16 Weights FP8 Weights INT4 Weights DeepSeek-V4-Pro 1.6T ~3.2 TB ~1.6 TB ~800 GB DeepSeek-V4-Flash 284B ~568 GB ~284 GB ~142 GB Kimi K2.6 ~1T ~2.0 TB ~1.0 TB ~500 GB Now map that to real hardware, before KV cache and batch headroom:V4-Pro at FP8 (~1.6 TB): needs roughly 12–16x H200 (141 GB each) or 20+ H100 80GB. A single 8-GPU node is not enough at any practical precision.V4-Flash at INT4 (~142 GB): fits on 2x H100 NVL (94 GB each = 188 GB) with headroom for KV cache at moderate context.K2.6 at FP8 (~1.0 TB): fits on 8x H200 (1,128 GB total) — one node, tight but workable.This is the first place I see proposal math go wrong in the wild: "just rent an 8x H100 box" only works for the smallest of these three models, and only with aggressive quantization.Azure Reality Check: What You Can Actually Rent in Southeast AsiaThis is where it gets interesting for regional teams. I checked the live Azure retail pricing API for Southeast Asia (Singapore) while writing this:Standard_ND96isr_H200_v5 — 8x NVIDIA H200 (141 GB each, 1,128 GB total HBM3e), 96 vCPUs. Available in Southeast Asia at $110.24/hour (≈ $80,475/month pay-as-you-go). This is your workhorse for serious MoE serving in-region.Standard_NC80adis_H100_v5 — 2x NVIDIA H100 NVL (94 GB each = 188 GB), 80 vCPUs. $18.148/hour (≈ $13,248/month). The entry point for INT4-quantized serving.Standard_NC40ads_H100_v5 — 1x H100 NVL (94 GB), 40 vCPUs. $9.074/hour (≈ $6,624/month). Fine for experimentation, too small for these models in production.What's not on the Southeast Asia retail price list: the 8x H100 ND_H100_v5 series and the 8x A100 ND96asr_A100_v4 series. If your architecture document assumes "rent an ND96asr_v4 in Singapore," that assumption needs revisiting — the in-region frontier is now H200-based. (The A100 workhorse appears in other regions, e.g. around $27.20/hour Linux in West US 3, if you can tolerate the latency and data residency implications.)Risk: GPU capacity is quota-gated and supply-constrained. Request quota increases early, and have a fallback region agreed with your compliance team before you commit to a timeline.Cost Analysis: The Real NumbersUsing verified Southeast Asia pay-as-you-go pricing, 730 hours/month, plus realistic storage, networking, and operational overhead:Scenario A: DeepSeek-V4-Pro, FP8, on 2x ND96isr_H200_v5 (16x H200) Cost Component Monthly (USD) 2x ND96isr_H200_v5 (16x H200) ~$160,950 Storage (weights ~1.6 TB + working set) ~$400 Networking and egress ~$300–$600 Operational overhead (monitoring, patching, team time) ~$5,000–$8,000 Total ~$167,000–$170,000 Scenario B: Kimi K2.6, FP8, on 1x ND96isr_H200_v5 (8x H200) Cost Component Monthly (USD) ND96isr_H200_v5 (8x H200) ~$80,475 Storage (~1 TB weights) ~$300 Networking and egress ~$200–$400 Operational overhead ~$4,000–$6,000 Total ~$85,000–$87,000 Scenario C: DeepSeek-V4-Flash, INT4, on 1x NC80adis_H100_v5 (2x H100 NVL) Cost Component Monthly (USD) NC80adis_H100_v5 (2x H100 NVL) ~$13,248 Storage ~$150 Networking and egress ~$100–$200 Operational overhead ~$2,000–$3,000 Total ~$15,500–$16,600 Reserved capacity (1–3 year) can cut the VM line by roughly 40–60% — that's the first lever to pull if you commit.The API SideDeepSeek's published V4 API pricing (per million tokens, cache-miss rates): V4-Flash costs $0.22 input / $0.66 output off-peak (peak: $0.44 / $1.32); V4-Pro costs $0.66 input / $1.98 output off-peak (peak: $1.32 / $3.96). Cache-hit input drops to $0.007–$0.022, which matters enormously for RAG workloads with repeated context.Break-EvenTake Scenario C at ~$16K/month all-in. At V4-Flash off-peak API rates with a typical 3:1 input-to-output mix (blended ≈ $0.33 per million tokens), break-even lands around 47 billion tokens per month. That's billions of tokens per day, every day, on one workload.The math is unambiguous: unless you are at extreme scale, self-hosting is a compliance, sovereignty, or customization decision — not a cost decision. Anyone selling you self-hosting as the cheap option has not done this arithmetic.Practical Deployment: V4-Flash on AzureHere's a working deployment of DeepSeek-V4-Flash (INT4) on the NC80adis_H100_v5 — the most accessible entry point. Kimi K2.6 on the H200 node follows the same pattern, scaled up.Step 1: Provision# Resource group in Southeast Asia az group create --name rg-llm-inference --location southeastasia # 2x H100 NVL inference VM (request quota first: az vm list-skus --all) az vm create \ --resource-group rg-llm-inference \ --name vm-v4-flash \ --image Canonical:0001-com-ubuntu-server-jammy:22_04-lts-gen2:latest \ --size Standard_NC80adis_H100_v5 \ --admin-username azureuser \ --ssh-key-values ~/.ssh/id_rsa.pub \ --os-disk-size-gb 256 \ --data-disk-sizes-gb 1024 \ --nsg nsg-llm-inference # Never expose the inference port publicly — internal access only az network nsg rule create \ --resource-group rg-llm-inference \ --nsg-name nsg-llm-inference \ --name allow-internal-inference \ --priority 100 \ --source-address-prefixes VirtualNetwork \ --destination-port-ranges 8000 \ --protocol Tcp \ --access Allow Step 2: Install the Stacksudo apt update && sudo apt upgrade -y # NVIDIA driver + container toolkit (use the Azure-tuned NVIDIA package repo) sudo apt install -y nvidia-driver-570-open nvidia-utils-570 python3 -m venv ~/llm-env && source ~/llm-env/bin/activate pip install -U vllm transformers accelerate nvidia-smi # confirm both H100 NVL GPUs visible (2x 94 GB) Step 3: Serve# INT4 (W4A16) checkpoint of V4-Flash (~142 GB fits in 188 GB). # Note: the official deepseek-ai checkpoint ships FP8 attention + FP4 experts; # FP4 experts need Blackwell tensor cores, so on Hopper (H100) you need a # W4A16/INT4 conversion such as this community checkpoint. vllm serve baicai1145/DeepSeek-V4-Flash-0731-W4A16 \ --host 0.0.0.0 \ --port 8000 \ --tensor-parallel-size 2 \ --dtype bfloat16 \ --kv-cache-dtype fp8 \ --trust-remote-code \ --max-model-len 4096 \ --gpu-memory-utilization 0.95 # The checkpoint's tested default is max-model-len 4096; raise it only as # your KV-cache headroom allows (FP8 KV cache helps). Step 4: Verifycurl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "baicai1145/DeepSeek-V4-Flash-0731-W4A16", "messages": [ {"role": "system", "content": "You are a helpful enterprise assistant."}, {"role": "user", "content": "Summarize the key compliance requirements for personal data processing in Malaysia under PDPA 2010."} ], "max_tokens": 1024, "temperature": 0.7 }' Step 5: Production HardeningFor production, add:Internal Azure Load Balancer or Application Gateway (private frontend) in front of the endpoint — SSL termination and health probes included. A VM is not a PaaS private-link resource, so don't reach for Private Endpoint here; an internal load balancer plus VNet integration is the correct pattern.Azure Monitor + Prometheus/Grafana for GPU utilization, KV-cache pressure, TTFT, and throughput.Private endpoints for storage and Key Vault so model weights and credentials never touch the public internet.RBAC and Key Vault for any API credentials your serving layer needs.A capacity fallback plan: GPU stock fluctuates; know your alternate region before you need it.Decision Framework: Quick Reference Factor DeepSeek-V4 Kimi K2.6 Best for General-purpose enterprise AI, documents, multilingual, reasoning Coding agents, agentic workflows, multimodal pipelines Flagship V4-Pro: 1.6T total / 49B active ~1T total / 32B active Lean option V4-Flash: 284B total / 13B active None yet Context window 1M tokens 256K tokens Multimodal Text-focused Native image/video input (video experimental) License MIT Modified MIT (UI attribution above 100M MAU / $20M monthly revenue) Cheapest viable serving (verified) INT4 Flash on 2x H100 NVL ≈ $13.2K/month VM FP8 on 8x H200 ≈ $80.5K/month VM My Recommendation for Southeast Asian Enterprise TeamsAfter working through these numbers across several client engagements, here's my honest take:Start with DeepSeek-V4-Flash. It's the pragmatic entry point — INT4 quantized, it fits on the NC80adis_H100_v5 that's actually priced and available in Singapore, and it covers the majority of enterprise use cases: summarization, extraction, internal Q&A, report generation.Choose Kimi K2.6 when the workload is code or agents. If you're building developer tooling, automated code review, or multi-step autonomous workflows, K2.6's long-horizon coding and swarm orchestration are genuinely ahead of the curve. Budget for the H200 node.Reserve V4-Pro for compliance-mandated frontier quality. At ~$167K/month all-in for a 16x H200 deployment, it's a serious budget line — justify it with a regulation, not a roadmap.And don't self-host unless you have to. I know that's a strange note to end a deployment guide with, but the break-even math says it plainly: below tens of billions of tokens per month, the API wins on cost by orders of magnitude. Hybrid is the right default — API for development and most production traffic, self-hosted only where sovereignty, customization, or scale demand it.Two years ago we were debating whether open models could match GPT-3.5. Today we're deploying open models that trade blows with the frontier — and the enterprise decision is no longer about capability. It's about operations, compliance, and honest arithmetic.Key TakeawaysDeepSeek-V4 (MIT) and Kimi K2.6 (modified MIT) are frontier-class open-weight models — but MoE sparsity cuts compute per token, not memory: all weights still need GPU RAM.The honest GPU math: V4-Pro needs 12–16x H200 at FP8; K2.6 needs an 8x H200 node; V4-Flash at INT4 fits on 2x H100 NVL — the only option affordable on a departmental budget.Azure Southeast Asia reality: the 8x H100 and 8x A100 ND-series aren't on the regional retail price list — the in-region workhorse is the ND96isr_H200_v5 at $110.24/hour (≈ $80.5K/month).Self-hosting is a compliance or scale decision, not a cost decision: at verified DeepSeek API rates, break-even for the cheapest self-hosted scenario sits around 47 billion tokens/month.Deploy with vLLM behind an internal load balancer, NSG-locked, with Azure Monitor for GPU and latency metrics — that's the battle-tested pattern for enterprise LLM serving in this region.Have questions about LLM deployment on Azure in Southeast Asia? Reach out at wenfeng.my/ — I'm always happy to chat architecture. --- ## Azure Landing Zone Cost Optimization: Right-Size Without Overspending URL: https://wenfeng.my/azure-landing-zone-cost-optimization-rightsizing/ Author: admin Published: 2026-08-19 06:30:29 Modified: 2026-10-04 15:33:08 Every landing zone deployment I've reviewed in the past year has told the same story: the hub works beautifully, the networking is sound, the governance policies are in place — and the hub's monthly bill is meaningfully higher than it needs to be. Not because anyone made a mistake, but because the standard landing zone pattern defaults toward Premium SKUs, larger VPN gateways, and comprehensive monitoring, as if every enterprise needs enterprise-maximum from day one.Here's what I keep seeing in practice: a firewall running well under its throughput ceiling on Premium. A VPN gateway sized for 1 Gbps of aggregate traffic while the office link carries a couple of hundred megabits. A Log Analytics workspace ingesting tens of gigabytes a day, much of it diagnostics nobody has queried in six months.None of this is negligence. The Cloud Adoption Framework's defaults are designed to be safe, and "safe" is the right default when you don't know the workload yet. The problem starts when teams deploy the defaults and then never revisit them.I've been working with Malaysian enterprises on Azure landing zones since the Malaysia West region launched, and cost optimization is the conversation that keeps coming up — usually with the wrong starting point. Most teams hear "cost optimization" as "turn things off." It isn't. It means right-sizing what you have: deliberate SKU choices based on measured workload patterns.Let me walk through the playbook I use, grounded in verified Azure documentation and current Malaysia West pricing, with the CLI commands and IaC templates you can take back to your team.The Over-Provisioning Problem: Why It HappensFour patterns show up in almost every landing zone review I do in Kuala Lumpur and across Southeast Asia:1. The "Premium for Safety" firewall trap. Teams select Azure Firewall Premium because the most sensitive reference architectures recommend it. Premium is genuinely valuable — it adds fully managed IDPS and outbound TLS inspection. But if your inspection requirement is L3–L7 filtering, threat-intelligence-based alerting, and web categories, Standard delivers that with autoscaling up to 30 Gbps. Premium scales to 100 Gbps, which is far more than most mid-size hubs will ever push.2. VPN gateway over-sizing. A typical Malaysian enterprise office — SaaS applications, occasional file transfers, remote-worker VPN — rarely sustains more than 200–300 Mbps across the site link. That is VpnGw1AZ territory (650 Mbps aggregate benchmark). Yet I regularly find VpnGw2AZ or larger deployed "because the template said so."3. Log Analytics over-ingestion. Diagnostic settings get configured once — all categories enabled, long retention — and nobody revisits them. The workspace quietly grows to tens of gigabytes per day. NSG flow logs, every Application Gateway access log, every Key Vault audit event: all ingested, all billed, mostly unread.4. Azure Advisor recommendations ignored. Advisor surfaces specific cost recommendations — right-size underutilized VMs, delete unattached disks, purchase reservations. Teams dismiss them as "risky" and keep paying for idle capacity. The governance fear is real, but the answer isn't ignoring the data — it's building a review process around it.The Landing Zone Cost PlaybookHere's the practical framework I run with clients.Step 1: Get the BaselineBefore optimizing anything, know what you're actually spending, and where. The Cost Management query REST API — called via az rest, since the az costmanagement CLI extension has no built-in query subcommand — gives you a cost breakdown by resource group: # Cost breakdown by resource group, month to date (via Cost Management REST API) az rest --method post \ --uri "https://management.azure.com/subscriptions/<sub-id>/providers/Microsoft.CostManagement/query?api-version=2023-11-01" \ --body '{"type":"ActualCost","timeframe":"MonthToDate","dataset":{"granularity":"None","aggregation":{"totalCost":{"name":"Cost","function":"Sum"}},"grouping":[{"type":"Dimension","name":"ResourceGroup"}]}}' \ --query "properties.rows" -o table # Current VPN gateway SKU az network vnet-gateway show \ --resource-group rg-hub-networking \ --name vgw-hub-myw \ --query "sku.name" -o tsv # Current Azure Firewall SKU tier az network firewall show \ --resource-group rg-hub-security \ --name fw-hub-myw \ --query "sku.tier" -o tsv Then measure actual VPN utilization for the last 30 days. VPN gateways expose tunnel and gateway bandwidth metrics in Azure Monitor: # Gateway S2S Bandwidth (REST metric name: AverageBandwidth), last 30 days az monitor metrics list \ --resource "/subscriptions/<sub-id>/resourceGroups/rg-hub-networking/providers/Microsoft.Network/virtualNetworkGateways/vgw-hub-myw" \ --metric "AverageBandwidth" \ --aggregation Average \ --interval P1D \ --query "value[0].timeseries[0].data[-10:]" \ -o table Numbers, not vibes. This is the entire foundation of the exercise.Step 2: Right-Size Hub ServicesNot every hub service needs the same treatment. Here's the decision matrix I use, with verified Malaysia West pay-as-you-go rates (Azure Retail Prices API, listed in USD): ServiceOver-provisioned defaultRight-sized optionMonthly cost: default → right-sized (USD, Malaysia West PAYG) Azure FirewallPremium (USD 1.75/hr deployment)Standard (USD 1.25/hr deployment)≈1,278 → ≈913 (deployment only, excludes capacity units and data processed) VPN GatewayVpnGw2AZ (USD 0.54/hr)VpnGw1AZ (USD 0.21/hr)≈394 → ≈153 Log AnalyticsPAYG ingestion (USD 2.691/GB)100 GB/day commitment tier (USD 229.32/day)Pay-per-GB → ≈6,880/mo flat at 100 GB/day Azure BastionStandard (USD 0.29/hr first gateway)Basic (USD 0.19/hr)≈212 → ≈139 The firewall decision is where the biggest money is, so let's be precise about the SKUs, because there is a widespread misconception here:Basic is capped at 250 Mbps throughput. It is not a "small enterprise firewall" — it suits small environments only. I would not put it in a production landing zone hub.Standard autoscales up to 30 Gbps and includes threat intelligence, DNS proxy, and web categories. This is the right choice for most mid-size enterprise hubs.Premium adds fully managed IDPS and TLS inspection, scaling to 100 Gbps. Choose it when those security features are genuinely required — typically highly sensitive applications such as payment processing — not as a blanket default.For a realistic scenario — a 500-user Malaysian enterprise, three branch offices, hybrid connectivity, about 120 concurrent remote users — the hub math looks roughly like this (USD/month, Malaysia West, PAYG, 730 hours): Over-provisioned hub (Premium-everything defaults): Azure Firewall Premium (deployment 1,278 + ~1 avg capacity unit 80): ~1,358 VPN Gateway VpnGw2AZ (394) + 3 S2S conns (33) + 120 P2S conns (876): ~1,303 Log Analytics PAYG @ 80 GB/day (80 x 2.691 x 30): ~6,458 Azure Bastion Standard (0.29 x 730): ~212 ------------------------------------------------------------------------ Total: ~9,331 Right-sized hub (measured workload justifies every downgrade): Azure Firewall Standard (deployment 913 + ~1 avg capacity unit 51): ~964 VPN Gateway VpnGw1AZ (153) + 3 S2S conns (33) + 120 P2S conns (876): ~1,062 Log Analytics PAYG after pruning to 40 GB/day (40 x 2.691 x 30): ~3,229 Azure Bastion Basic (0.19 x 730): ~139 ------------------------------------------------------------------------ Total: ~5,394 That's roughly 42% lower — and here's the honest read of those numbers: the single biggest saving is the Log Analytics line, and it comes from pruning ingestion, not from any pricing trick. SKU right-sizing (firewall, gateway, Bastion) contributes roughly a fifth of the total; the rest is deciding you don't need to ingest 40 GB/day of diagnostics nobody queries.Note that the VPN line barely moves: at 120 concurrent P2S users, the connection charges (USD 0.01/hr each) dominate over the gateway deployment fee. The gateway SKU downgrade is real money (USD 394 → 153), but don't expect it to transform a hub with heavy remote access. Do that math against your measured bandwidth before and after, not before.At higher ingestion volumes the commitment tier earns its place: above roughly 85 GB/day, PAYG (USD 2.691/GB) costs more per GB than the 100 GB/day commitment tier's effective USD 2.293/GB (USD 229.32/day in Malaysia West). But buying a tier to cover pruned volumes — say committing to 100 GB/day when you're actually ingesting 40 GB/day — means paying for headroom you engineered away. Prune first, commit second.Step 3: VPN Gateway Right-Sizing — and the 2026 SKU Consolidation You Must Know AboutVPN gateway costs are the most misunderstood part of a landing zone, and in 2026 there's a compliance angle too.Microsoft is consolidating the VPN Gateway SKU portfolio: non-availability-zone SKUs (VpnGw1–VpnGw5) can no longer be created as new gateways since November 1, 2025, and are scheduled for deprecation after September 2026. All new deployments should use the AZ SKUs (VpnGw1AZ–VpnGw5AZ), which are also getting price reductions as part of the consolidation.The current AZ SKU benchmarks: SKUGenAggregate throughput benchmarkS2S/VNet tunnelsP2S IKEv2/OpenVPN VpnGw1AZ1650 Mbps30250 VpnGw2AZ1 / 21 Gbps / 1.25 Gbps30500 VpnGw3AZ1 / 21.25 Gbps / 2.5 Gbps301,000 VpnGw4AZ25 Gbps1005,000 VpnGw5AZ210 Gbps10010,000 If you still have non-AZ gateways in your landing zone, upgrade them now — VpnGw1→VpnGw1AZ and VpnGw2→VpnGw2AZ are supported in-place upgrades, and Microsoft's consolidation guidance says no downtime is expected for gateways that use Standard SKU public IP addresses (if your gateway still carries a Basic SKU public IP, upgrade that to Standard first): az network vnet-gateway update \ --resource-group rg-hub-networking \ --name vgw-hub-myw \ --sku VpnGw1AZ For right-sizing between AZ tiers, parameterize the SKU in your IaC so the decision is explicit and reviewable. Terraform: resource "azurerm_virtual_network_gateway" "hub_vpn" { name = "vgw-hub-myw" location = azurerm_resource_group.hub_networking.location resource_group_name = azurerm_resource_group.hub_networking.name type = "Vpn" vpn_type = "RouteBased" sku = var.vpn_gw_sku ip_configuration { name = "vpnGwConfig" public_ip_address_id = azurerm_public_ip.vgw_pip.id private_ip_address_allocation = "Dynamic" subnet_id = azurerm_subnet.gateway_subnet.id } } variable "vpn_gw_sku" { description = "VpnGw1AZ: up to 650 Mbps / 30 S2S tunnels. VpnGw2AZ: 1-1.25 Gbps / 500 P2S. Size on measured bandwidth, not template defaults." type = string default = "VpnGw1AZ" validation { condition = contains(["VpnGw1AZ", "VpnGw2AZ", "VpnGw3AZ", "VpnGw4AZ", "VpnGw5AZ"], var.vpn_gw_sku) error_message = "Only AZ-supported SKUs are allowed (VpnGw1-5 non-AZ SKUs are scheduled for deprecation after September 2026)." } } One caveat: down-sizing a gateway (say VpnGw2AZ to VpnGw1AZ) isn't always an in-place operation — check the SKU upgrade eligibility matrix first; some transitions require delete/recreate with a maintenance window. Plan it like a change, not a config tweak.Step 4: Log Analytics Cost ControlsLog Analytics is the silent budget killer in landing zones. Three controls: # Check current workspace configuration az monitor log-analytics workspace show \ --resource-group rg-hub-monitoring \ --workspace-name law-hub-myw \ --query "{DailyQuotaGB:dailyQuotaGb, Retention:retentionInDays, Sku:sku.name}" \ -o table # Set a daily ingestion quota cap to prevent cost surprises (in GB/day) az monitor log-analytics workspace update \ --resource-group rg-hub-monitoring \ --workspace-name law-hub-myw \ --quota 100 Then prune what you actually ingest — this is where the real savings are. Audit diagnostic settings on the noisy resources and disable categories nobody queries.For IaC, the correct way to express a commitment tier is the CapacityReservation SKU with capacityReservationLevel — there is no separate commitment-plan child resource: resource lawWorkspace 'Microsoft.OperationalInsights/workspaces@2023-09-01' = { name: 'law-hub-myw' location: resourceGroup().location properties: { sku: { name: 'CapacityReservation' capacityReservationLevel: 100 // GB/day commitment tier } retentionInDays: 90 } } Step 5: Prevent Re-Over-Provisioning with Azure PolicyRight-sizing is wasted if the next deployment over-provisions again. Use Azure Policy to bound SKU choices at the management group level: targetScope = 'managementGroup' resource skuPolicy 'Microsoft.Authorization/policyDefinitions@2021-06-01' = { name: 'restrict-vpn-gateway-skus' properties: { displayName: 'Restrict VPN gateway SKUs to right-sized AZ SKUs' policyType: 'Custom' mode: 'Indexed' policyRule: { if: { allOf: [ { field: 'type'; equals: 'Microsoft.Network/virtualNetworkGateways' } { field: 'Microsoft.Network/virtualNetworkGateways/sku.name'; notIn: allowedSkus } ] } then: { effect: 'deny' } } parameters: { allowedSkus: { type: 'Array' defaultValue: [ 'VpnGw1AZ' 'VpnGw2AZ' 'VpnGw3AZ' ] } } } } The governance mindset flips: instead of enforcing maximum SKUs "for safety," you cap the ceiling and require justification for anything above the reviewed baseline.Step 6: Build a Monthly Review CadenceThe most important optimization is the simplest: a scheduled review. Here's the script I leave with clients: #!/bin/bash # Monthly landing zone cost review — run against your hub subscription SUBSCRIPTION="<your-hub-subscription-id>" az account set --subscription "$SUBSCRIPTION" echo "=== Landing Zone Cost Review — $(date +%Y-%m) ===" echo "--- Azure Advisor cost recommendations ---" az advisor recommendation list \ --category Cost \ --query "[].{Recommendation:shortDescription.solution, Impact:impact}" \ -o table echo "--- Unattached managed disks ---" az disk list \ --query "[?managedBy==null].{Name:name, SizeGB:diskSizeGb, Sku:sku.name}" \ -o table echo "--- Idle public IPs ---" az network public-ip list \ --query "[?ipConfiguration==null].{Name:name, IP:ipAddress, Sku:sku.name}" \ -o table Run it monthly. Share the output with finance. Cost awareness built on data, not guesswork, is what survives the next budget cycle.For teams automating this into a FinOps dashboard, the same query works from Python via the Cost Management REST API: import requests from azure.identity import DefaultAzureCredential credential = DefaultAzureCredential() token = credential.get_token("https://management.azure.com/.default").token body = { "type": "ActualCost", "timeframe": "MonthToDate", "dataset": { "granularity": "None", "aggregation": {"totalCost": {"name": "Cost", "function": "Sum"}}, "grouping": [{"type": "Dimension", "name": "ResourceGroup"}], }, } r = requests.post( "https://management.azure.com" "/subscriptions/<sub-id>/providers/Microsoft.CostManagement/query" "?api-version=2023-11-01", headers={"Authorization": f"Bearer {token}"}, json=body, timeout=30, ) for row in r.json()["properties"]["rows"]: print(f"{row[1]:<40} USD {row[0]:,.2f}") Pitfalls I Keep SeeingTreating Firewall Basic as an enterprise option. 250 Mbps is its ceiling. For a landing zone hub, the realistic choice is Standard vs. Premium, decided by whether you need IDPS/TLS inspection — not by defaulting to Premium.Deploying non-AZ VPN gateway SKUs in 2026. You can't create new ones, and existing ones are scheduled for deprecation after September 2026. If you inherited them, upgrade to the matching AZ SKU now.Buying Log Analytics commitment tiers before pruning ingestion. The 100 GB/day tier (USD 229.32/day in Malaysia West) only beats PAYG above roughly 85–90 GB/day. Below that, reduce ingestion first; then commit.Session recording surprise on Bastion. Teams downsize expecting to keep session recording — it's a Premium SKU feature. Map compliance needs to the SKU before you save a few hundred ringgit.One-off optimization. SKU choices drift. Without a monthly review cadence and policy guardrails, the hub quietly returns to Premium-everything within two quarters.Key TakeawaysLanding zone defaults are safe, not optimal. The CAF pattern over-provisions hub services by design; right-sizing is a deliberate review activity. In the worked scenario the hub drops roughly 42% — most of it from pruning Log Analytics ingestion, the rest from SKU choices.The firewall SKU decision is the biggest SKU lever — but know the real numbers. Basic caps at 250 Mbps, Standard autoscales to 30 Gbps, Premium reaches 100 Gbps with IDPS/TLS inspection. Most mid-size hubs belong on Standard.VPN gateway consolidation is now a compliance item. Non-AZ VpnGw1–5 SKUs can't be newly created and are scheduled for deprecation after September 2026. Move to VpnGw1AZ–5AZ, and size by measured bandwidth — a typical 200–300 Mbps office link needs VpnGw1AZ (USD 0.21/hr in Malaysia West), not VpnGw2AZ (USD 0.54/hr).Log Analytics savings come from pruning first, committing second. The 100 GB/day commitment tier beats PAYG only above ~85 GB/day; daily quota caps prevent runaway months either way.Make it governance, not a project. Azure Policy SKU caps plus a monthly Advisor-and-waste review keep the savings permanent.Law Wen Feng is a Principal Solution Architect based in Malaysia, specializing in Azure cloud architecture, landing zones, and enterprise infrastructure. Follow his work at wenfeng.my/. Related Reading Multi-Agent AI Systems in 2026 — Architecture Patterns (MCP, A2A, Swarm) for Pro Bicep vs Terraform in 2026 — The Enterprise Verdict for Azure-First Teams &#8211 Azure Immutable Backups: Building Ransomware-Proof Recovery with WORM Vaults and --- ## LLM Inference Optimization 2026: The Shift from Software Tweaks to Hardware-Software Co-Design URL: https://wenfeng.my/llm-inference-optimization-2026-software-hardware-co-design/ Author: admin Published: 2026-08-18 23:30:19 Modified: 2026-10-04 15:31:16 If you've been building production LLM systems in the last two years, you know the pain. A model works beautifully in the demo, the stakeholders sign off, and then reality hits: inference costs are astronomical, latency numbers are embarrassing, and your GPU budget just became your CFO's favourite argument against AI adoption.For most of 2023 through 2025, the industry's answer was software-level optimization — quantize the model, batch smarter, optimize the attention kernel, and pray the math works out. And to be fair, it worked remarkably well for a while. But we've hit a ceiling. The marginal returns on software-only tuning are shrinking, and the conversation in 2026 has fundamentally shifted. The new frontier is hardware-software co-design — building silicon and software together, from the ground up, to squeeze every last cycle out of inference workloads.Let me walk you through this evolution, what it means for enterprise teams in Southeast Asia (particularly those betting on Azure), and how to think about your next infrastructure decision.The Software-First Era: What We Got RightLet's give credit where it's due. The software optimization toolkit that emerged between 2023 and 2025 was transformative. Before diving into the new paradigm, it's worth understanding what got us here.Quantization: Making Models Smaller Without Breaking ThemThe journey from FP32 to FP16 was table stakes. The real breakthrough was pushing further — INT8, INT4, and nuanced approaches like GPTQ and AWQ that preserved model quality while slashing memory requirements. By 2025, FP8 had become the default serving precision for many production deployments on NVIDIA hardware, roughly halving memory bandwidth pressure versus FP16 with minimal quality degradation on most workloads.What made quantization powerful wasn't just the memory savings. It was the secondary effect: smaller models fit on less expensive hardware, which meant you could serve more requests per dollar. For enterprise teams watching their Azure spend, this was existential.Batching Strategies: Squeezing More From the Same GPUsDynamic batching, continuous batching (popularized by vLLM's PagedAttention), and smarter request scheduling were the unsung heroes. The idea that you could manage KV cache memory like virtual memory pages — allocating GPU memory to whichever requests needed it most — was a genuine paradigm shift in throughput.I remember working with a financial services client in Kuala Lumpur who went from needing 8 A100 GPUs to serving the same workload on 4, just by moving from static batching to continuous batching with PagedAttention. That's a 50% cost reduction with zero model changes.FlashAttention: When the Kernel MattersThen came FlashAttention and its successors. The insight was elegant: the bottleneck in transformer inference isn't compute — it's memory bandwidth. By restructuring how attention is computed to minimize HBM (High Bandwidth Memory) accesses, FlashAttention cut attention's memory footprint from quadratic to linear in sequence length and ran up to 3x faster than standard PyTorch attention, with each subsequent generation pushing further. Kernel-level engineering was clearly still able to unlock massive gains.Speculative Decoding: Trading Compute for LatencySpeculative decoding offered another clever trick: use a smaller, faster model to draft tokens, then verify them in parallel with the larger model. Lower latency without sacrificing output quality. For interactive applications — chatbots, coding assistants, real-time translation — this was a game-changer.These techniques collectively made LLM inference commercially viable. But here's the uncomfortable truth: we've been optimizing the software stack on hardware that was designed for general-purpose GPU computing, not for LLM inference specifically. And that mismatch is now the binding constraint.Why Software-Only Optimization Has Hit Its CeilingThere are three structural limits that pure software approaches can't overcome.1. The Memory Bandwidth WallLLM token generation is fundamentally memory-bandwidth-bound, not compute-bound. When you're generating tokens one at a time, the GPU spends most of its time waiting for weights to arrive from HBM. Software can reduce how much data needs to move — through caching, compression, and scheduling — but it can't change the physics of the memory bus. On an H100, you get 3.35 TB/s of HBM3 bandwidth. On a B200, you get 8 TB/s of HBM3e. That's a hardware improvement, not a software one.2. The Kernel Fragmentation ProblemEvery software optimization introduces complexity. FlashAttention has its kernels. Mixture-of-experts routing has another. Sliding-window attention, state-space models, multi-head latent attention — each new architecture adds specialized kernels, each with its own launch overhead and memory access pattern. The software stack becomes a Rube Goldberg machine of workarounds, and every model family needs its own tuning effort.3. The Diminishing Returns CurveQuantization from FP32 to INT4 gave us roughly 8x compression. Going below INT4 buys a little more at a devastating quality cost. Batch size and scheduling optimizations follow a similar curve — each increment helps less than the last. We've picked the low-hanging fruit.The path forward requires rethinking the problem at a deeper level: what if the hardware itself was designed to run LLM inference natively?Enter Hardware-Software Co-DesignThis is where the industry is heading in 2026, and it's happening faster than most enterprise teams realize.Hardware-software co-design means jointly optimizing the silicon architecture, the compiler stack, and the runtime as a single system for LLM workloads. It's not about faster GPUs. It's about purpose-built systems where the memory hierarchy, the interconnect, and the instruction set are all designed with transformer inference in mind.Custom Silicon: Purpose-Built for TransformersThe most visible examples:NVIDIA GB200 NVL72: A rack-scale system with 36 Grace CPUs and 72 Blackwell GPUs connected over NVLink. NVIDIA claims up to 30x inference performance for real-time LLM inference on large mixture-of-experts models versus an 8-GPU H100 HGX system. Read that number carefully: the rack contains nine times the GPUs, so the per-GPU gain is far more modest than the headline. The real architectural shift is the NVLink domain letting 72 GPUs behave as one inference unit, plus native FP4 support in the second-generation Transformer Engine.Google TPU v6e (Trillium): Google's sixth-generation TPU delivers roughly 4x peak compute performance versus TPU v5e, more than doubles HBM capacity and bandwidth, and improves energy efficiency by about 67% — a combination that matters enormously for inference economics at scale.Microsoft Maia 100: Microsoft's first custom AI accelerator, built on TSMC 5nm with over 100 billion transistors, liquid-cooled, and deployed in Azure datacenters for first-party and OpenAI workloads. The strategic signal matters more than the spec sheet: hyperscalers no longer believe general-purpose GPUs are the end of their inference cost curve.Cerebras CS-3: The wafer-scale design keeps model weights on-chip and sidesteps the memory bottleneck entirely. Cerebras's hosted inference service has published token generation rates on Llama models that are an order of magnitude faster than conventional GPU serving for latency-critical workloads — a radical architectural bet that is starting to pay off in specific niches.Fused Kernels: Combining Operations at the Silicon LevelOne of the most impactful co-design patterns is kernel fusion — but not the software-level fusion we've been doing with tools like Triton. Hardware-level fusion means the silicon executes multi-operation sequences (attention + activation + projection) without round-tripping to memory between steps.On Blackwell, the second-generation Transformer Engine dynamically manages precision — including FP4 — across operations, and combined with CUDA's fused kernel libraries, much of the attention block's forward pass executes without writing intermediate results back to HBM. Pure software stacks on older hardware cannot replicate this, because it requires the hardware to manage precision and data movement jointly.Compiler-Driven OptimizationThe other half of co-design is the compiler. Tools like CUTLASS and Triton showed that kernel authoring doesn't have to be a manual, artisanal process. The new generation of AI compilers takes a model specification and generates kernels optimized for a specific hardware configuration, reasoning about memory layout, tensor core utilization, and operation fusion simultaneously.This matters for enterprise teams because it means you don't need a bench of CUDA kernel experts to get good performance. Co-design pushes the complexity into the toolchain, making it accessible to ordinary application developers.What This Means for Azure-First Enterprise TeamsSo what does this mean if you're an Azure-first enterprise in Southeast Asia deploying LLM workloads today? Let me break it down practically.The Infrastructure Decision Has Never Been More ComplexIn 2024, the calculus was straightforward: rent GPU VMs on Azure, optimize your model, done. In 2026, you're choosing between four fundamentally different deployment models: OptionControlOptimization ceilingOps burdenBest for Azure AI Foundry / Azure OpenAI (managed)LowPlatform-definedMinimalMost enterprises Self-managed vLLM/SGLang on ND-series VMsFullLimited by your teamHighTeams with systems expertise Serverless endpoints (pay-per-token)LowPlatform-definedMinimalSpiky or low-volume workloads Edge/hybrid (small local model + cloud offload)MediumSplit across tiersMediumLatency-critical first-pass responses The right choice depends on your workload characteristics, latency requirements, and team capabilities — not just the sticker price.Cost Optimization Has a New DimensionHardware-software co-design changes the cost equation fundamentally. When a new generation delivers materially more tokens per second per dollar, that translates directly to fewer accelerators for the same workload. For a typical enterprise deployment serving high request volumes at a P95 latency target, the gap between an H100-era stack and a co-optimized Blackwell-era stack can be significant in infrastructure cost — roughly 40-60% in our illustrative modeling, but validate against your own workload before planning any budget.But here's the catch: extracting those gains requires the entire stack to be co-optimized. Running an H100-era software stack on Blackwell hardware won't get you the generational improvement. You need to:Use hardware-native inference frameworks (recent vLLM or SGLang builds with Blackwell backends)Enable FP4 or FP8 precision where your quality evaluations allowLeverage NVLink-topology-aware scheduling for multi-GPU configurationsUse the Transformer Engine's dynamic precision managementThis is the "co-design" part. The hardware and software have to speak the same language.The Skills Gap Is RealI'll be honest with you: finding engineers who understand both the hardware architecture and the software stack is difficult. In Southeast Asia, this challenge is amplified. The talent pool is growing — universities in Singapore, Malaysia, and Thailand are ramping up AI systems programs — but the intersection of systems engineering and AI is still rare.My advice: don't try to build this expertise entirely in-house. Lean on managed services where the co-design has already been done for you (Azure AI Foundry is a strong option). Invest your engineering talent in the application layer — retrieval pipelines, evaluation frameworks, agent orchestration — where you can actually differentiate. Let the platform handle the hardware-software integration.Regional Considerations for Southeast AsiaA few things I've observed working with teams across the region:Data sovereignty matters. Many Southeast Asian enterprises have data residency requirements. Azure's Southeast Asia (Singapore), East Asia (Hong Kong), and Malaysia West regions are where to check accelerator availability first. Plan your inference architecture around regional capacity, not global announcements.Network latency is a real constraint. A 10-20ms round trip between Kuala Lumpur and Singapore adds up fast when you're streaming tokens. Consider whether a small local model for first-pass responses plus cloud offload for complex reasoning is the right architecture.Cost sensitivity is higher. Enterprise budgets in Southeast Asia tend to be tighter than in the US or Europe. The efficiency gains from co-optimized stacks aren't nice-to-haves — they're what make AI deployment financially viable.A Decision Framework for 2026If you're evaluating inference infrastructure this year, here's the framework I use with clients.Step 1: Profile your workload. Latency requirements (sub-100ms real-time, 100-500ms interactive, or batch-tolerant)? Throughput needs and burstiness? Model complexity — single dense model or mixture-of-experts? What precision is acceptable?Step 2: Match to the right optimization layer. If latency is critical, co-optimized stacks or managed endpoints give the best tail numbers. If throughput per dollar matters most, managed services with automatic batching and scaling usually win. If you need maximum control, self-managed vLLM/SGLang on GPU VMs gives full visibility at a higher engineering cost.Step 3: Plan for the transition. Hardware generations now ship roughly yearly. Design your serving layer to be hardware-agnostic behind an API, keep reservation terms short enough to ride the refresh cycle, and budget for a software stack upgrade whenever you migrate hardware.Step 4: Measure what matters. Track cost per million tokens end-to-end, time to first token (TTFT), inter-token latency (ITL), throughput at your P95 latency target, and actual accelerator utilization.Practical: Deploying and Benchmarking on AzureTheory is cheap; here's what the work actually looks like. These examples reflect the patterns I use with clients.Check GPU capacity before you commitQuota, not money, is usually the first bottleneck. Check what's actually available in your region: # What ND-series (GPU) capacity exists in Southeast Asia? az vm list-skus --location southeastasia \ --resource-type compute --all \ --query "[?contains(name, 'ND')].{Name:name, vCPUs:capabilities.vCPUs}" \ --output table # Request quota early — GPU quota approvals are not instant az vm list-usage --location southeastasia \ --query "[?contains(name.value, 'NDS')]" --output table Managed route: serverless model endpoint (Azure AI Foundry)For most enterprises, this is where you should start — Microsoft owns the hardware-software co-design problem: # Deploy a serverless endpoint for an open model az ml serverless-endpoint create \ --name ep-llm-inference \ --resource-group rg-ai-prod \ --workspace-name aif-hub-01 \ --model-id "Llama-3.3-70B-Instruct" # Query it — same OpenAI-compatible surface your app already uses az ml serverless-endpoint invoke --name ep-llm-inference \ --resource-group rg-ai-prod --workspace-name aif-hub-01 \ --input-data '{"messages":[{"role":"user","content":"Explain KV cache"}]}' Self-managed route: infrastructure as codeWhen you need full control of the serving stack, define the accelerator VM in code. Terraform first: resource "azurerm_linux_virtual_machine" "vllm_inference" { name = "vm-nd-gpu-01" resource_group_name = azurerm_resource_group.ai.name location = "southeastasia" size = "Standard_ND96isr_H200_v5" admin_username = "azureuser" network_interface_ids = [azurerm_network_interface.gpu_nic.id] admin_ssh_key { username = "azureuser" public_key = file("~/.ssh/id_rsa.pub") } os_disk { caching = "ReadWrite" storage_account_type = "Premium_LRS" disk_size_gb = 1024 # models are large; don't undersize this } source_image_reference { publisher = "microsoft-dsvm" offer = "ubuntu-hpc" sku = "2204" version = "latest" } } And the equivalent Bicep for the managed-services side — an AI Services account locked down the way a security review will demand: param location string = resourceGroup().location resource aiServices 'Microsoft.CognitiveServices/accounts@2024-10-01' = { name: 'ais-inference-001' location: location kind: 'AIServices' sku: { name: 'S0' } properties: { customSubDomainName: 'ais-inference-001' disableLocalAuth: true // Entra ID only — no shared keys publicNetworkAccess: 'Disabled' // private endpoint only } } Benchmark what actually matters: TTFT and ITLVendor demos measure cherry-picked prompts. Measure your own workload with a streaming client: import time from openai import OpenAI client = OpenAI( base_url="https://ep-llm-inference.southeastasia.inference.ai.azure.com/v1", api_key="<entra-token-or-key>", # never hardcode in real life ) def measure(prompt: str, model: str) -> dict: t0 = time.perf_counter() ttft = None tokens = 0 stream = client.chat.completions.create( model=model, messages=[{"role": "user", "content": prompt}], stream=True, ) for chunk in stream: if chunk.choices and chunk.choices[0].delta.content: if ttft is None: ttft = time.perf_counter() - t0 tokens += 1 total = time.perf_counter() - t0 itl = (total - ttft) / max(tokens - 1, 1) return { "ttft_ms": round(ttft * 1000, 1), # latency users *feel* "itl_ms": round(itl * 1000, 1), # reading-speed constraint "tokens": tokens, } Run this across your real prompt distribution — short and long contexts, batch and interactive — before any hardware decision.Serve with a modern stack and watch utilizationOn the self-managed VM, launch vLLM with the features that matter, then verify the silicon is actually working: # Continuous batching + prefix caching + FP8 — the 2026 baseline vllm serve meta-llama/Llama-3.3-70B-Instruct \ --tensor-parallel-size 8 \ --enable-prefix-caching \ --max-num-seqs 256 \ --quantization fp8 # Is the GPU saturated, or is your pipeline starving it? nvidia-smi dmon -s u -c 60 # 60 samples of SM + memory utilization If utilization sits below ~50% under load, your bottleneck is upstream — batching config, request size, or CPU-side preprocessing — not the accelerator.Pitfalls I See RepeatedlySix mistakes show up in nearly every inference engagement I touch.New silicon, old stack. Teams rent Blackwell-class hardware and deploy an H100-era container image. They pay rack prices for Hopper-class tokens per second. Pin your framework versions to your hardware generation, and re-benchmark after every upgrade.Optimizing the wrong latency metric. TTFT and ITL have different drivers: prefill optimization helps TTFT, decode-side tuning helps ITL. A chat UI cares about ITL; a RAG pipeline cares about total time. Decide which metric your users actually experience before tuning anything.Quantizing without an evaluation gate. FP4 and INT4 regressions are task-dependent — fine for summarization, sometimes catastrophic for structured extraction. Run your evaluation suite before and after any precision change, not just perplexity.Ignoring KV cache in capacity planning. Long-context workloads eat HBM through the KV cache, not just weights. A model that "fits" on paper can still collapse under concurrent long-context load. Tune --max-num-seqs against your real context-length distribution.Vendor peak claims treated as capacity planning numbers. "Up to 30x" headline figures assume large MoE models, ideal batch sizes, and best-case precisions. Your workload is not the demo workload. Budget on your own benchmarks, with margin.Reservations that outlive the hardware generation. GPU generations now move roughly yearly. A 3-year reservation on last generation's accelerator can strand you paying above-market rates for below-market performance. Match commitment length to refresh cycles.Looking AheadThe co-design trend will only accelerate. NVIDIA's Rubin platform is on the roadmap for late 2026 with Rubin Ultra following in 2027, and competitors are pushing the same thesis from different angles — AMD's Instinct MI400 series targets 2026 availability, and custom silicon players like Cerebras, SambaNova, and Groq each attack a different corner of the latency-throughput-cost triangle.For enterprise teams, the message is clear: the era of treating inference hardware as a commodity black box is ending. Understanding the hardware-software interaction — even at a high level — is becoming essential for sound infrastructure decisions. You don't need to write CUDA kernels, but you do need to understand why FP4 on Blackwell behaves differently than FP4 on Hopper, and what that means for your deployment strategy.The teams that thrive won't be the ones with the biggest GPU budgets. They'll be the ones that align their workload characteristics with the right hardware-software stack. And in 2026, that alignment requires a fundamentally different approach than the one we used two years ago.Key TakeawaysSoftware-only optimization has reached diminishing returns. Quantization, batching, and kernel tuning delivered massive gains in 2023-2025, but the memory bandwidth wall and kernel fragmentation mean further gains require hardware-level change.Hardware-software co-design is the new paradigm. Purpose-built systems (GB200 NVL72, TPU v6e, Maia 100), fused kernels, and compiler-driven optimization deliver generational improvements that pure software approaches cannot match — but read vendor headline numbers critically.The infrastructure decision is now a stack decision. It's not which GPU to rent — it's whether the entire stack (framework, runtime, precision, interconnect) is co-optimized for that hardware. Managed services like Azure AI Foundry absorb that complexity for you.For Southeast Asian enterprises, co-optimized stacks are a cost imperative. Tighter budgets, data sovereignty requirements, and regional latency make efficiency gains necessary, not optional.Profile your workload before chasing hardware. Measure TTFT, ITL, cost per million tokens, and utilization on your real traffic. Start with the workload profile, then match to the stack — never the other way around.Have questions about optimizing your inference stack? Reach out via wenfeng.my/ — I'd love to hear what your team is building. --- ## Cloud Migration Playbook for Malaysian SMEs — From Assessment to Landing Zone in 90 Days URL: https://wenfeng.my/cloud-migration-playbook-for-malaysian-smes-from-assessment-to-landing-zone-in-90-days/ Author: admin Published: 2026-08-18 06:30:10 Modified: 2026-08-18 06:30:10 Something shifted in the Malaysian cloud landscape in May 2025. Microsoft cut the ribbon on Azure Malaysia West — the country's first Azure region, AI-ready hyperscale infrastructure with three availability zones, located in Greater Kuala Lumpur. It removed the single biggest excuse for keeping workloads on-premise: data residency. A second region, Southeast Asia 3, is already planned for Johor Bahru. Suddenly, the question from Malaysian boards wasn't "where will my data live?" but "why haven't we started?"A year later, I still see the same pattern across the SMEs I work with. Everyone agrees cloud migration makes sense. The budget is approved. But teams stay stuck in an endless assessment loop — collecting inventory spreadsheets, running Azure Migrate assessments that sit in dashboards nobody opens, debating migration strategy in meetings that produce no action. Meanwhile, system integrators are building dedicated Azure migration practices, and Malaysia West is filling up with enterprise workloads from organisations that figured out how to execute.The gap isn't knowledge. It's execution. So here's the playbook I've refined across a dozen Malaysian SME migrations: a concrete 90-day sprint from "we should move to Azure" to "we have a production landing zone with workloads running." Real commands, real timelines, and realistic costs in MYR.Why Malaysian SMEs Are Moving NowMalaysia's cloud adoption accelerated sharply after Malaysia West went live. The drivers:PDPA compliance — the Personal Data Protection (Amendment) Act 2024 came into force in phases from 1 January 2025, with data breach notification and DPO appointment obligations effective 1 June 2025. Data sovereignty matters, and Malaysia West provides it.Government cloud direction — MyDigital and the public sector's cloud-first push are pulling GLCs and their supply chains toward cloud.Talent availability — Azure-skilled architects are no longer unicorns in KL. Cognizant, HCL, and local SIs have stood up migration practices.Cost pressure — MYR weakness makes hardware procurement expensive. OpEx cloud models look genuinely attractive when your CapEx cycle means 18 months of PO approvals.The technology, the talent, and the business case have aligned. But only if you execute within a reasonable timeframe. Ninety days is enough — if you follow a structured playbook.Phase 1: Assessment & Discovery (Days 1–21)Week 1: Inventory with Azure MigrateStop with the spreadsheet. Deploy the Azure Migrate appliance on your VMware or Hyper-V host — Microsoft provides a scripted deployment — and let discovery populate your project automatically. Once discovery is running, verify what the appliance is reporting: # Verify servers discovered by your Azure Migrate project # (az migrate is a CLI extension; it installs automatically on first use) az migrate get-discovered-server \ --project-name migrate-my \ --resource-group rg-migration \ --source-machine-type HyperV \ --output table Assessment itself is run from the Azure Migrate hub in the portal — discovery populates the inventory, and you create an assessment against a server group right there. If you want repeatable, versioned assessments, drive the same engine programmatically through the Azure Migrate AssessmentProjects REST API (the Az.Migrate PowerShell module covers replication and migration; assessments are exposed via REST): # Create an assessment via the Azure Migrate REST API (api-version 2023-03-15) az rest --method put \ --url "/subscriptions/{sub-id}/resourceGroups/rg-migration/providers/Microsoft.Migrate/assessmentProjects/migrate-my/groups/production-workloads/assessments/day21-prod-assessment?api-version=2023-03-15" \ --body '{ "properties": { "azureLocation": "malaysiawest", "sizingCriterion": "PerformanceBased", "reservedInstance": "RI3Year" } }' # Poll until assessment status is Completed, then read assessed machines az rest --method get \ --url "/subscriptions/{sub-id}/resourceGroups/rg-migration/providers/Microsoft.Migrate/assessmentProjects/migrate-my/groups/production-workloads/assessments/day21-prod-assessment?api-version=2023-03-15" Set the assessment target to Malaysia West and use performance-based sizing. In the assessment properties, configure reserved-instance pricing — for steady-state production VMs this is the single biggest lever you have. Reserved Instances save up to ~40% on one-year commitments and up to 65% on three-year commitments versus pay-as-you-go. Run the assessment numbers yourself; on a single mid-size VM the annual saving can easily exceed RM15,000. Real money for an SME.Weeks 2–3: Cost Modelling in MYRAzure Migrate gives you a headline estimate, but you need to translate it into business language. I pull live pricing from the Azure Retail Prices REST API and build my own model — it's scriptable and auditable: import requests url = "https://prices.azure.com/api/retail/prices" flt = ("armRegionName eq 'malaysiawest' and " "armSkuName eq 'Standard_D4s_v5' and " "serviceName eq 'Virtual Machines'") resp = requests.get(url, params={"$filter": flt, "currencyCode": "USD"}).json() for item in resp["Items"][:6]: print(item["type"], item["retailPrice"], item.get("reservationTerm", ""), item["unitOfMeasure"]) This returns consumption, DevTest, and reserved pricing side by side — exactly what you need for a board comparison. Typical monthly run rates I've seen for Malaysian SME workloads in Malaysia West: Workload ProfileMonthly MYR (illustrative)Annual MYR Small (5–10 VMs, basic apps)RM4,000–8,000RM48,000–96,000 Medium (15–30 VMs, SQL, web apps)RM15,000–35,000RM180,000–420,000 Large (50+ VMs, microservices)RM50,000–120,000RM600,000–1,440,000 Compare these against your current on-premise TCO — power, cooling, DC rental, hardware refresh cycles, admin staff. For most SMEs, cloud lands 20–40% cheaper over three years once you count avoided hardware refreshes. Treat the table as illustrative; your assessment output is the number that matters.Day 21 DeliverableA completed Azure Migrate assessment with:Performance-based sizing recommendations pinned to Malaysia WestReserved-instance cost estimates, translated into MYRA migration strategy per workload: rehost, replatform, or refactorPhase 2: Landing Zone Build (Days 22–60)This is where most SMEs stall. They finish assessment, then spend months debating architecture. Skip the analysis paralysis — use the Azure landing zone (ALZ) IaC Accelerator, the successor to the old CAF Enterprise-Scale Terraform repo (deprecated in January 2025 and in extended support until the repository is archived on 1 August 2026). The accelerator supports both Bicep and Terraform and deploys governance, networking, and monitoring as one opinionated package.Weeks 4–5: Deploy the Platform FoundationFor most Malaysian SMEs I recommend the small-enterprise topology: a platform subscription with hub networking, plus a workload subscription with spoke VNets. After the accelerator bootstrap, workload spokes are plain Bicep — easy to own and extend: // workload-spoke.bicep — minimal production spoke param location string = resourceGroup().location param addressPrefix string = '10.10.0.0/20' resource vnet 'Microsoft.Network/virtualNetworks@2024-03-01' = { name: 'vnet-prod-myw' location: location properties: { addressSpace: { addressPrefixes: [addressPrefix] } subnets: [ { name: 'snet-workload' properties: { addressPrefix: addressPrefix privateEndpointNetworkPolicies: 'Enabled' } } ] } } output vnetId string = vnet.id # Resource-group-scoped deployment — location is inherited from the resource group az deployment group create \ --resource-group rg-prod-network-myw \ --template-file workload-spoke.bicep Weeks 6–7: Hardening for Malaysian RequirementsOnce the foundation is deployed, harden it. Enable Defender for Cloud plans for the workloads you actually run (you pay per plan, so scope it deliberately): # Enable Defender plans for your core workload types az security pricing create --name VirtualMachines --tier Standard az security pricing create --name SqlServersVirtualMachines --tier Standard az security pricing create --name StorageAccounts --tier Standard # Route subscription activity logs into Log Analytics for PDPA audit trail az monitor diagnostic-settings create \ --name "sub-activity-to-law" \ --resource "/subscriptions/{sub-id}" \ --workspace "/subscriptions/{sub-id}/resourceGroups/rg-mgmt/providers/Microsoft.OperationalInsights/workspaces/law-myw-logs" \ --logs '[{"categoryGroup":"audit","enabled":true}]' Then configure data classification. Sensitivity labels (e.g., a "PDPA Personal Data" label) are managed in the Microsoft Purview portal and applied to storage accounts and databases holding personal data — this is portal and policy work, not a CLI flag, so put it on the hardening checklist and assign an owner.Day 60 DeliverableA production landing zone in Malaysia West with:Hub-spoke network topologyMicrosoft Entra ID integration and RBACDefender for Cloud enabled on workload plansActivity and audit logging into Log Analytics for PDPA evidenceTagging and policy guardrails enforced via Azure PolicyPhase 3: Migration Execution (Days 61–90)Data Migration PatternsThree patterns cover most Malaysian SME estates.Pattern 1: Lift-and-shift (VM rehost)The fastest path, using Azure Migrate: Server Migration. Replication is configured in the portal or via the Az.Migrate module: # Start replication for a discovered server (agentless) # MachineId = the discovered machine's resource ID (get it from Get-AzMigrateDiscoveredServer) # Full parameter set: Get-Help New-AzMigrateServerReplication New-AzMigrateServerReplication ` -MachineId "/subscriptions/{sub-id}/resourceGroups/rg-migration/providers/Microsoft.OffAzure/HyperVSites/{site-name}/machines/{machine-id}" ` -LicenseType NoLicenseType ` -TargetResourceGroupId "/subscriptions/{sub-id}/resourceGroups/rg-prod-workloads-myw" ` -TargetNetworkId "/subscriptions/{sub-id}/resourceGroups/rg-prod-workloads-myw/providers/Microsoft.Network/virtualNetworks/vnet-prod-myw" ` -TargetSubnetName "snet-workload" ` -TargetVMName "vm-app01" ` -TargetVMSize "Standard_D4s_v5" ` -DiskType "Standard_LRS" ` -OSDiskID "{os-disk-uuid}" # Monitor until replication is healthy, then schedule cutover Get-AzMigrateServerReplication -MachineName "vm-app01" ` -ProjectName "migrate-my" -ResourceGroupName "rg-migration" Pattern 2: SQL Server to Azure SQLTwo 2026 tooling updates matter here. The classic Azure Database Migration Service SQL scenarios were retired on 15 March 2026, and the Azure SQL migration extension for Azure Data Studio was retired together with Azure Data Studio itself on 28 February 2026 — don't build either into a new migration plan. Microsoft's current guidance, by target:Azure SQL Managed Instance — the guided path is the SQL Server migration experience in Azure Arc: discovery, assessment, and near-zero-downtime migration in a single flow. At scale, automate with the Azure DMS PowerShell cmdlets or Azure CLI.SQL Server on Azure VMs — the Azure DMS portal experience (online or offline, backups from Azure Blob storage or an SMB share), or the SSMS migration component if you can connect to both ends.Azure SQL Database — the DMS portal experience for schema + data (offline), or Striim for online, near-zero-downtime migration.The workflow shape is unchanged: run the assessment (readiness flags plus a right-sized SKU recommendation — Managed Instance for near-parity lift, Azure SQL Database if you can modernise), choose online or offline mode, start the migration, and cut over in your change window.Pattern 3: Web apps to Azure App ServiceFor .NET and Java web applications, replatforming to App Service is often the best value move. Containerise and deploy: # Containerised .NET 8 app in Malaysia West az webapp create \ --resource-group rg-prod-workloads-myw \ --plan plan-apps-myw \ --name app-web-frontend \ --deployment-container-image-name mcr.microsoft.com/dotnet/aspnet:8.0 # Integrate into the landing zone VNet az webapp vnet-integration add \ --resource-group rg-prod-workloads-myw \ --name app-web-frontend \ --vnet vnet-prod-myw \ --subnet snet-workload PDPA Compliance ChecklistBefore cutover, verify:Data classification — personal data is labelled and encrypted at rest (encryption is default on Azure storage and SQL, but verify it hasn't been disabled anywhere).Access logging — access to personal data is logged with retention agreed with your DPO (seven years aligns with Malaysian tax record-keeping practice).Consent and purpose — processing purposes match your PDPA notices.Breach notification — Defender for Cloud alerts feed your incident process; the PDPA requires notification of qualifying breaches within 72 hours.Erasure requests — your data architecture can actually delete data when asked. # Verify encryption is enabled on the storage account holding audit data az storage account show \ --name stauditmyw \ --resource-group rg-mgmt \ --query "encryption.services.blob.enabled" \ --output tsv The Cutover PlaybookRun cutover on days 85–88 with a rollback window to day 90:Day 85 — final data sync (Azure Migrate continuous replication; Azure SQL online mode keeps log shipping warm).Day 86 — DNS switchover for internal applications.Day 87 — external DNS switchover, monitoring on high alert.Days 88–90 — hypercare. Watch dashboards, resolve issues, confirm performance against the baselines you captured during assessment.Common Pitfalls (and How to Avoid Them)Pitfall 1: Underestimating network connectivity. Malaysia West is local, but your users still need adequate bandwidth to it. Budget for ExpressRoute or at minimum a site-to-site VPN during migration and hypercare — get a fresh quote; connectivity pricing moves.Pitfall 2: Ignoring licensing. Azure Hybrid Benefit for Windows Server and SQL Server can cut VM costs dramatically if you already own qualifying licenses. Check entitlements before paying for license-included pricing.Pitfall 3: Over-engineering the landing zone. The small-enterprise topology is enough for most Malaysian SMEs. Don't deploy a full multi-region enterprise-scale platform with Virtual WAN when a single hub-spoke in Malaysia West serves you fine. You can grow into complexity; you can't shrink out of it easily.Pitfall 4: Skipping the PDPA conversation. I've seen migrations stall at the last minute because Legal wasn't engaged early. Bring your DPO or compliance officer to the Week 1 kickoff. PDPA compliance is a design constraint, not an afterthought.Pitfall 5: No rollback plan. Keep the on-premise environment warm until you've validated production performance in Azure. Set DNS TTL low (300 seconds) during cutover so you can flip back fast if something misbehaves.The Cost RealityLet me be direct about numbers for a typical Malaysian SME — 20 servers, a SQL database estate, and a few web applications:Assessment phase — RM0. Azure Migrate's discovery and assessment tooling is free.Landing zone run cost — roughly RM800–1,500/month for the platform subscription (management, monitoring, networking).Migration tooling — minimal; the primary costs are replication storage and any temporary bandwidth.Post-migration run rate — RM15,000–25,000/month with Reserved Instances, for the workload profile above.Professional services — if you engage an SI for the 90-day engagement, budget in the region of RM80,000–150,000 depending on scope.Compare that against your current on-premise TCO. If you're due a hardware refresh within 18 months, the business case practically writes itself.Key TakeawaysAssessment is a solved problem — stop dwelling there. Azure Migrate is free, and Malaysia West is live with three availability zones in Greater Kuala Lumpur. Run discovery in Week 1 and commit to a migration plan by Day 21. Endless assessment is procrastination wearing a spreadsheet.Use the ALZ accelerator, not custom architecture. The Azure landing zone IaC Accelerator (Bicep or Terraform) gives you enterprise-grade governance out of the box, and the classic CAF Terraform tooling is being retired — don't build on it. Your landing zone should be production-ready by Day 60.PDPA compliance is a design constraint, not a checklist item. Classify personal data, log access with agreed retention, verify encryption, and make sure your breach workflow can hit the 72-hour window. Build it into the landing zone from Day 1, not Day 89.Budget honestly, in MYR. Pull live pricing from the Retail Prices API, model reserved-instance terms, and compare against your real on-premise TCO — including the hardware refresh you're about to buy anyway.The 90-day timeline is aggressive but real. The trick is parallelism: the landing zone build doesn't need your server inventory, so start it while assessment finishes. Lift-and-shift 80% of workloads, replatform the 20% where the value justifies it.Malaysia West removed the data sovereignty barrier. The question is no longer whether to migrate — it's whether you'll execute before your competitors do. --- ## Azure Site Recovery 2026: NVMe VMs, Shared Disks, and Performance Plus Replication URL: https://wenfeng.my/azure-site-recovery-2026-nvme-shared-disks/ Author: admin Published: 2026-08-17 23:30:31 Modified: 2026-10-04 15:27:37 Every DR assessment I run starts the same way: map the production estate, then check what Azure Site Recovery (ASR) can actually protect. For years, that second step produced an uncomfortable list of exceptions. Modern VMs with NVMe disk controllers? Not supported. Windows Server Failover Clusters on shared disks? Not supported. Disks with Performance Plus enabled? Partial support with caveats.Those gaps have now closed. The current ASR support matrix lists NVMe storage interfaces as supported for Azure-to-Azure replication, shared disks as supported via Windows Server Failover Clustering (WSFC), and Performance Plus disks as supported. If your DR strategy was constrained by any of these, it is worth re-evaluating.This article covers what is supported today, what is still excluded, and how to configure each of the three — with the exact PowerShell and CLI commands.Why These Gaps MatteredThese were not edge cases. Each gap blocked a common production pattern:NVMe VMs are the modern default. The v6-series general-purpose and memory-optimized families (Da/Ea/Fa v6), plus storage-optimized Ebsv5/Ebdsv5 SKUs, use NVMe disk interfaces. As fleets upgraded from older generations, an increasing share of VMs quietly fell out of ASR scope.Clustered workloads need shared storage. SQL Server Failover Cluster Instances (FCI), SAP ASCS/SCS, and scale-out file servers all depend on shared disks. Without shared-disk replication, DR for these meant maintaining a parallel cluster in the recovery region by hand.Performance Plus is increasingly common. Enabling Performance Plus on a disk raises its IOPS and throughput limits without paying for a larger tier — attractive for database disks. But if the performance configuration did not survive replication, your failover environment ran slower than production.The result was DR coverage that looked complete on a slide but had holes exactly where the most important workloads lived.What Is Supported Now1. NVMe Disk Controller SupportPer the current Azure-to-Azure support matrix, NVMe is supported for Azure-to-Azure replication:Windows: supported on Gen2 VMs that use the NVMe interface — Da/Ea/Fa v6-series, Ddsv6, Edsv6, Ebsv5/Ebdsv5, and others.Linux: supported for RHEL 9, SLES 15, and Ubuntu 24 on the same Gen2 NVMe families — currently in preview.What is still excluded:Ephemeral OS disks and local NVMe disks are not replicated (they are local-only storage by design).Mixed-controller VMs (SCSI + NVMe) are not supported — this includes VM SKUs such as Lsv3.Practically: if your fleet standardizes on v6-series compute with managed NVMe data disks, you can now enable ASR without per-VM exceptions. Linux NVMe support being in preview means it is usable for evaluation and non-critical workloads, but check the preview status before relying on it for production RPO commitments.Enabling replication for an NVMe VM follows the standard Azure-to-Azure flow — nothing NVMe-specific to configure. The CLI extension command looks like this (fabric, container, and mapping created first): az site-recovery protected-item create -g rg-dr --fabric-name fabric-source \ -n rpi-vm-nvme --protection-container container-source \ --vault-name asr-vault --policy-id $POLICY_ID \ --provider-details '{a2a:{fabric-object-id:'$VM_ID',vm-managed-disks:[{disk-id:'$OS_DISK_ID',primary-staging-azure-storage-account-id:'$CACHE_SA_ID',recovery-resource-group-id:'$TARGET_RG_ID}],recovery-azure-network-id:'$RECOVERY_VNET_ID',recovery-container-id:'$RECOVERY_CONTAINER_ID',recovery-resource-group-id:'$TARGET_RG_ID',recovery-subnet-name:default}}' Note the command name is az site-recovery protected-item create (the site-recovery CLI extension), not az site-recovery replication-protected-item create — that older name does not exist and appears in a lot of AI-generated guides.2. Shared Disks via Windows Server Failover ClusteringASR can now replicate Azure shared disks, with one hard constraint: the sharing must be orchestrated by WSFC on Windows Server 2016 or later, in an active-passive configuration. Linux shared-disk configurations are not supported.The support matrix for shared disks: DimensionSupported PlatformWindows VMs only Cluster solutionWSFC (Windows Server 2016+) Clustering configurationActive-Passive Shared disk typesStandard SSD, Premium SSD, Premium SSD v2, Ultra Disk partitioningBasic Explicitly unsupported:Active-Active clustersProtecting multiple clusters as a single groupMixing clustered and non-clustered VMs in one protection groupNon-clustered distributed appliances that don't use WSFCVMs in a cluster attached to different shared disk sets — every protected VM must share the same set of disksThe key operational behavior: ASR treats the cluster as a single unit. Recovery points are cluster-consistent across all VMs and all disks (including the shared disk), failover is one action, and reprotect/failback is one action. You don't orchestrate per-node recovery yourself.Two more constraints to plan around:All-or-nothing enablement: you can only enable replication successfully when all VMs attached to a shared disk are selected. You can exclude the shared disk itself (with a warning), but you cannot protect some nodes and not others.No domain controllers: ASR shared-disk support does not cover Active Directory VMs — keep your DCs out of the protection group.Crash-consistent recovery points only: ASR for shared disks supports crash-consistent recovery points — app-consistent snapshots are not available for shared-disk clusters, so plan the SQL/SAP recovery flow around crash-consistent points (SQL FCI recovers cleanly from them via database recovery at startup).3. Performance Plus Disk SupportPerformance Plus raises the IOPS and throughput limits of a managed disk to the expanded maximums for its size — with no extra charge for the capability itself. ASR's support matrix confirms Performance Plus disks are supported for replication (Standard HDD, Standard SSD, and Premium SSD disks, using premium storage accounts during replication).Two facts about Performance Plus that shape your DR design:Eligibility: Standard HDD, Standard SSD, or Premium SSD managed disks, 513 GiB or larger.Creation-only: Performance Plus can only be enabled when the disk is created. You cannot flip it on an existing disk. The migration path is snapshot → create new disk from snapshot with the flag. # Create a 513 GiB Premium SSD with Performance Plus enabled az disk create \ -g rg-prod -n data-disk-perfplus \ --size-gb 513 --sku Premium_LRS \ --performance-plus true # Migrate an existing disk: snapshot, then recreate with the flag az snapshot create -g rg-prod -n snap-data-disk --source data-disk-existing az disk create -g rg-prod -n data-disk-migrated \ --size-gb 513 --sku Premium_LRS \ --source snap-data-disk --performance-plus true The DR implication: Performance Plus is a property of the source disk at creation time, and ASR replicates the disk data to replica disks in the target region. If performance characteristics matter at failover (and for database workloads they always do), validate in a test failover that the recovered disks deliver the IOPS/throughput you expect — and remember that for Premium SSD v2, the IOPS copied is the value at the time replication was enabled, not live values.Practical Setup: SQL FCI on Shared DisksThe most common shared-disk scenario is SQL Server FCI. Here is the PowerShell flow using Az.RecoveryServices — note that shared-disk replication is a PowerShell/portal feature; the CLI extension does not expose the protection-cluster concept.After creating the vault, fabrics, containers, and network mappings (the standard A2A setup), you create a protection cluster — the ASR container that groups all replicated items belonging to one WSFC cluster: # Create the protection cluster for the WSFC $clusterJob = New-AzRecoveryServicesAsrReplicationProtectionCluster -AzureToAzure ` -Name "sql-fci-cluster" -ProtectionContainerMapping $forwardpcm $cluster = Get-AzRecoveryServicesAsrReplicationProtectionCluster ` -ProtectionContainer $pc -Name "sql-fci-cluster" Then enable replication for each node, passing the cluster and per-disk replication config (normal disks and the shared disk in the same configuration): # Disk replication config: include every disk, including the shared one $disk1 = New-AzRecoveryServicesAsrAzureToAzureDiskReplicationConfig -ManagedDisk ` -LogStorageAccountId $cacheSaId -DiskId $osDiskId ` -RecoveryResourceGroupId $targetRgId ` -RecoveryReplicaDiskAccountType Premium_LRS ` -RecoveryTargetDiskAccountType Premium_LRS $disk2 = New-AzRecoveryServicesAsrAzureToAzureDiskReplicationConfig -ManagedDisk ` -LogStorageAccountId $cacheSaId -DiskId $sharedDiskId ` -RecoveryResourceGroupId $targetRgId ` -RecoveryReplicaDiskAccountType Premium_LRS ` -RecoveryTargetDiskAccountType Premium_LRS $disks = @($disk1, $disk2) # Enable protection for this cluster node $enableJob = New-AzRecoveryServicesAsrReplicationProtectedItem -AzureToAzure ` -Name $rpiName -AzureToAzureDiskReplicationConfiguration $disks ` -ReplicationProtectionCluster $cluster -AzureVmId $vmId ` -ProtectionContainerMapping $forwardpcm -RecoveryResourceGroupId $targetRgId ` -RecoveryAvailabilitySetId $avsetId -RecoveryProximityPlacementGroupId $ppgId ` -RecoveryAzureNetworkId $networkId -LogStorageAccountId $cacheSaId Repeat for the second node. The disk replication configuration must contain both the normal disks and the shared disk information, and every VM in the cluster must be included.Then: run a test failover. For clustered workloads this is not optional — you need to confirm the cluster quorum comes up in the recovery region, the shared disk attaches to the surviving node, and SQL/SAP actually starts. Budget a maintenance window and do it before your first DR audit, not during one.Planning Numbers That Hold UpRather than invent RPO/RTO figures, anchor on what ASR actually guarantees and configure: ParameterASR default / control Crash-consistent recovery point frequency5 minutes (Azure-to-Azure) App-consistent snapshot frequencyConfigurable per policy for standard A2A replication — not available for shared-disk clusters (crash-consistent points only) Recovery point retentionConfigurable per policy (e.g., 24 h) Failover RTODepends on disk size, churn, and VM start time — measure with test failovers Your actual RTO depends on churn rate, disk sizes, and how fast the target VMs boot — the only honest way to publish an RTO number is to measure it with test failovers and re-measure as the workload grows.Cost model: ASR is billed per protected instance (each VM counts as one instance), with the first 31 days of protection free per instance. On top of the service fee you pay for replica storage in the target region, storage transactions, and egress. There is no documented separate per-shared-disk ASR fee — the shared disk is replicated like any other disk, so it adds replica storage cost rather than a licensing line. Performance Plus itself carries no extra charge. Check the current pricing page for your regions before quoting numbers to a customer.Pitfalls to Plan AroundPerformance Plus cannot be retrofitted. It is creation-only. If your production disks don't have it and your DR disks need to match, you'll be doing snapshot-migrate-dance on production disks. Plan that migration deliberately rather than discovering it mid-failover-test.Mixed SCSI+NVMe VMs are out of scope. If a VM has both controller types (e.g., Lsv3-style SKUs), it cannot be replicated. Split the workload or change SKU before committing to an ASR-based DR design.Linux NVMe support is preview. Fine for validation and lower-tier workloads; confirm GA status before writing production RPO commitments against it.Shared disks are Windows/WSFC/active-passive only. Linux Pacemaker clusters, active-active configurations, and non-WSFC distributed appliances are not covered — those need a different DR pattern (application-level replication, geo-replicated storage, or a warm standby).All-or-nothing cluster protection. Selecting half the cluster fails or produces warnings. Decide the cluster boundary up front and protect the whole thing.Don't trust AI-generated ASR CLI examples. The Azure CLI site-recovery extension command names differ from what most generated guides show (there is no az site-recovery replication-protected-item create), and shared-disk flows are PowerShell/portal only. Validate commands against the current docs before running them.Premium SSD v2 IOPS is frozen at enablement. If source v2 disk IOPS changes after replication is enabled, the failover disk gets the old value. Adjust after failover if needed.Key TakeawaysNVMe VMs are now protectable by ASR — Windows is supported and Linux (RHEL 9, SLES 15, Ubuntu 24 on Gen2 NVMe families) is in preview. Ephemeral/local NVMe and mixed-controller VMs remain excluded.Shared-disk DR means WSFC clusters survive region loss — SQL FCI, SAP ASCS/SCS, and file server clusters can be protected, failed over, and failed back as a single unit with cluster-consistent recovery points. Windows, active-passive, all nodes together.Performance Plus is supported and free, but creation-only and 513 GiB+ — design your disk provisioning around it rather than trying to retrofit it later.Test failover is mandatory for clustered workloads — quorum behavior and shared-disk attachment in the recovery region are exactly where silent DR failures hide.Anchor RPO/RTO on measured test failovers, not vendor-agnostic estimates: ASR gives you 5-minute crash-consistent frequency and configurable app-consistent snapshots; your RTO is what you measure.If your DR estate includes any of these workload types, this is a good quarter to close the coverage gap — the features are in the support matrix now, and the configuration is documented. The only thing left is the test failover. --- ## Azure Resiliency — Building and Testing a Real DR Plan for Malaysian Enterprises URL: https://wenfeng.my/azure-resiliency-building-and-testing-a-real-dr-plan-for-malaysian-enterprises/ Author: admin Published: 2026-08-17 06:30:31 Modified: 2026-10-04 15:27:48 I've lost count of the enterprise clients I've visited who can produce a beautifully formatted DR runbook on request — complete with recovery time objectives, recovery point objectives, and escalation matrices. Then I ask the follow-up question: "When was the last time you actually tested it?"The room goes quiet.This is the dirty secret of disaster recovery in Southeast Asia. We document it. We present it to auditors. We put it in the compliance binder. And then we pray we never have to use it — because the honest truth is that most DR plans have never been exercised end-to-end. The runbook is theory. The failover procedure is fiction.Two things changed that equation in mid-2026. First, the Resiliency in Azure service — the rebrand and expansion of what was Azure Business Continuity Center — shipped Availability Zone Down Drills and Recovery Orchestration Plans in public preview, giving us platform-native ways to actually exercise zone failure and orchestrated failover. Second, and just as important for my Malaysian clients: Malaysia West (malaysiawest) launched as a nonpaired region, which breaks the default DR architecture playbook most of us have been following for a decade.This article is the implementation guide I wish someone had handed me six months ago: what these new capabilities actually do, how Malaysia West's unpaired status changes your DR design, and how to build and test a real plan. I've covered the fundamentals of Azure Resiliency in [Azure Resiliency Gets Real — Availability Zone Drills, Zonal Recovery Orchestration](https://wenfeng.my/azure-resiliency-in-2026-availability-zone-drills-and-zonal-recovery-that-actually-work/, so here I'm going deeper on the build-and-test workflow.Caveat up front: Availability Zone Down Drills and Recovery Orchestration Plans are in public preview as of mid-2026. ARM API versions in the Microsoft.AzureResilienceManagement provider are preview-stamped (2025-02-01-preview through 2026-06-01-preview). Schemas and portal workflows may change before GA. Treat everything here as preview-grade and validate against the current docs before you wire it into production procedures.The Problem: Documented but UntestedHere's the realistic picture of DR readiness at most mid-to-large Malaysian enterprises I work with:The runbook exists — usually a Word document or Confluence page written by a consultant two years ago. It references resource groups and naming conventions that have since changed.The backup infrastructure is there — Azure Site Recovery is configured, database backups flow to geo-redundant storage, and someone created a Recovery Services vault.Nobody has actually failed over — the last "test" was a tabletop exercise where people walked through the runbook verbally. The actual failover has never been executed.Nobody knows how long recovery really takes — the RTO in the document says 4 hours, but that was an estimate from deployment day. Nobody has measured real recovery time under realistic conditions.This is dangerous for reasons specific to Malaysian enterprises. Bank Negara Malaysia's Risk Management in Technology (RMiT) policy document — dated 1 January 2020, with a revised version in force from June 2023 and further revisions in exposure draft — sets a continuous-compliance expectation: financial institutions must maintain and produce defensible, current evidence that recovery capabilities actually work, including regular DR and cyber-resilience testing. If your evidence is a two-year-old runbook and a tabletop exercise, you're going to struggle to demonstrate effectiveness when BNM asks.PDPA matters too. Malaysia's PDPA Amendment Act 2024 introduced mandatory data breach notification — data users must notify the Commissioner of qualifying breaches within the prescribed timeframe. A disaster that exposes personal data can trigger that obligation, and your DR plan should include a breach-assessment step at the start of any failover event, not as an afterthought.The historical blocker has been cost and risk: testing DR meant either accepting production risk (real failover against live traffic) or maintaining a parallel environment that's expensive and diverges from reality. Azure Resiliency's preview features attack exactly that problem — but they do it in a way that surprises people, so let's be precise about what they actually are.What Azure Resiliency Actually Gives YouThree capabilities matter for DR testing, all preview as of mid-2026:1. Availability Zone Down DrillsAZ Down Drills live in Infrastructure Resiliency Manager. They let you simulate an availability zone outage for a Service Group — a logical grouping of the resources that must survive a zone failure together — and assess cross-zone resiliency before a real outage forces you to find out.Here's the part that surprises people: the drill is a real fault injection, not a harmless simulation. Under the hood it uses Azure Chaos Studio-style faults — VM shutdowns, Virtual Machine Scale Set zone shutdowns, AKS node pool shutdowns, forced database failovers — injected against the resources in your target zone. Resources with native zonal resiliency absorb it; resources relying on a Recovery Plan get failed over and reprotected as part of the drill. For resource types without native support, you plug in custom fault logic via Azure Automation runbooks.That has two consequences. First, schedule drills like a real exercise — notify stakeholders, pick a maintenance window, and expect real (if brief) impact on the drilled resources. Second, the drill produces exactly the evidence auditors want: run-level status, resource-level health and downtime metrics, attestation, and drill history.2. Recovery Orchestration PlansRecovery Orchestration Plans are executable recovery sequences for zonal failover. Key facts from Microsoft's own documentation, because they shape the whole design:Zonal only. The service explicitly "supports only zonal recovery and doesn't support regional recovery." If you need cross-region DR, that's still Site Recovery, failover groups, geo-replication, and Traffic Manager — the orchestration plan is not that tool (yet).Ordering via groups. Resources recover in groups: resources within a group fail over in parallel; groups execute sequentially. That's how you encode "databases before application tier, application tier before web tier."Readiness checks every 24 hours. Automated checks detect drift — resources added or removed, protection health degraded — so the plan fails loudly in advance rather than silently during an incident.Runbook integration. Azure Automation runbooks can run as pre-scripts and post-scripts at group level, and manual action steps can pause execution until a human completes an intervention.It orchestrates, it doesn't protect. VMs still need Azure Site Recovery; other resources need their native HA solution. The plan sequences the failover; it is not a replication technology.3. Zonal Resiliency Goals and RecommendationsInfrastructure Resiliency Manager also scores your estate against zonal resiliency goals and produces resource-level recommendations with cost-impact indicators (High / Medium / Low / None). This is the cheapest way to find the gaps before the drill finds them for you — single-zone VMs, non-zone-redundant disks, load balancers without zone-redundant front ends.The Malaysia West Trap: Your Region Has No PairThis is the part that matters most for my local readers. If you deployed into Malaysia West assuming the classic playbook — primary region, paired secondary, GRS handles the rest — you have a gap.Per Microsoft's region pairs documentation, Malaysia West is a nonpaired region. Its paired region is N/A. It's in the same category as Indonesia Central, Mexico Central, and several European regions: newer regions built around availability zones rather than region pairs.The practical consequences:Storage GRS has no automatic paired target. Geo-redundant storage replicates to the paired region; with no pair, you must choose your secondary region explicitly (or rely on the service's own multi-region options).No Microsoft-managed recovery-pair sequencing. You don't get the geography-wide-outage recovery prioritization that paired regions provide.Cross-region failover is entirely your design. Azure Site Recovery, SQL failover groups, Cosmos DB multi-region, Traffic Manager — you pick the secondary region yourself. For most Malaysian enterprises the sensible choices are Southeast Asia (Singapore) or East Asia (Hong Kong), both of which are paired with each other.The good news: zonal resiliency is strong. Malaysia West launched with three availability zones, and Microsoft's Malaysia region playbook pushes zone redundancy as the first line of defence. That's exactly why the new Azure Resiliency features land so well here: the platform's story for this region is zones first, cross-region second — and now there's tooling to prove both layers. # Verify which regions are actually available to your subscription # and confirm Malaysia West has no pair az account list-locations \ --query "[?contains(name,'malaysia') || name=='southeastasia' || name=='eastasia'].{region:name, displayName:displayName, pairedRegion:metadata.pairedRegion}" \ --output table Building the Plan: A Practical WalkthroughHere's the workflow I now use with clients running production in Malaysia West. Zonal resilience is the foundation; cross-region is the backstop.Step 1: Define Service GroupsA Service Group is the unit of everything in Resiliency: drills, recovery plans, and goals all hang off it. Define it as "the set of resources that must survive a zone failure together" — typically one application's full stack: web tier, app tier, database, cache, load balancer.Service Groups are managed through the Resiliency area of the Azure portal. (The underlying resource type sits in the Microsoft.AzureResilienceManagement provider, but the portal is the practical surface — there is no dedicated az resiliency CLI extension as of mid-2026.)Step 2: Register the Supporting ProvidersAZ Down Drills pull in several platform services. Register them on the subscription where the drill's Chaos Workspace will live — registration takes 15-20 minutes, so do it before your drill window, not during: # Prerequisites for AZ Down Drills (portal: Subscriptions > Resource providers) for RP in Microsoft.AzureResilienceManagement Microsoft.Chaos Microsoft.Insights \ Microsoft.OperationalInsights Microsoft.Automation; do echo "Registering $RP ..." az provider register --namespace "$RP" done # Wait for registration to complete for RP in Microsoft.AzureResilienceManagement Microsoft.Chaos Microsoft.Insights \ Microsoft.OperationalInsights Microsoft.Automation; do az provider show --namespace "$RP" --query "registrationState" --output tsv done Step 3: Close the Obvious Gaps FirstRun the zonal resiliency goals assessment and fix the cheap ones before spending a drill on them. Meanwhile, verify the basics with plain CLI — these commands are the ones I run in every engagement: # Which resource groups live in Malaysia West? az group list \ --query "[?location=='malaysiawest'].{name:name, location:location}" \ --output table # Inventory a critical resource group az resource list --resource-group rg-prod-core \ --query "[].{type:type, name:name, location:location}" \ --output table # Zone-redundant disks or not? (LRS vs ZRS) az disk list --resource-group rg-prod-core \ --query "[].{name:name, sku:sku.name, zones:zones}" \ --output table # Storage redundancy — check whether you're on GRS and where the secondary is az storage account show --name stcoredata001 --resource-group rg-prod-core \ --query "{sku:sku.name, primary:primaryLocation, secondary:secondaryLocation}" Step 4: Set Up Protection and the Recovery PlanProtection is still your job — the orchestration plan only sequences what's already protected:VMs: enable Azure Site Recovery replication (zone-to-zone for zonal DR, region-to-region for the cross-region backstop).Azure SQL: zone-redundant deployment, plus a failover group if you want cross-region.AKS: zone-redundant node pools across at least two zones.Storage: ZRS for zone redundancy; explicit secondary region choice for cross-region.Then create the Recovery Orchestration Plan inside the Service Group. In Bicep terms, the published ARM schema makes the shape clear — planType is Zonal, sequencing happens through recoveryGroupsSetting (a required defaultGroup plus optional additionalGroups), and each group carries preActions/postActions that are either CustomRunbook (an Azure Automation runbook with an identity and parameters) or ManualAction steps: // Recovery Orchestration Plan — zonal failover (preview schema) // Source of truth: learn.microsoft.com/azure/templates/ // microsoft.azureresiliencemanagement/recoveryplans resource drPlan 'Microsoft.AzureResilienceManagement/recoveryPlans@2026-06-01-preview' = { name: 'drplan-prod-core' // pattern: 5-24 chars, letters/numbers/hyphens properties: { planDescription: 'Zonal failover for prod core app in Malaysia West' planType: 'Zonal' recoveryGroupsSetting: { defaultGroup: { properties: { description: 'Database tier first' groupUniqueId: 'grp-database' orderId: 1 postActions: [ { name: 'db-health-check' description: 'Runbook: verify DB health post failover' type: 'CustomRunbook' timeoutInMinutes: 10 actionResourceId: runbookDbHealth.id associatedIdentity: { type: 'UserAssigned' userAssignedIdentity: uamiDr.id } parameters: { targetEnv: 'production' } } ] } } additionalGroups: [ { properties: { description: 'App tier after database' groupUniqueId: 'grp-app' orderId: 2 preActions: [ { name: 'drain-lb' description: 'Manual: confirm load balancer drained' type: 'ManualAction' } ] } } ] } } } Note what's not in that schema: there is no sourceRegion/targetRegion, and no testFailover endpoint — the plan is zonal, and testing happens through drills, not a separate test-failover mode. If you see example payloads with those fields, they're wrong.Two RBAC roles exist for this work: Azure Resilience Management Recovery Contributor and Recovery Administrator — assign least-privilege accordingly.Step 5: Define and Run an AZ Down DrillCreate the drill on the Service Group (portal: Service groups → your group → Resiliency → Drills). You'll pick a subscription/region for the Chaos Workspace and Log Analytics workspace (can differ from the workload), assign an identity for fault injection and recovery plan execution, review the drill scope, and design faults per resource — default fault duration is 10 minutes per resource.Before executing, the portal runs a readiness check: Resync and check readiness must show Ready. This is where drift gets caught — excluded resources, broken role assignments, metrics not configured.Then execute. The drill runs as one job with three stages:Fault Injection — the zone's resources take the injected faults.Failover — the Recovery Plan orchestrates recovery of affected resources.Reprotection — resources are reprotected to restore the protection baseline.You advance each stage ("Mark step as complete") and monitor Service Group health throughout. At the end you attest the drill result and record notes. That attestation record is your RMiT evidence: who ran it, when, what the measured downtime was per resource.Step 6: The Cross-Region Backstop (Still the Old Tools)Since Recovery Orchestration Plans are zonal-only, your regional DR stays on the classic stack. The verification commands are worth automating: # Site Recovery replication health for the cross-region backstop az site-recovery protected-item list \ --resource-group rg-prod-dr \ --vault-name rsv-prod-dr \ --query "[].{name:name, state:replicationHealth, rpo:recoveryPointTime}" \ --output table # Azure SQL: failover group status (if configured cross-region) az sql failover-group show \ --name fog-prod \ --server sql-prod-malaysiawest \ --resource-group rg-prod-data \ --query "{replicationState:replicationState, partner:partnerServers[0].id}" Design guidance for Malaysian enterprises: keep personal-data workloads in-region where PDPA residency arguments apply (Malaysia West primary + zone redundancy), use Southeast Asia as the explicit cross-region backstop, and document the cross-border transfer assessment in your DPIA.PDPA Considerations for Malaysian EnterprisesIf you process personal data of Malaysian citizens — and most enterprise applications do — your DR plan needs to account for PDPA:Cross-region failover is a cross-border transfer. Failing over from Malaysia West to Southeast Asia moves personal data into Singapore. Under PDPA Section 129 — the cross-border transfer provision, in Part X (Miscellaneous) — you need to ensure comparable protection in the receiving jurisdiction and document the assessment. Singapore is generally a defensible choice, but write it down in your DPIA — don't assume it.Breach notification is now mandatory. The PDPA Amendment Act 2024 added mandatory breach notification (the original PDPA 2010 had no such provision). A disaster event that risks exposing personal data can trigger notification to the Commissioner. Build a breach-assessment step into the first hour of every failover procedure.Data minimisation in DR. Replicate only what recovery genuinely needs to the secondary region. Test data, dev data, and unnecessary PII don't belong in your DR site.Tested DR is compliance evidence. PDPA's security principle requires demonstrating that you took reasonable precautions. A quarterly AZ drill with attestations and measured RTO is far stronger evidence than a runbook PDF. # Tag DR resources for audit traceability az tag create --resource-id "/subscriptions/{sub}/resourceGroups/rg-prod-dr" \ --tags PDPA=applicable DataClassification=PersonalData DRTestFrequency=quarterly Pitfalls I've SeenPitfall 1: Treating the drill as a harmless simulation. AZ Down Drills inject real faults — VMs actually shut down, databases actually fail over. Run them in a controlled window with stakeholders notified. The "risk-free testing" pitch is about recovery being safe because protection is in place, not about the drill touching nothing.Pitfall 2: Assuming Malaysia West has a pair. I've seen three teams now discover at DR design time that GRS had nowhere to replicate. Check az account list-locations output for your actual region before you architect anything.Pitfall 3: Expecting the orchestration plan to do regional failover. It's zonal only — that's documented, not a bug. If your runbook's scope is "region down," the plan covers the zone-level part of it; the regional part is still your Site Recovery and failover group design.Pitfall 4: Forgetting the human factor. The best orchestration plan is useless if the on-call engineer doesn't know how to trigger it at 3 AM. Put the drill trigger steps in the on-call runbook, and make sure at least two people can execute independently.Pitfall 5: Letting the plan rot. Readiness checks run every 24 hours precisely because environments drift. If you're seeing Needs Attention states and ignoring them, you've re-created the original problem: a plan that looked fine on paper. Drill quarterly, re-validate after every significant architecture change.Pitfall 6: Not involving the security team. Failover changes your security posture temporarily — network rules, firewall behaviour, identity endpoints in the surviving zone or secondary region. Security should be in the drill, not briefed after it.What This CostsLet's talk numbers, with the honest caveat that these are order-of-magnitude estimates for a mid-sized enterprise (4-6 app VMs, managed databases, load balancers):The Resiliency features themselves: preview tooling is currently part of the platform — the line items you'll actually pay for are the underlying protection (Site Recovery replicated instances, zone-redundant storage and disks, SQL failover groups), not the orchestration.Zonal redundancy premium: mostly in storage and database replication. Compute and load balancer costs barely move between single-zone and multi-zone; zone-redundant disks and SQL zone redundancy are where the premium lives. My [companion article on Azure Resiliency](https://wenfeng.my/azure-resiliency-in-2026-availability-zone-drills-and-zonal-recovery-that-actually-work/ walks a component-by-component cost table.Cross-region warm standby: the real money. A warm standby for a typical enterprise workload in Southeast Asia runs roughly MYR 8,000-15,000/month depending on SKUs and reserved capacity. Cold/pilot-light designs can cut that substantially at the cost of RTO.Drill effort: budget 2-4 engineering hours per AZ drill including readiness fixes, execution, and attestation write-up. Quarterly drills are the sweet spot I recommend.Set that against the cost of discovering your failover is broken during an actual outage. For a 500-person organisation, even one hour of regional downtime lands comfortably in six figures of MYR in lost productivity and revenue. The ROI of tested recovery is not close.Key TakeawaysTest, don't just document. AZ Down Drills and Recovery Orchestration Plans (both preview) remove most of the excuses for untested DR. RMiT's shift toward control effectiveness means untested runbooks are becoming a compliance liability, not just a technical one.Malaysia West has no paired region. Design zones-first for HA, and pick your cross-region secondary explicitly — Southeast Asia is the common choice for Malaysian enterprises. Verify with az account list-locations before you build on assumptions.Drills are real fault injection. They use Chaos Studio-style faults with real (if controlled) impact on the drilled zone. Schedule them like exercises, and use the attestation output as your audit evidence.Recovery Orchestration Plans are zonal only. They sequence failover groups, run Automation runbooks as pre/post-scripts, and check readiness daily — but they don't protect resources and don't do regional failover. Keep Site Recovery and failover groups for the cross-region layer.PDPA follows you into DR. Cross-border transfer assessment for your secondary region, mandatory breach notification steps in the failover procedure, and minimised data in the DR site. Your DPIA should name all three.The era of "we have a DR plan, trust us" should be over. Azure Resiliency — even in preview — finally gives Malaysian enterprises the tooling to prove resilience instead of claiming it. The question is no longer whether you can afford to test your DR plan. It's whether you can afford to walk into your next BNM review with a runbook nobody has ever executed. --- ## Azure Kubernetes Service (AKS) 2026 — What Build Announcements Mean for Production Workloads URL: https://wenfeng.my/aks-2026-build-announcements-production-workloads/ Author: admin Published: 2026-08-16 23:30:19 Modified: 2026-10-04 15:27:59 Microsoft Build 2026 produced a substantial wave of Azure Kubernetes Service announcements, and most of them point in the same direction: AKS is shifting from "managed Kubernetes" toward an opinionated platform for running AI-heavy production workloads with as little undifferentiated cluster work as possible.If you run or plan production workloads on AKS — whether you are a Malaysian enterprise standardizing on malaysiawest, or a regional team operating clusters across Azure, on-premises, and other clouds — the useful question is not "what was announced?" but "what requires action now, and what can wait?"This article cuts through the marketing. Everything below is checked against the official AKS release notes and Microsoft Learn documentation, with dates and commands you can verify yourself.What Build 2026 Actually Announced for AKSMicrosoft's own framing of the Build announcements was honest about the problem: running training and inference at scale means cost, latency, and reliability are constantly in tension. The announcements expand what you can control at four layers: the cluster itself, hardware proximity, fleet operations, and the AI workload layer.1. AKS Automatic grows up: managed system node pools are GAAKS Automatic reached general availability in September 2025 (release notes 2025-09-21), with its compute-based billing model taking effect on 19 October 2025. At Build 2026, the headline cluster-layer announcement was that managed system node pools in AKS Automatic are now generally available (release notes 2026-05-29).Previously, even on AKS Automatic you still had to plan capacity, patch, and scale the system node pool that runs core cluster components like CoreDNS and metrics-server. Now Azure handles that entire lifecycle. Three practical consequences:System VMs are not charged to your subscription — AKS absorbs the compute cost of the managed system pool.System components stop competing with your workloads for resources, which matters a lot when your user nodes are expensive GPU machines.Security posture is hardened by default: new Automatic clusters with managed system node pools block customer-supplied SSH keys, deny kubectl port-forward to the system pool, restrict read access to kube-system secrets, and enforce a ValidatingAdmissionPolicy that blocks Services from setting spec.externalIPs.AKS Automatic clusters also come preconfigured with the Standard tier (uptime SLA plus a financially backed pod readiness SLA — 99.9% of qualifying pod readiness operations complete within five minutes), a managed virtual network on Azure CNI Overlay powered by Cilium, a managed NAT gateway for egress, and — from AKS 1.36 — Gateway API ingress via the application routing add-on. New Automatic clusters also preconfigure LocalDNS in Required mode. # Create an AKS Automatic cluster (verified syntax from the AKS quickstart) az aks create \ --resource-group rg-production \ --name aks-automatic-prod \ --sku automatic \ --enable-hosted-system # Confirm the managed system profile az aks show \ --resource-group rg-production \ --name aks-automatic-prod \ --query hostedSystemProfile The practical impact: for new workloads that do not need exotic node-level customization, AKS Automatic removes most of the remaining node-lifecycle work. One important caveat: new AKS Automatic clusters can no longer be created without managed system node pools, and if you have an existing Automatic cluster without them, the documented path is to recreate the cluster and migrate workloads — there is no in-place flip.AKS Automatic is GA in more than 35 regions, including malaysiawest, southeastasia, and eastasia.2. The AI workload layer becomes first-classThis is where Build 2026 differed most from previous years. Three announcements target AI training and inference directly:AKS on bare metal (public preview). You can now run AKS on dedicated machines without a hypervisor, with direct access to NVLink, RDMA, and high-performance networking — while keeping the same AKS control plane and APIs. This is aimed squarely at large training jobs, latency-sensitive inference, and high-throughput data pipelines, where virtualization overhead translates directly into cost and runtime.Anyscale on Azure (public preview). Managed Ray, running on AKS, inside your own Azure subscription, integrated with Entra ID, and billed through your existing agreement. Kubernetes handles scheduling and cluster lifecycle; Ray coordinates distributed execution within a workload, including mixed CPU+GPU scheduling, heterogeneous and fractional GPU allocation, and dynamic job scaling. Microsoft's reference customer is Wayve, which uses AKS, Ray, and Anyscale to connect thousands of GPUs into a flexible training supercomputer for its autonomous driving models.AI Runway and KAITO. AI Runway (an open-source project introduced at KubeCon + CloudNativeCon Europe 2026) gives you a Kubernetes-native path from "I have a model" to "I have a serving endpoint": select a model, validate it fits your available GPU memory, review a cost estimate, and deploy. That creates a ModelDeployment custom resource that drives the rest of the workflow. AI Runway sits as a platform layer with multiple serving providers — KAITO, NVIDIA Dynamo, KubeRay, and LLM-D — and supports engines including vLLM, SGLang, TensorRT-LLM, and llama.cpp. Underneath, KAITO — the Kubernetes AI Toolchain Operator, a CNCF open-source project — estimates resource requirements, provisions nodes through Karpenter, and launches the optimized runtime (vLLM today). From there, standard Kubernetes machinery takes over: KEDA autoscales on workload metrics, Gateway API handles routing, and rollouts follow familiar patterns.The Royal Bank of Canada example from Build is worth studying: a self-service AI platform on AKS where dev teams provision GPU resources and deploy models through their own CI/CD, with KAITO handling production serving and model images kept in a private ACR — the whole path wrapped in private endpoints, Entra ID, and Key Vault. That is the compliance-friendly pattern Malaysian financial institutions will recognize.3. Fleet Manager for Arc-enabled clusters is GAAzure Kubernetes Fleet Manager for Arc-enabled clusters is now generally available, extending fleet-level control beyond Azure into on-premises and other-cloud clusters. From one control plane you can roll out updates progressively with health checks between stages, place workloads based on GPU availability and SKU, and apply RBAC consistently. If you operate more than two or three clusters — which is most enterprises I talk to — this is the feature that turns "many clusters" into "one system."4. Azure Container Linux is GAAzure Container Linux, a minimal container-optimized Linux distribution maintained by Microsoft, reached general availability. The smaller package footprint reduces patching overhead and limits configuration drift across clusters. For fleets, a consistent host-OS baseline is easier to secure over time — and it is one fewer OS variant to test during the Ubuntu migration discussed below.The Unglamorous Deadline: Ubuntu 22.04 End of SupportNot a Build announcement, but arguably the most urgent item for anyone running AKS in production today. From the release notes of 2026-04-02:30 June 2027 — AKS stops supporting Ubuntu 22.04 and stops providing security updates. After that date you cannot create new Ubuntu 22.04 node pools, AKS produces no new node images, and existing pools stop receiving security patches.30 April 2028 — AKS removes Ubuntu 22.04 node images and code entirely; scaling and remediation operations fail.17 March 2027 — Ubuntu 20.04 retires: no new node pools, no new node images, and no security patches after that date (node images and code are removed on 17 March 2028).Long-term support (LTS) for Kubernetes 1.33 or later requires Ubuntu 24.04.Migration options are documented and, in the common case, automatic: if your node pools use the default Ubuntu OS SKU, they migrate to Ubuntu 24.04 automatically when you upgrade to Kubernetes 1.35 or later. If you pinned Ubuntu2204, update the OS SKU to Ubuntu (on Kubernetes 1.35+) or Ubuntu2404 (on 1.32+). # Audit every cluster: which pools are on which OS SKU and Kubernetes version az aks show --resource-group rg-prod --name aks-prod \ --query "agentPoolProfiles[].{pool:name, osSku:osSku, version:currentOrchestratorVersion, nodes:count}" \ --output table # Refresh node images (also how CVE mitigations reach your nodes) az aks nodepool upgrade \ --resource-group rg-prod \ --cluster-name aks-prod \ --name systempool \ --node-image-only We saw exactly what waiting costs during the Azure Linux 2.0 retirement: support ended 30 November 2025, the node image froze, and from 31 March 2026 affected pools could no longer scale. Plan the Ubuntu migration in the second half of 2026, not in Q2 2027.Practical Moves for Production ClustersCut idle compute costs with Spot and scale-to-zeroNode compute dominates AKS spend. Two well-proven levers: # Spot node pool for interruption-tolerant workloads az aks nodepool add \ --resource-group rg-prod \ --cluster-name aks-prod \ --name spotpool \ --priority Spot \ --eviction-policy Delete \ --spot-max-price -1 \ --enable-cluster-autoscaler \ --min-count 0 \ --max-count 10 For queue-driven workloads, the KEDA add-on scales deployments to zero when there is nothing to process: apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: worker-scaler spec: scaleTargetRef: name: queue-worker minReplicaCount: 0 maxReplicaCount: 50 triggers: - type: azure-servicebus metadata: queueName: requests namespace: sb-prod One warning from the KEDA documentation: do not combine a KEDA ScaledObject with a HorizontalPodAutoscaler on the same workload — KEDA uses an HPA internally, and the two will fight.Use the new safety net: node pool rollbackNode pool rollback became generally available in the August 2026 release (v20260807). If a node pool upgrade goes wrong, you have seven days to restore the previous Kubernetes version and node image: az aks nodepool rollback \ --resource-group rg-prod \ --cluster-name aks-prod \ --name systempool Combined with planned maintenance windows, this removes much of the weekend-upgrade anxiety from AKS operations.Speed up deployments with Artifact StreamingAlso GA in mid-2026: Artifact Streaming pulls container images from Azure Container Registry on demand, fetching only the layers a pod needs at startup. For large AI images — where multi-gigabyte model-serving images are normal — this materially shortens pod startup time during scale-out events.Pitfalls I See Repeatedly1. Treating AKS Automatic as an in-place upgrade path. It is not. Existing Automatic clusters without managed system node pools must be recreated and migrated, and Standard clusters are not "converted" to Automatic. Evaluate Automatic for new workloads; plan Standard-to-Automatic as a migration project.2. Ignoring node image versions after a security bulletin. Nodes are not patched in place. When AKS issued bulletin AKS-2026-0003 (CVE-2026-31431, a kernel vulnerability that lets a pod escalate to root on the node), the mitigation shipped in node images 202604.13.0 and 202604.24.0 — existing nodes only became protected after a node image upgrade. If your clusters sit on old node images, you are carrying CVEs that already have fixes.3. Staying on Ingress NGINX. The upstream Ingress NGINX project ended maintenance in March 2026, and the AKS application routing add-on's NGINX experience remains production-supported only through November 2026. New Automatic clusters (1.36+) already default to Gateway API. Start migration planning now.4. Waiting until 2027 for the Ubuntu migration. The Azure Linux 2.0 retirement showed the failure mode: frozen images, then scaling failures. Teams that audit OS SKUs now can ride the migration inside normal Kubernetes version upgrades instead of an emergency project.5. Putting preview features on the production path. AKS on bare metal and Anyscale on Azure are public previews. Design your production AI path on GA primitives (GPU node pools, KAITO, KEDA, Fleet Manager) and use the previews for evaluation and benchmarking until they reach GA.Recommendations for Malaysian EnterprisesPilot AKS Automatic in `malaysiawest` for your next new workload — the managed system node pools, Cilium networking, and pod readiness SLA remove most operational overhead.Audit OS SKUs across all clusters this quarter and schedule the Ubuntu 24.04 transition with your normal upgrade cadence in late 2026.For AI platforms, follow the RBC pattern: KAITO for serving, private ACR for model images, private endpoints plus Entra ID workload identity for data access. If you need managed distributed training, evaluate Anyscale on Azure in preview.Adopt the new operational safety nets: node pool rollback, Artifact Streaming, and automatic availability zone placement (availabilityZones=["auto"], globally enabled in the August 2026 release).Put Ingress NGINX migration on the 2026 roadmap before the November 2026 support boundary.Key TakeawaysAKS Automatic is the low-operations default now. Managed system node pools are GA, system pool compute is free to your subscription, and the default security posture is stricter than most hand-built clusters.The 2026 AI story is GPU operations on Kubernetes — bare-metal preview, Anyscale/Ray, and the AI Runway + KAITO toolchain make AKS a credible substrate for both training and inference.Ubuntu 22.04 support ends 30 June 2027. Audit OS SKUs now; the default Ubuntu SKU migrates automatically on Kubernetes 1.35+.Fleet Manager for Arc GA means one control plane across Azure, on-premises, and other clouds — progressive rollouts, GPU-aware placement, consistent RBAC.Use the safety features the platform just gained: seven-day node pool rollback, Artifact Streaming for fast image pulls, and automatic zone placement.If you are planning an AKS modernization or an AI platform on Azure and want a second opinion on the architecture, reach out — I am happy to review your design. --- ## Microsoft Fabric Data Agents — Building Agentic AI on Your Enterprise Data URL: https://wenfeng.my/microsoft-fabric-data-agents-building-agentic-ai-on-your-enterprise-data/ Author: admin Published: 2026-08-16 06:30:45 Modified: 2026-10-04 15:28:10 When Microsoft announced that Fabric Data Agents reached general availability at FabCon 2026 — alongside the news that Fabric is now serving more than 31,000 customers, the fastest-growing data platform in Microsoft's history — I had the same reaction I always have when a platform vendor ships "agents": cautious interest, tempered by years of watching marketing outrun reality.Then I read the actual documentation, tested the setup flow, and mapped the feature against what enterprises in Malaysia genuinely need. My conclusion: this one is worth your attention, but not for the reasons the launch blogs suggest. The interesting part isn't that Microsoft bolted a chatbot onto Fabric. It's that they built a read-only, identity-enforced, governance-aware conversational layer directly over OneLake — and that design decision changes everything about how you should evaluate it.The Problem: The Data Grounding GapEvery enterprise team I work with has the same frustration. You can build a Copilot Studio bot in an afternoon, but getting it to reliably answer questions that require joining data across your ERP, data warehouse, and operational systems takes months. The agent can talk, but it can't think over the data that actually matters.The root cause is grounding. Point a large language model at raw warehouse tables and you get three failure modes:Missing business semantics. The model sees cust_id in one table and customer_number in another and has no idea they're the same entity. It doesn't know whether "revenue" means gross or net, or whether intercompany sales are included.Expensive RAG pipelines. The alternative — copy data out, chunk it, embed it, maintain a vector index — creates a second data estate that drifts out of sync with the first one. Every schema change becomes an integration project.Governance holes. Most DIY patterns either embed everything (permission boundaries vanish) or pass user credentials through hand-written code that nobody wants to audit.Fabric Data Agents attack the problem from the opposite direction: instead of moving data to the model, the model's queries run where the governed data already lives.What a Fabric Data Agent Actually IsLet me be precise, because precision matters here. A Fabric Data Agent is a conversational Q&A system that answers natural language questions over Fabric data sources. It is now generally available, and these are its defining characteristics:Supported data sources: warehouses, lakehouses, Power BI semantic models, KQL databases (including Eventhouse), mirrored databases, ontologies, and Microsoft Graph. Each agent can use up to five data sources in any combination.Read-only by design. This is the single most important fact about the product, and the one most launch coverage missed. Fabric Data Agents strictly enforce read-only connections. They don't trigger pipelines, don't run notebooks, don't modify anything. If you've seen demos implying agents "take action" on your warehouse, that's Operations Agents and Copilot Studio territory — a different capability.No model plumbing required. You don't supply an Azure OpenAI key or manage tokens. Fabric uses a Microsoft-managed Azure OpenAI Assistant service and handles authentication for you.Queries run as the requesting user. The agent reads schemas and executes SQL, DAX, or KQL only with the data the asking user can already access. Least privilege is enforced by identity, not by prompt engineering.Under the hood, the flow is: question parsing against security and responsible-AI policies, identification of the most relevant data source, then query generation — NL2SQL for lakehouses and warehouses, NL2DAX for semantic models, NL2KQL for KQL databases — followed by validation, execution, and a human-readable answer.Where It Sits in Fabric IQMicrosoft introduced Fabric IQ at Build 2026 — now generally available — as the business context layer for agents. Three layers matter here:Unified Data — your estate in OneLake, with shortcuts to SharePoint and OneDrive now GA, and shortcuts from Fabric Data Warehouses in preview.Business Intelligence — governed Power BI semantic models that encode business definitions.Operational Intelligence — ontologies (in preview) that model business entities, relationships, rules, and actions.Data Agents consume all three. When you ask "What were our top 10 customers by revenue in Q1?", a well-configured agent resolves "revenue" against your semantic model rather than guessing from a column name. That semantic grounding is the difference between a demo and something a CFO will sign off on.Setting Up Your First Data AgentHere's the practical walkthrough, based on the GA product.Step 1: Capacity and Tenant SettingsPrerequisites: an F2 or higher Fabric capacity (or Power BI Premium P1+ with Fabric enabled), plus at least one supported data source with data in it.Enablement is done in the Fabric Admin Portal → Tenant settings — there's no API for this part, and don't let anyone tell you otherwise:Users can use Copilot and other features powered by Azure OpenAI — the master switch.Capacities can be designated as Fabric Copilot capacities — lets capacity admins assign Copilot capacity.Cross-geo processing and storage settings — and this one matters for Malaysia.Malaysia note: if your Fabric capacity sits outside the EU data boundary and the US — which Malaysian capacities do — you must explicitly enable the tenant settings allowing data sent to Azure OpenAI to be processed, stored, and (for conversation history) held outside your capacity's region. These settings may take up to an hour to propagate. For organizations subject to PDPA or sector regulations, this is a documented, deliberate consent decision — get it approved by your compliance owner before flipping the switch, and record the decision.Step 2: Create the Agent and Select Data SourcesIn your workspace, create a new Data agent item, name it (e.g., "Finance Q&A"), and select its data sources. Remember the five-source limit — it's a design constraint, not an inconvenience. It forces you to scope each agent to one business domain, which is exactly the governance posture you want.Step 3: Instructions and Example QueriesThis is where quality is won or lost. You can add:Organization-specific instructions — plain-language guidance on terminology, tone, and how to interpret ambiguous questions.Example queries — up to 100 per data source. Seed these with the questions your business users actually ask, including the tricky ones with known joins and filters.Data source guidance — hints about which source covers which topic.The improved NL2SQL engine (preview as of mid-2026) uses these example queries to follow your team's patterns and asks clarifying questions when intent is ambiguous. Invest in examples early; it's the highest-leverage hour in the whole setup.Step 4: Bring Outside Data In — The Right WayIf your source data lives outside Fabric, there are two correct patterns:OneLake shortcuts for lake and file-based sources — ADLS Gen2, Amazon S3, Google Cloud Storage, Azure Blob, Dataverse, SharePoint/OneDrive. You can create them programmatically via the Fabric REST API: # Get a Fabric API token with the Azure CLI TOKEN=$(az account get-access-token \ --resource https://api.fabric.microsoft.com \ --query accessToken -o tsv) # Create an ADLS Gen2 shortcut inside a lakehouse item curl -s -X POST \ "https://api.fabric.microsoft.com/v1/workspaces/${WORKSPACE_ID}/items/${LAKEHOUSE_ITEM_ID}/shortcuts" \ -H "Authorization: Bearer ${TOKEN}" \ -H "Content-Type: application/json" \ -d '{ "path": "Tables/landing", "name": "PartnerProducts", "target": { "adlsGen2": { "location": "https://contosoadls.dfs.core.windows.net", "subpath": "/mycontainer/data/ContosoProducts", "connectionId": "91324db9-8dc4-4730-a1e5-bafabf1fb91e" } } }' Note the endpoint shape: shortcuts hang off /v1/workspaces/{workspaceId}/items/{itemId}/shortcuts — the item being your lakehouse — and the target must specify exactly one supported destination. OneLake shortcuts do not point at Azure SQL tables; for operational databases, use mirrored databases, which are a first-class Data Agent data source and keep operational data queryable without ETL.Step 5: Test, Then Choose Your ChannelTest in the built-in chat pane, then share. As of June 2026, Data Agents can be published into Microsoft 365 Copilot (GA — business users discover and chat with your agent inside M365 Copilot with Entra ID permissions enforced), embedded in Copilot Studio bots, consumed as a tool by Microsoft Foundry agents (with observability telemetry in preview), or exposed as an MCP server for other agent ecosystems.Where Does This Fit? The Decision FrameworkUse Fabric Data Agents when your data lives in Fabric, the job is governed Q&A and conversational analytics, and the team building it is data-focused rather than AI/ML-focused. Time-to-value is hours, not weeks.Use Copilot Studio when you need external channels (Teams, web chat, WhatsApp), multi-turn orchestration, or a customer-facing bot. A Fabric Data Agent can be embedded as a capability inside a Copilot Studio bot — that combination covers most internal scenarios cleanly.Use Azure AI Foundry Agent Service when you need full control: custom tools, external APIs, specific models, multi-agent routing. You pay for that control in engineering effort.For most Malaysian enterprises I advise, the honest answer is: start with a Fabric Data Agent on your highest-value governed dataset, and only escalate to Foundry when you hit a requirement Data Agents genuinely can't express.Governance: The Part That Actually MattersThis is where Fabric Data Agents earn enterprise trust:Identity-enforced access. Every query executes under the asking user's Entra ID identity and workspace permissions. No service-principal loophole, no shared credential.Semantic model security carries through. Row-level and column-level security defined on Power BI models apply to agent answers. Model-level Read permission is sufficient — no workspace membership needed.Microsoft Purview integration. Purview DLP policies for Fabric Data Warehouse are GA and can restrict agent access to sensitive assets; access restriction policies for KQL and SQL databases are in preview. Agent prompts and responses can flow into Purview audit, eDiscovery, and retention.Azure AI Content Safety can be layered on to screen responses.Workspace outbound access protection (preview) governs what external endpoints agents may reach.For Malaysian banks and GLCs that must demonstrate data access controls to regulators, this audit chain — identity in, policy enforcement, logged prompts and responses out — is the feature that turns a pilot into a production approval.Limitations Nobody DemosI'll save you the discovery cycle:Answers are capped at 25 rows and 25 columns. Data Agents are built for conversational insight, not data export. If users need full datasets, that's a report or an API, not an agent.Strictly read-only. No pipeline triggers, no notebook runs, no writes. Plan around it rather than against it.Chat history influences follow-ups. A narrowed earlier answer constrains later ones; starting a fresh session is sometimes the correct debugging step.Five data sources per agent. Decompose by business domain.Cross-geo consent is mandatory outside EU/US. For Malaysian capacities, that's a compliance decision, not a checkbox.Purview policies can truncate or block answers. Expected behavior — but set user expectations, or they'll report "the agent is broken."What This Means for Malaysian EnterprisesMost enterprises here are data-rich but insight-poor. The bottleneck was never the warehouse — it was getting business users to use the data. SQL skills are scarce, report queues are long, and data teams are backlogged. Data Agents remove the query-language barrier while keeping governance intact, which is the combination Malaysian compliance environments actually permit.If you're already on Fabric, this is an incremental capability on capacity you already pay for. If you're evaluating Fabric for modernization, Data Agents strengthen the business case: you're not just building a data platform, you're building the interface people will actually talk to.Key TakeawaysFabric Data Agents are GA, read-only, and identity-enforced — conversational Q&A over up to five Fabric data sources, with queries executing under each user's own permissions. No keys, no RAG pipeline, no vector store.Semantic models and ontologies are the differentiator — NL2SQL/NL2DAX/NL2KQL over governed business definitions is what makes answers trustworthy instead of plausible.Governance is architectural, not procedural — Entra ID enforcement, RLS/CLS inheritance, and Purview DLP/audit give you an evidence chain your regulators can inspect.Know the constraints before promising anything — 25-row caps, five sources, read-only, and mandatory cross-geo consent for Malaysian capacities.Position it correctly — Data Agents for governed answers, Copilot Studio for channels and orchestration, Foundry Agent Service for full custom control. They compose; they don't compete. --- ## Azure AI Foundry Model Catalog: Pick the Right 10K+ Model Fast (2026 Guide) URL: https://wenfeng.my/azure-ai-foundry-model-catalog-enterprise/ Author: admin Published: 2026-08-15 23:30:39 Modified: 2026-10-04 15:32:46 Azure AI Foundry Model Catalog: Choosing From 10K+ Models for Enterprise WorkloadsMicrosoft's official documentation says the Foundry Models catalog now contains over 10,000 models, with roughly 50 new ones published every month. That is not a rounding error — it is the combined weight of Azure OpenAI, Anthropic, DeepSeek, Meta, Mistral, Cohere, xAI, Microsoft's own models, and a very long tail of Hugging Face community models, all sitting behind one catalog.For enterprises, this flips the problem. Two years ago the question was "can we even get access to a good model on Azure?" Today the question is "how do we evaluate 10,000 options without burning a quarter on experiments?" Model selection has quietly become an architecture discipline: the wrong choice costs you 20x on tokens, the wrong deployment type breaks your data residency story, and the wrong lifecycle choice puts a preview model under a production SLA.This article is the decision framework I use with clients, grounded in what the catalog actually contains today — verified against the Foundry portal, Microsoft Learn, and the Azure Retail Prices API.What Is Actually in the CatalogThe catalog is split into two tiers, and understanding the split is the first filtering decision:Foundry Models sold by Azure — Microsoft hosts, sells, and supports these under Microsoft Product Terms. They are billed through your Azure subscription, covered by Azure SLAs, and include all Azure OpenAI models plus selected models from providers like DeepSeek, Meta, Mistral, Cohere, and xAI. Some support fungible provisioned throughput, meaning your reserved capacity flexes across models.Foundry Models from partners and community — the vast majority of the catalog. Anthropic's Claude family and hundreds of Hugging Face models live here. The provider maintains the model; Azure provides the integration rails. Support and SLA terms vary by provider.The Foundry portal's publisher tab gives a sense of the curated lineup right now: Publisher Models in catalog (portal, Aug 2026) Microsoft 85 Azure OpenAI 67 Meta 39 Mistral AI 17 Anthropic 12 Cohere 7 xAI 7 DeepSeek 5 One honest observation: the portal's model browser currently enumerates around 4,000 models, while the "10,000+" figure includes the full Hugging Face hub collection available for managed compute deployment. Both numbers are true; they measure different depths of the same catalog. For most enterprise workloads, the curated few hundred are the realistic shopping list.Current flagship examples, straight from the portal: gpt-5.6-sol / terra / luna (the newest GPT-5.6 series), claude-opus-5 and claude-sonnet-5 (1M-token context, 128k output), DeepSeek-V4-Pro and V4-Flash, grok-4.1-fast-reasoning, Llama-4-Maverick-17B, Phi-4-reasoning-plus, Kimi K2.7-Code, MiniMax M2.5, and a model-router that dispatches across models for you — a signal of where the platform is heading.A Working Decision FrameworkStep 1: Start From the Task, Not the BrandThe catalog's filters are the right first move: inference task (chat completion, embeddings, image generation, speech, OCR), supported features (reasoning, tool calling), deployment options, region, and lifecycle (Preview / GA / Deprecated). There is also a performance leaderboard and side-by-side model comparison — useful starting points, not verdicts.Rough mapping I use: Workload Where to look first General chat / copilot GPT-5.x series, Claude Sonnet/Opus, DeepSeek-V4 Agentic / tool-calling Claude family, GPT-5.x, Grok fast variants High-volume cheap inference DeepSeek-V4-Flash, gpt-oss-120b, Phi-4, MiniMax Coding gpt-5.3-codex, Kimi K2.7-Code, Grok code variants Document processing / OCR Mistral OCR and Document AI, Cohere Command A RAG embeddings text-embedding-3-large/small, Cohere embeddings, Embed v4 Image / video / voice gpt-image-2, Flux, Sora 2, MAI-Voice, gpt-realtime Step 2: Choose the Tier by Operational AppetitePick sold by Azure when you need Microsoft support, an Azure SLA, and clean procurement. Pick partners and community when you need a specific capability fast — Anthropic models and the newest open models typically land here first. Note that partner models have billing constraints: pay-as-you-go access depends on your Azure subscription's billing account being in a country where the provider offers the model. CSP subscriptions can't purchase third-party model offers at all (a catalog-wide rule), and Anthropic additionally excludes student, free-trial, and credit-only accounts. That catches procurement teams off guard — check before you build.Step 3: Pick the Deployment Type — This Is Where Cost and Compliance Are DecidedThis is the step most teams skip, and it is the one that bites. Serverless API deployments come in several types, each with a SKU code you will see in the CLI and pricing pages: Deployment type Data processing Billing Use when Global Standard Any Azure region Pay-per-token, lowest price Default starting point Data Zone Standard US, EU, or APAC zone only Pay-per-token Regional compliance Standard (single region) Deployment region only Pay-per-token Strict in-country residency Provisioned (PTU) Per scope above Reserved capacity Predictable latency, high volume Batch Per scope above ~50% discount Async, non-time-sensitive jobs Developer Any Azure region Lowest cost Evaluating fine-tuned models, no SLA Two semantics matter enormously for compliance reviews. Data at rest always stays in the designated Azure geography — but inference processing depends on the type: Global Standard may process requests in any Azure region, Data Zone keeps processing inside the US/EU/APAC zone, and Standard stays in the deployment region. Microsoft's own guidance: start with Global Standard (it launches first, is cheapest, and has the broadest coverage), and move only when residency or throughput demands it. New deployment types arrive in a fixed order — Global first, Data Zone next, single-region last with no guaranteed date.There are also two non-deployment paths worth knowing. Instant access (preview) lets you call supported models by name with zero deployment — currently preview-only and limited to projects in West US 3, but excellent for evaluation. And managed compute (preview) hosts open-source and custom-weight models on dedicated GPUs — but Microsoft explicitly states this preview has no SLA and is not recommended for production. Plan around that.Step 4: Model the Real CostHere is verified retail pricing from the Azure Retail Prices API (Global Standard where noted, USD per 1M tokens, August 2026): Model Input Output gpt-5.6-sol (short context) $5.00 $30.00 gpt-5.4 $2.50 $15.00 gpt-5.4-mini $0.75 $4.50 Kimi K2.7-Code (Data Zone) $1.05 $4.40 DeepSeek-V4-Pro $1.74 $3.48 Mistral-Large-3 $0.50 $1.50 MiniMax M2.5 (Data Zone) $0.33 $1.32 gpt-oss-120b (Data Zone) $0.17 $0.66 grok-4.1-fast $0.20 $0.50 DeepSeek-V4-Flash $0.19 $0.51 Now run your actual workload through it. For a service doing 100M tokens per month at a 70/30 input/output split:On gpt-5.4: ~$625/monthOn gpt-5.4-mini: ~$188/monthOn DeepSeek-V4-Flash: ~$29/monthThat is a 20x+ spread for identical token volume. The point is not "buy the cheapest" — it is that the spread is large enough that selection must be driven by your quality evaluation, not by assumption. And for batch-tolerant workloads, Global Batch halves the price again. Total cost of ownership still includes your evaluation effort, fine-tuning data, and operational overhead — but token math is where most budgets leak.Deploying From the Catalog: The Actual CommandsForget older tutorials using az ml online-endpoint — that is the previous-generation Azure ML pattern. Foundry Models deployments use the core Azure CLI (2.60+, no extension needed):# Create a Foundry (AI services) resource az cognitiveservices account create \ -n $accountName -g $resourceGroupName \ --custom-domain $accountName --location eastus2 \ --kind AIServices --sku S0 # Discover which models and SKUs are available to you az cognitiveservices account list-models \ -n $accountName -g $resourceGroupName \ | jq '.[] | {name: .name, format: .format, version: .version, sku: .skus[0].name}' # Deploy a model (Global Standard) az cognitiveservices account deployment create \ -n $accountName -g $resourceGroupName \ --deployment-name Phi-4-mini-instruct \ --model-name Phi-4-mini-instruct \ --model-version 1 \ --model-format Microsoft \ --sku-capacity 1 \ --sku-name GlobalStandard # Verify az cognitiveservices account deployment show \ --deployment-name Phi-4-mini-instruct \ -n $accountName -g $resourceGroupName \ | jq '.properties.provisioningState' The same flow works in Bicep via az deployment group create against the AI services deployment template, which is how I deliver it for repeatable environments. Inference then goes through the unified endpoint: https://<resource>.openai.azure.com/openai/v1/, passing your deployment name as the model field.For instant access prototyping, the new Foundry SDK keeps it clean:from azure.identity import DefaultAzureCredential from azure.ai.projects import AIProjectClient project = AIProjectClient( endpoint=PROJECT_ENDPOINT, # https://<resource>.ai.azure.com/api/projects/<project> credential=DefaultAzureCredential(), ) openai = project.get_openai_client() response = openai.responses.create(model="gpt-5-mini", input="...") Authentication: use Microsoft Entra ID (keyless) over API keys. For RBAC, Microsoft's current guidance is to use the renamed Foundry roles (Foundry User for inference, Foundry Owner for creating deployments) rather than the older Cognitive Services* roles, and wire the endpoint into your VNet — private networking is supported on serverless deployments.The Malaysia West Reality CheckThis matters to me directly, so I checked it properly. The Azure Retail Prices API currently lists roughly 100 Foundry Models offers purchasable in Malaysia West — and every one of them is Global Standard. The good news: partner and Microsoft-owned models are there, including DeepSeek V4 Flash/Pro, Grok 4.x, Kimi K2.5–K2.7, Mistral Large 3, Llama 3.3/4 Maverick, gpt-oss-120b, Phi-4 family (including reasoning variants), Flux image models, OCR/Document AI, and embeddings.The honest gap: Azure OpenAI's GPT models show zero presence in Malaysia West in the pricing data. GPT-5.x deployments sit in regions like East US 2, Sweden Central, West Europe, Japan East, Australia East, and Singapore (Southeast Asia). If your workload must run on Azure OpenAI, Singapore is the practical neighbor for Malaysian teams.One correction worth flagging: MiniMax M2.5 is not purchasable in Malaysia West today (its Data Zone offers live in US/EU regions only), even though it appears in the catalog globally.The practical guidance for Malaysian enterprises:Global Standard from Malaysia West works for partner models — but remember inference may process in any Azure region. If your regulator asks "where was this prompt processed?", Global Standard's answer is "somewhere in Azure's global network."Data Zone Standard (APAC) is the compliance middle ground — processing stays inside the APAC zone, though you deploy from a region that supports the Data Zone SKU.Single-region Standard deployments are the strict answer, but they arrive last per model and have no guaranteed availability date. Do not promise them to a client without checking current region support.Self-hosting open-weight models (gpt-oss, Llama, Qwen from the community catalog) on your own GPU infrastructure remains the only option that keeps both storage and inference fully in-country — at the cost of owning the serving stack.Pitfalls I Keep SeeingBenchmark shopping. The catalog's leaderboard measures the vendor's benchmarks, not your documents, your language mix, or your tool-calling patterns. Shortlist from the leaderboard, decide on your own evaluation set.Preview models under production SLAs. Managed compute is preview with no SLA, several frontier models are Preview-labeled, and instant access is preview in one region. Lifecycle is a first-class filter in the catalog for a reason — use it.Ignoring deployment-type semantics. "We deployed in region X" is not a data residency statement if the SKU is Global Standard. Match the SKU to the compliance requirement, not the other way around.Hardcoding one model. The catalog churns — ~50 models a month in, and Azure OpenAI runs a published retirement schedule. Abstract the model behind a deployment name and config, and re-evaluate quarterly.Copying stale deployment tutorials. Anything built on az ml online-endpoint for model catalog work predates the current az cognitiveservices flow. Check the doc date before you trust the snippet.Key Takeaways10,000+ models makes selection an architecture decision — filter by task, tier, deployment type, region, and lifecycle before you ever open a playground.The deployment type decides cost and compliance, not just the model. Start Global Standard, escalate to Data Zone or single-region only when requirements demand it.Real pricing shows 20x+ spreads between frontier and economy models on identical token volume — evaluate on your workload, then do the token math.Malaysia West can buy ~100 Global Standard offers today but has no Azure OpenAI GPT presence; Singapore is the nearest Azure OpenAI region for Malaysian teams.Prototype with instant access, productionize with the CLI — and keep models behind deployment names so quarterly re-evaluation doesn't become a rewrite.Sources: Microsoft Learn — Foundry Models overview, Foundry Models sold by Azure, Models from partners and community, Deployment types, Instant access, Create model deployments (CLI/Bicep); Foundry portal model catalog (accessed August 2026); Azure Retail Prices API (August 2026). Related Reading Azure AI Foundry as the Unified Control Plane: Deploying, Evaluating, and Govern Azure Databricks + OneLake Unity Catalog Integration — The End of the Data Platf Copilot Studio vs Azure AI Foundry — The Enterprise Decision Framework for AI Ag --- ## GitHub Actions 2026 Security Roadmap — Hardening CI/CD Against Supply Chain Attacks URL: https://wenfeng.my/github-actions-2026-security-roadmap-hardening-cicd-against-supply-chain-attacks/ Author: admin Published: 2026-08-15 06:30:09 Modified: 2026-10-04 15:31:05 Earlier this year, a friend running a development team in Cyberjaya pinged me with a question that stuck: "We just got a security audit flag on our GitHub Actions workflows. They're saying we're exposed to supply chain attacks. What does that even mean for a CI/CD pipeline?"It means everything. And the timing is not coincidental — GitHub published its 2026 Actions security roadmap in March, and for the first time, CI/CD supply chain security is front and centre in their platform strategy. The message is clear: the way most teams build and deploy software today has a structural weakness, and the attackers have noticed.I've been advising Malaysian enterprises on cloud-native architecture and DevSecOps for years, and this is the shift I've been waiting for. GitHub is finally treating CI/CD pipelines not as convenience layers, but as critical infrastructure that needs the same security posture as production systems. Let me walk through what happened, what GitHub is building about it, and what you can do today — starting this week.The Problem: Your CI/CD Pipeline Is a Supply Chain Attack SurfaceHere's the uncomfortable truth that most development teams haven't internalised yet: every uses: directive in your GitHub Actions workflow is a trust decision. When you write uses: actions/checkout@v4, you are trusting not just the code in that action at this moment, but every future change to that version tag, every transitive dependency behind it, and the maintainer's ability to keep their own account secure.And the attackers know this.The GitHub roadmap announcement itself names the pattern directly, citing incidents targeting tj-actions/changed-files, Nx, and trivy-action. The tj-actions case is the one I use when I brief enterprise teams, because it is brutally instructive: in March 2025, attackers compromised a popular utility action referenced by more than 23,000 repositories. The malicious update injected code that dumped repository secrets straight into workflow logs — and every repository with public logs had those secrets exposed to anyone watching. Unit 42's assessment traced a targeted attack against Coinbase that expanded into this widespread incident, all through one compromised action.GitHub describes the consistent attacker playbook across these incidents:Vulnerabilities allow untrusted code execution inside workflows.Malicious workflows run without observability or control.Compromised dependencies spread across thousands of repositories.Over-permissioned credentials get exfiltrated via unrestricted network access.The mechanics come down to one word: mutability. Version tags like @v4 are pointers that maintainers can re-point at any time. When a tag is retagged — legitimately or maliciously — every workflow referencing it executes the new code on its very next run. No pull request, no review, no approval. A compromised dependency propagates across your entire fleet of workflows instantly.The attack chain that keeps me up at night goes like this: attacker compromises a popular action maintainer's account, pushes a malicious update behind a familiar version tag, the update silently exfiltrates secrets and injects backdoors into build artifacts, and your production deployment ships the compromised output. All within the window between the tag update and your next workflow run. No firewall catches this. No SAST tool flags it. The damage is done before anyone notices.This isn't theoretical. It happened to 23,000+ repositories with a single action. It will happen again — which is exactly why GitHub published this roadmap.GitHub's 2026 Security Roadmap: What's Actually ComingThe roadmap, published by GitHub's product security and product management teams in late March 2026, organises the response into three layers: securing the ecosystem (deterministic dependencies), reducing the attack surface (policies, secure defaults, scoped credentials), and hardening the infrastructure (observability and network boundaries for runners). Notably, GitHub is explicit that this is not a rearchitecture of Actions — it's a shift toward making secure behaviour the default.Layer 1: A More Secure Ecosystem — Deterministic DependenciesToday, action dependencies are resolved at runtime from mutable references. What runs in CI isn't always fixed or auditable, and GitHub admits that while immutable commit SHAs help, they're hard to manage at scale and transitive dependencies stay opaque.The fix is workflow-level dependency locking: a dependencies: section in workflow YAML that locks all direct and transitive dependencies to commit SHAs. GitHub's analogy is Go's go.mod + go.sum, but for workflows. You'll be able to resolve dependencies via the GitHub CLI, commit the generated lock data into your repository, and update by re-running resolution and reviewing the diffs.What this changes in practice:Deterministic runs — every workflow executes exactly what was reviewed.Reviewable updates — dependency changes show up as diffs in pull requests.Fail-fast verification — hash mismatches stop execution before jobs run.Full visibility — composite actions no longer hide nested dependencies.On the publishing side, GitHub is moving toward immutable releases with stricter release requirements, creating a central enforcement point for detecting and blocking malicious code before it enters the ecosystem. The public-preview (3–6 months) and GA (~6 months) targets apply to the dependency lockfiles; GitHub has not published dates for immutable releases yet.Layer 2: Reducing the Attack Surface — Policy-Driven Execution and Scoped SecretsThis layer addresses what GitHub calls the gap between flexibility and security: workflows can run in response to many events, triggered by various actors, with varying permissions — and attacks like Pwn Requests show how subtle differences in triggers, permissions, and execution contexts get abused.Workflow execution protections are built on GitHub's ruleset framework. Instead of reasoning about security across individual YAML files, administrators define central policies controlling who can trigger workflows (actor rules) and which events are allowed (event rules). For example, an organisation can restrict workflow_dispatch to maintainers, or prohibit pull_request_target entirely so workflows triggered by external contributions run without access to repository secrets. Critically, rules support evaluate mode — policies report what they would have blocked before you turn enforcement on, so you can adopt safely.Scoped secrets fix the credential side. Today, secrets are scoped at repository or organisation level and flow broadly by default — especially through reusable workflows, where implicit secret inheritance blurs trust boundaries. Scoped secrets bind credentials to explicit execution contexts: specific repositories or organisations, branches or environments, workflow identities or paths, and trusted reusable workflows. Modified or unexpected workflows simply won't receive credentials. GitHub is also separating code contribution from credential management: write access to a repository will no longer grant secret management permissions, moving toward least privilege by default.Most of this lands in public preview within 3–6 months and GA around 6 months.Layer 3: Endpoint Monitoring and Control for CI/CD InfrastructureThe third layer treats runners as what they are: systems that execute untrusted code, handle sensitive credentials, and interact with external services — with historically limited visibility and controls.Actions Data Stream delivers near real-time execution telemetry to your existing systems — Amazon S3 or Azure Event Hub / Data Explorer — with at-least-once delivery guarantees and a common schema. You get workflow and job execution details across repositories and organisations, dependency resolution and action usage patterns, and eventually network activity and policy enforcement outcomes. CI/CD becomes observable like any other production system.The native egress firewall is, in my view, the most important item on the entire roadmap. GitHub-hosted runners currently allow unrestricted outbound network access — which is exactly how exfiltration happens in attacks like tj-actions. The firewall operates outside the runner VM at Layer 7, so it remains immutable even if an attacker gains root inside the runner environment. Organisations define precise egress policies: allowed domains and IP ranges, permitted HTTP methods, TLS and protocol requirements. The adoption path is sane: monitor all outbound traffic first (every request audited and correlated to the workflow run, job, step, and initiating command), build allowlists from real data, then enforce.Actions Data Stream targets public preview in 3–6 months and GA in 6–9 months; the egress firewall targets public preview in 6–9 months.Hardening Your Pipelines: A Practical GuideThe roadmap features are months away. Here's what a hardened GitHub Actions setup looks like today — four concrete changes you can make this week, each with real YAML you can copy and adapt.1. Pin Actions to Full SHA HashesThis is the single most impactful change you can make right now. Instead of referencing actions by mutable version tags, pin them to specific commit hashes. # BEFORE — vulnerable to tag mutation attacks - uses: actions/checkout@v4 - uses: actions/setup-node@v4 - uses: docker/build-push-action@v5 # AFTER — pinned to immutable commit hashes - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1 - uses: actions/setup-node@60edb5dd545a775178f52524783378180af0d1f8 # v4.0.2 - uses: docker/build-push-action@2cdde995de11925a030ce8070c3d77a52ffcf1c0 # v5.3.0 The comment with the version tag matters — it tells future maintainers which version the hash corresponds to. Tools like pin-github-action and GitHub's dependency review can automate this. Yes, SHA pinning is manual overhead at scale — that's precisely the problem the upcoming dependencies: lockfile solves. Until it ships, pinning is your best defence.2. Switch to OIDC for Cloud DeploymentsStop storing cloud credentials as GitHub secrets. Use OpenID Connect (OIDC) to give your workflows short-lived, identity-based access to Azure, AWS, or GCP. # OIDC authentication to Azure — no stored secrets required permissions: id-token: write contents: read jobs: deploy: runs-on: ubuntu-latest environment: production steps: - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 - name: Azure Login via OIDC uses: azure/login@a65d910e8af852a8061c627c456678983e180302 # v2.2.0 with: client-id: ${{ vars.AZURE_CLIENT_ID }} tenant-id: ${{ vars.AZURE_TENANT_ID }} subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }} - name: Deploy to Azure run: | az webapp deploy \ --resource-group rg-production \ --name myapp-prod \ --src-path ./dist/app.zip \ --type zip The key differences: id-token: write permission enables OIDC token generation, the client-id and tenant-id are stored as repository variables (not secrets), and the Azure service principal is configured to trust GitHub's OIDC issuer. No secrets to rotate, no credentials to leak. Even if a workflow is compromised, the attacker gets a short-lived token bound to your repository and branch context — not a permanent credential.3. Sign Your Build Artifacts with SigstoreSigstore integration in GitHub Actions is mature enough for production use. Here's how to sign your build outputs so downstream consumers can verify they came from your pipeline. jobs: build-and-sign: runs-on: ubuntu-latest permissions: contents: read id-token: write # required for Sigstore OIDC identity packages: write # for GHCR publishing steps: - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 - name: Build container image run: | docker build -t ghcr.io/myorg/myapp:${{ github.sha }} . - name: Login to GHCR uses: docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567 # v3.3.0 with: registry: ghcr.io username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - name: Push image run: docker push ghcr.io/myorg/myapp:${{ github.sha }} - name: Install Cosign uses: sigstore/cosign-installer@dc72c7d5c4d10cd6bcb8cf6e3fd625a9e5e537da # v3.7.0 - name: Sign the container image run: | cosign sign --yes ghcr.io/myorg/myapp:${{ github.sha }} Now any consumer of your image can verify its provenance: # Verify the signature cosign verify ghcr.io/myorg/myapp:<commit-sha> \ --certificate-identity-regexp="https://github.com/myorg/myapp" \ --certificate-oidc-issuer="https://token.actions.githubusercontent.com" # Verify SLSA provenance — confirm the build came from your repo's workflow cosign verify-attestation \ --type slsaprovenance \ ghcr.io/myorg/myapp:<commit-sha> \ --certificate-identity-regexp="https://github.com/myorg/myapp" \ --certificate-oidc-issuer="https://token.actions.githubusercontent.com" 4. A Reusable Workflow Template for Malaysian Dev TeamsHere's a reusable workflow template that combines all three hardening measures above, plus environment-based deployment controls. I've designed this for teams deploying Azure-hosted applications from GitHub — a pattern I see frequently across Malaysian enterprises. # .github/workflows/deploy-hardened.yml name: Hardened Build & Deploy on: push: branches: [main] pull_request: branches: [main] permissions: contents: read id-token: write packages: write security-events: write jobs: security-scan: name: Security Scan runs-on: ubuntu-latest steps: - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 - name: Run Trivy vulnerability scanner uses: aquasecurity/trivy-action@6e7b7d1fd3e4fef0c5fa8cce1229c54b2c9bd0d8 # v0.24.0 with: scan-type: 'fs' scan-ref: '.' format: 'sarif' output: 'trivy-results.sarif' - name: Upload to GitHub Security tab uses: github/codeql-action/upload-sarif@v3 # first-party; pin to a release SHA in high-security environments with: sarif_file: 'trivy-results.sarif' build: name: Build & Sign runs-on: ubuntu-latest needs: security-scan outputs: image-digest: ${{ steps.build.outputs.digest }} steps: - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 - name: Set up Docker Buildx uses: docker/setup-buildx-action@988b5a0280414f521da01fcc63a27aeeb4b104db # v3.6.1 - name: Login to GHCR uses: docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567 with: registry: ghcr.io username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - name: Build and push id: build uses: docker/build-push-action@2cdde995de11925a030ce8070c3d77a52ffcf1c0 with: push: true tags: | ghcr.io/${{ github.repository }}:${{ github.sha }} ghcr.io/${{ github.repository }}:latest cache-from: type=gha cache-to: type=gha,mode=max - name: Install Cosign uses: sigstore/cosign-installer@dc72c7d5c4d10cd6bcb8cf6e3fd625a9e5e537da - name: Sign container image run: | cosign sign --yes \ ghcr.io/${{ github.repository }}@${{ steps.build.outputs.digest }} deploy-staging: name: Deploy to Staging runs-on: ubuntu-latest needs: build if: github.event_name == 'push' && github.ref == 'refs/heads/main' environment: staging steps: - name: Azure Login (OIDC) uses: azure/login@a65d910e8af852a8061c627c456678983e180302 with: client-id: ${{ vars.AZURE_CLIENT_ID }} tenant-id: ${{ vars.AZURE_TENANT_ID }} subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }} - name: Deploy to Staging run: | az webapp config container set \ --resource-group ${{ vars.RESOURCE_GROUP }} \ --name ${{ vars.APP_NAME }} \ --container-image ghcr.io/${{ github.repository }}@${{ needs.build.outputs.image-digest }} deploy-production: name: Deploy to Production runs-on: ubuntu-latest needs: [build, deploy-staging] if: github.event_name == 'push' && github.ref == 'refs/heads/main' environment: production # requires manual approval in GitHub steps: - name: Azure Login (OIDC) uses: azure/login@a65d910e8af852a8061c627c456678983e180302 with: client-id: ${{ vars.AZURE_CLIENT_ID }} tenant-id: ${{ vars.AZURE_TENANT_ID }} subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }} - name: Deploy to Production run: | az webapp config container set \ --resource-group ${{ vars.RESOURCE_GROUP }} \ --name ${{ vars.APP_NAME }} \ --container-image ghcr.io/${{ github.repository }}@${{ needs.build.outputs.image-digest }} This template gives you: SHA-pinned actions across the board, OIDC-based Azure authentication with no stored secrets, Sigstore image signing, Trivy security scanning with results uploaded to GitHub's security dashboard, and environment-gated production deployments with required reviewers.Common Pitfalls I See in the WildHaving reviewed GitHub Actions setups across Malaysian enterprises, here are the mistakes that keep recurring:1. Over-permissioned GITHUB_TOKEN. The number one issue. Teams add permissions: write-all at the workflow level because "something wasn't working." Every workflow should start with minimal permissions and elevate only what's needed per job. GitHub's secure-defaults push will help new repositories, but if you migrated from an older repository, your existing workflows still carry the old permissive behaviour.2. Treating GitHub Secrets as secure when they're not. Secrets in GitHub are encrypted at rest, but they're decrypted and available as plaintext environment variables during workflow execution. Any step — including a compromised action — can read them. This is why OIDC matters: there's no standing secret to steal if you're using identity-based authentication.3. Not pinning internal actions. Teams often pin third-party actions to SHA hashes but forget their own internal actions. Internal actions need the same treatment — a compromised developer account can push a malicious update to your own action just as easily as to a public one.4. Assuming the runner can't phone home. GitHub-hosted runners allow unrestricted outbound network access today. A compromised step can POST your secrets to any endpoint on the internet, and nothing stops it. Until the roadmap's native egress firewall ships, assume exfiltration is possible and minimise what's worth stealing: OIDC tokens instead of long-lived credentials, scoped secrets, short retention for logs and artifacts that may contain sensitive output.5. Skipping the security scan step. The hardened template above includes Trivy scanning because I've seen too many teams deploy container images with critical CVEs. The scan takes seconds and catches the obvious stuff. There's no excuse for skipping it.What This Means for Malaysian Dev TeamsIf you're building software for Malaysian enterprises — especially in financial services, healthcare, or government — these changes are not optional. PDPA compliance increasingly extends to the security of your development toolchain, and auditors are starting to ask questions about CI/CD integrity. If you stream Actions telemetry via the upcoming Data Stream into Azure Event Hub and Data Explorer, you get an audit trail your compliance team will actually thank you for.The good news is that the tooling has caught up. GitHub's 2026 roadmap, combined with Sigstore and OIDC, gives you a path to a complete hardened pipeline without requiring enterprise security budgets. The reusable template above is a starting point — adapt it to your team's stack and your organisation's compliance requirements.Start with the three highest-impact changes: pin your actions to SHA hashes, switch to OIDC for cloud deployments, and set minimum permissions on your GITHUB_TOKEN. These three changes alone eliminate the most common attack vectors — the same vectors that made tj-actions so damaging.The broader lesson is this: CI/CD pipelines are no longer plumbing. They're critical infrastructure that handles your source code, your secrets, and your deployment credentials. Treat them with the same security posture you'd apply to your production systems — because that's exactly what they are.Key TakeawaysPin every action to a full SHA hash, not a version tag. Mutable tags like @v4 can be re-pointed by a compromised maintainer — that's how tj-actions hit 23,000+ repositories. A SHA hash is immutable and verifiable. This is the single most effective defence available today, and GitHub's upcoming dependencies: lockfile will make it manageable at scale.Replace stored secrets with OIDC authentication wherever possible. Short-lived identity tokens from GitHub's OIDC provider eliminate the standing-credential leakage vector entirely. Azure, AWS, and GCP all support OIDC federation with GitHub — use it for every cloud deployment.Sign your build artifacts with Sigstore and generate SLSA provenance. Artifact attestation lets downstream consumers verify that your builds came from your pipeline, were produced from your source code, and weren't tampered with. It's the supply chain integrity layer that catches what vulnerability scanning misses.Enforce minimum permissions now, and prepare for policy-driven execution. Start with read-only GITHUB_TOKEN and elevate per job. When GitHub's ruleset-based execution protections and scoped secrets land, you'll be able to centralise actor rules, event rules, and credential scoping at the organisation level — plan for that migration now.Treat your CI/CD pipeline as production infrastructure. It has access to your source code, your secrets, and your deployment credentials. Apply the same security principles: least privilege, audit logging, network boundaries, and artifact signing. The gap between "works" and "secure" in CI/CD is where supply chain attacks live. --- ## Copilot Studio vs Azure AI Foundry — The Enterprise Decision Framework for AI Agents in 2026 URL: https://wenfeng.my/copilot-studio-vs-azure-ai-foundry-the-enterprise-decision-framework-for-ai-agents-in-2026/ Author: admin Published: 2026-08-15 02:00:00 Modified: 2026-10-04 15:28:32 If you've been watching the Microsoft AI landscape this year, you've noticed something interesting: the company that once consolidated everything under a single "Copilot" brand is now offering two fundamentally different paths for building AI agents. Copilot Studio and Azure AI Foundry serve overlapping but distinct audiences, and the choice between them — or the decision to use both — has real consequences for cost, capability, and time-to-production.I've been building multi-agent systems on Azure for the past year, including running my own content pipeline with Hermes and OpenClaw agents. Through that hands-on experience and my work with Malaysian enterprises, I've developed a clear mental model for when each platform makes sense. This article is the framework I wish I'd had at the start.Microsoft's 6 Core Capabilities: The Agent Maturity RoadmapBefore diving into the platform comparison, it's worth understanding the foundation Microsoft has laid. Their "6 core capabilities to scale agent adoption in 2026" framework defines what mature agent systems actually require — and both Copilot Studio and Azure AI Foundry address these capabilities from different angles.Capability 1: Anyone Can Turn Intent Into Agents. Building agents now starts with natural language, not code. In Copilot Studio, a sales operations manager can describe what they want — "monitor pipeline changes, flag at-risk deals, notify owners" — and the agent interprets intent through its underlying model. In Azure AI Foundry, developers achieve the same outcome through Prompt Flow with a visual graph editor that still requires understanding of LLM orchestration.Capability 2: Agents That Own Workflows End to End. Agents have evolved from suggesting actions to executing entire processes. Copilot Studio's agent flows handle automated approvals, escalations, and multi-step business processes. Azure AI Foundry achieves this through custom orchestration pipelines that developers build with the SDK.Capability 3: Coordinating Multiple Agents for Real Outcomes. This is where the Agent2Agent (A2A) protocol becomes critical. The A2A protocol — originally developed by Google and now stewarded by the Linux Foundation — enables agents built on different platforms to communicate, delegate, and collaborate. Both Copilot Studio and Azure AI Foundry support A2A — but with different implications for who controls the coordination layer.Capability 4: Flexibility to Control Your Agent Models. Organisations can now select the optimal AI model for each specific task. Copilot Studio offers model selection, including Anthropic, Mistral, and xAI models plus Azure AI Foundry models. Azure AI Foundry provides direct access to a catalog of thousands of models with fine-grained control over deployment, quantization, and inference parameters.Capability 5: Agents That Act Across Your Systems. Two key technologies power this capability: Model Context Protocol (MCP) for simplified external system integration, and Computer Use agents that navigate interfaces like a human. Copilot Studio supports both — MCP for connecting to external tools and data sources, and Computer Use, generally available since May 2026, for automating desktop and web applications without APIs.Capability 6: Scaling Agents Without Sacrificing Control. Governance, lifecycle management, and evaluation built into the platform. Microsoft Agent 365 provides a unified view across Copilot Studio and M365 Copilot, while Azure AI Foundry integrates with Azure Policy, Entra ID, and Microsoft Defender for enterprise-grade governance.Both platforms cover all six capabilities. The difference is who does the work and how much control you retain.Copilot Studio: The Low-Code Agent BuilderCopilot Studio is Microsoft's answer to the question: "Can business users build AI agents without writing code?" The answer is increasingly yes — with caveats.What Copilot Studio Does WellThe authoring experience is genuinely low-code. You describe what you want in natural language, configure topics through a visual canvas, and connect to data sources and actions through pre-built connectors. A marketing team lead can build an agent that monitors campaign performance and suggests reallocations without involving a single developer.The A2A protocol support means a Copilot Studio agent can coordinate with agents built on other platforms. MCP integration simplifies connecting to external systems — instead of writing custom API integrations, you point the agent at MCP-compatible tools and it discovers available operations automatically. Computer Use agents navigate interfaces like a human — no API required. For Malaysia's enterprise landscape where plenty of applications lack APIs, this transforms RPA-like workflows without brittle selector-based approaches.Copilot Studio Pricing: The Credit ModelCopilot Studio switched from messages to Copilot Credits in September 2025. Prepaid packs run $200/month for 25,000 credits ($0.008/credit), with pay-as-you-go at $0.01/credit. Here's the key nuance: one credit buys a scripted FAQ answer, but a complex reasoning-model response costs 100 credits — a 100x cost variation based purely on agent design.A generative answer costs 2 credits; adding tenant graph grounding pushes it to 12 credits; premium reasoning exceeds 112 credits per interaction. For a mid-size Malaysian enterprise running five agents across departments, a realistic planning range is $3,000–6,000/month in credits alone, depending on how heavily the agents lean on premium reasoning.The critical pricing advantage: internal agents used by Microsoft 365 Copilot licensed users consume zero credits. For a 200-user enterprise paying Microsoft 365 Copilot's $30/user/month list price, the internal agent interactions are included. Only external-facing agents require credit purchases.The LimitationsCopilot Studio's low-code nature is simultaneously its strength and constraint. You're working within the platform's abstraction layer — fine-tuning model weights, custom inference pipelines, or advanced RAG architectures aren't within scope. For straightforward business process automation, it's excellent. For complex, performance-critical AI systems, you'll outgrow it.Azure AI Foundry: The Developer's AI PlatformAzure AI Foundry — now branded Microsoft Foundry — is what happens when you give developers the full AI stack without abstraction layers. Where Copilot Studio simplifies, Foundry exposes.What Azure AI Foundry Does WellThe model catalog is staggering: 10,000+ models from providers including OpenAI, Anthropic, Mistral AI, Meta, DeepSeek, Cohere, and NVIDIA. Need GPT-5.4 for complex reasoning, DeepSeek R1 for cost-efficient batch processing, and a Phi model for edge deployment? All deployable through a single platform.Custom model fine-tuning is first-class — hosted pipelines where you bring your data and the platform handles infrastructure, training, evaluation, and deployment. For a Malaysian bank needing a model fine-tuned on local regulatory documents, this gives you full control.Enterprise governance is built into the foundation, not bolted on. Azure AI Foundry integrates with Azure Policy for resource governance, Entra ID for identity, and Microsoft Defender for threat protection. Automated evaluations cover risk and safety, performance quality, and custom metrics you define.Prompt Flow provides a visual orchestration tool for building complex AI workflows. Think of it as a developer-grade visual programming environment for agent pipelines. You can build RAG chains, multi-step reasoning workflows, and tool-use agents with full visibility into each step.Azure AI Foundry Pricing: The Consumption ModelAzure AI Foundry pricing is consumption-based and model-specific. Azure OpenAI models follow per-token pricing (GPT-5.4 Global at $2.50/1M input, $15.00/1M output), while third-party models like DeepSeek R1 can be significantly cheaper. Fine-tuning incurs compute costs for training, hosting, and inference. Agent service adds token consumption plus knowledge connections and action tool services.For a mid-size Malaysian enterprise running 50M tokens/month on GPT-5.4 — mostly input tokens with a typical output share — expect roughly $250–750/month in inference alone, plus Azure AI Search for RAG ($200–600/month). The Provisioned Throughput Units (PTUs) model becomes attractive at scale — reserving capacity at fixed rates, with monthly and annual reservations discounting the per-PTU price for sustained high-volume workloads.The Trade-offAzure AI Foundry requires developers. Not business analysts who happen to be technical — actual developers who understand ML concepts, can debug prompt engineering issues, and can navigate Azure resource management. The platform's power comes with operational complexity. You're managing deployments, monitoring inference costs, handling model versioning, and maintaining fine-tuning pipelines.The Decision Matrix: Choosing for Malaysian EnterprisesAfter working with dozens of Malaysian organisations on AI agent adoption, here's the practical framework.Choose Copilot Studio When:Your primary builders are business analysts, not developers. The natural language authoring model works best when business users own the agent lifecycle.You're automating Microsoft-centric workflows. SharePoint document processing, Teams-based approvals, Outlook-driven escalations — Copilot Studio's M365 integration is unmatched.Speed to production matters more than customisation. A functional agent in days, not weeks.Internal consumption dominates. If most agents serve licensed M365 users, the zero-rated internal usage makes the cost model very attractive.Choose Azure AI Foundry When:You need custom model fine-tuning. Domain-specific models trained on proprietary data — Copilot Studio can reference Foundry models but can't fine-tune them.Performance optimization is critical. Speculative decoding, quantization, custom inference pipelines — Foundry gives you the full stack.Data sovereignty requires self-hosted models. Some Malaysian government and financial sectors mandate on-premise model hosting.Cost optimization at scale matters. Token-level cost control, model routing for cost vs. quality trade-offs, and batch processing at 50% discount.The Hybrid Approach (Increasingly Common)Most mature Malaysian enterprises end up using both platforms: Copilot Studio for business-owned agents that automate Microsoft-centric workflows, Azure AI Foundry for developer-built agents requiring custom models or complex orchestration, connected through the A2A protocol. This isn't a cop-out — it maps to how organisations actually operate.Practical Deployment: A Side-by-Side ExampleLet me walk through building the same capability — an internal IT support agent — on both platforms.Copilot Studio Approach # Prerequisites # 1. M365 Copilot license assigned # 2. Copilot Studio author role in Power Platform admin center # 3. Copilot Credit pack purchased ($200/month for 25K credits) # Internal users interacting with this agent = ZERO credit cost # Only external users consume credits In the Copilot Studio canvas, you'd create a topic for IT support, connect it to your SharePoint knowledge base through the built-in connector, configure generative answers with data grounding (10 credits per grounded response for external users, zero for internal), and optionally add agent actions that trigger Azure Logic Apps for ticket creation.Total setup time for a functional agent: 2-4 hours for a business analyst.Azure AI Foundry Approach # Create the Foundry resource (AI Services account) — az cognitiveservices is the # CLI surface for Foundry Models deployments (core CLI, no extension required) az cognitiveservices account create \ --name "it-support-agent" \ --resource-group "rg-ai-agents" \ --location "southeastasia" \ --kind AIServices \ --sku S0 \ --custom-domain "it-support-agent" # Deploy a model endpoint (GPT-4.1 mini for cost efficiency) az cognitiveservices account deployment create \ --name "it-support-agent" \ --resource-group "rg-ai-agents" \ --deployment-name "it-support-gpt41mini" \ --model-name "gpt-4.1-mini" \ --model-version "2025-04-14" \ --model-format OpenAI \ --sku-name "GlobalStandard" \ --sku-capacity 10 # Create an Azure AI Search service for RAG # (az search covers service and key management; index creation uses the # Azure AI Search REST API, not the CLI) az search service create \ --name "it-knowledge-search" \ --resource-group "rg-ai-agents" \ --location "southeastasia" \ --sku "standard" Then create the index and index your IT documentation through the Azure AI Search REST API (POST /indexes on the search service endpoint with your admin api-key).You'd then build a Prompt Flow graph connecting the search index to the model, add content safety filters, configure evaluation metrics, and deploy with Azure Policy governance. Total setup time: 1-2 weeks for a development team.The difference isn't just time — it's control. The Foundry approach gives you full visibility into each component, the ability to swap models independently, and the infrastructure to scale with PTU reservations. The Copilot Studio approach gets you to production faster with less operational overhead.Common PitfallsPitfall 1: Underestimating Copilot Studio credit consumption. A single reasoning-model response costs 100 credits. If you design an agent that routes every query through premium reasoning, your $200 credit pack evaporates in 250 interactions. Design for scripted responses first, escalate to generative only when needed.Pitfall 2: Over-engineering with Azure AI Foundry. Not every agent needs custom fine-tuning and prompt flow orchestration. If a Copilot Studio agent with data grounding solves the problem, the Foundry approach adds complexity without proportional value.Pitfall 3: Ignoring the A2A protocol. As you deploy more agents across platforms, the coordination problem grows exponentially. Design your agent architecture with A2A in mind from day one, even if you start with a single platform.Pitfall 4: Forgetting the governance layer. Both platforms offer governance tools — Agent evaluations in Copilot Studio, Azure Policy and automated evaluations in Foundry. Use them. In practice, governance gaps are a leading cause of production agent rollbacks.Pitfall 5: Not accounting for the full cost stack. Copilot Studio's credit cost is just one layer. Add M365 Copilot licenses, Azure service costs for connectors and Logic Apps, and Azure OpenAI charges for custom model usage. Azure AI Foundry's per-token cost is just one layer. Add Azure AI Search, compute, storage, and monitoring. Get the total cost picture before committing.ConclusionThe Copilot Studio vs Azure AI Foundry decision isn't about which is "better" — it's about which serves your specific use case, team capabilities, and cost constraints. Microsoft designed these platforms for different audiences within the same enterprise. The 6 core capabilities framework ensures both paths converge on the same maturity outcomes: agents that anyone can build, that own workflows end-to-end, that coordinate across systems, with flexible model control, broad system integration, and scalable governance.For Malaysian enterprises in 2026, the question isn't whether to adopt AI agents — the data makes that inevitable. The question is how to build the right agent architecture for your organisation's unique constraints. Start with the decision matrix above, pilot both platforms if your scale justifies it, and connect them through A2A when you're ready to orchestrate.The organisations that get this right won't just save time — they'll fundamentally reshape how work gets done.Key TakeawaysCopilot Studio and Azure AI Foundry serve different builder personas for the same enterprise. Business analysts own Copilot Studio agents; developers own Foundry agents. Use A2A protocol to coordinate between them rather than forcing a single platform choice.Copilot Studio's cost model is credit-based and deceptively variable. Internal M365 Copilot users are zero-rated, but external interactions range from $0.008 (scripted) to $1.00+ (reasoning) per interaction. Design for scripted responses first; reserve premium capabilities for high-value tasks.Azure AI Foundry's strength is control at scale. 10,000+ models, custom fine-tuning, PTU reservations for cost predictability, and full Azure governance integration. It requires developers but rewards them with a complete AI platform.The 2026 decision framework isn't either/or. The hybrid approach — Copilot Studio for business-owned workflows, Azure AI Foundry for developer-built custom agents — is the architecture that maps to how Malaysian enterprises actually operate.Governance is not optional. Microsoft's 6 core capabilities include governance as a foundational pillar. Both platforms offer the tools; the gap is adoption. Deploy agent evaluations, cost monitoring, and access controls from day one, not after your first production incident. --- ## Azure BCDR Strategy 2026 — Building a Resilient Recovery Plan for Malaysian Enterprises URL: https://wenfeng.my/azure-bcdr-strategy-2026-resilient-recovery-plan-malaysian-enterprises/ Author: admin Published: 2026-08-14 23:30:44 Modified: 2026-10-04 15:28:43 I have a confession to make. Early in my cloud architecture career, I helped a large Malaysian enterprise write a beautifully documented disaster recovery plan. Forty-seven pages. Colour-coded runbooks. RTO and RPO targets aligned to business tiers. It sat in a shared drive, got reviewed annually, and won executive approval. Then an actual regional incident hit, and we discovered the runbook referenced a virtual network that no longer existed, a backup vault in the wrong region, and an automation script that had been deprecated six months earlier. The plan on paper was flawless. The reality was terrifying.If you are reading this from a Malaysian enterprise running workloads on Azure, chances are your DR plan has the same gap. You have documentation. You might even have Azure Site Recovery configured. But when was the last time anyone actually tested the full recovery flow? Practitioners consistently report that a large share of enterprises have not validated their DR plans within the last twelve months — and the gap between documented DR and tested DR is where real risk lives.The good news: 2026 is genuinely different. Microsoft rebranded Azure Business Continuity Center into Resiliency in Azure, adding zone down drills, recovery orchestration plans, and resiliency recommendations under one roof. This article walks through a practical BCDR strategy using Azure-native services — including the one Malaysia West gotcha that breaks most textbook DR designs.Start with RPO/RTO Tiers, Not ToolsBefore touching the Azure portal, define recovery objectives. I use a three-tier model that maps business criticality to acceptable data loss and downtime: TierBusiness ImpactRPORTOExample Workloads Tier 1 — Mission CriticalRevenue loss, regulatory breach0–15 min< 1 hourCore banking, payment processing Tier 2 — Business ImportantProductivity loss, SLA breach1–4 hours4–8 hoursERP, CRM, internal applications Tier 3 — OperationalDelayed operations, acceptable downtime24 hours24–72 hoursDev/test, analytics, archival Regulatory context matters here. PDPA 2010 does not prescribe specific RPO/RTO numbers, but Bank Negara Malaysia's RMiT (Risk Management in Technology) guidelines and the Securities Commission Malaysia's equivalent expect licensed institutions to maintain documented, tested business continuity capabilities. Even if your regulator stays silent, your cyber insurance provider probably will not.Map every workload to a tier. This drives every downstream decision: backup frequency, replication strategy, and budget.The Malaysia West Gotcha: No Paired Region, No Cross-Region RestoreHere is the fact that invalidates most generic DR guidance for Malaysia: Malaysia West is a non-paired region.Most Azure regions come in pairs (Southeast Asia pairs with East Asia, for example). Geo-redundant storage replicates automatically to the paired region, and Azure Backup's Cross-Region Restore (CRR) lets you restore from that secondary region when the primary is down. That entire safety net assumes pairing exists.For Malaysia West, it does not. The practical consequences:Cross-Region Restore is not available for vaults in Malaysia West — CRR restores only into the paired region, and there is none.You cannot rely on "Azure handles the second region for me." Regional resilience must be designed explicitly.Malaysia West does have multiple Availability Zones, so zonal resilience is fully on the table — and it is where Microsoft's new tooling concentrates in 2026.The correct strategy for a Malaysian enterprise in 2026:Zonal first. Use Availability Zones within Malaysia West for Tier 1 and Tier 2 workloads. Zone-redundant deployments survive single-AZ failures without a second region.Explicit regional DR where the business case demands it. Replicate to Southeast Asia (Singapore) via Azure Site Recovery, or stand up an independent Recovery Services vault there for backup copies of crown-jewel data.Plan for a future second region. Microsoft has not announced a second Malaysian region yet. If one reaches GA in the future, it becomes the natural regional DR target — but until then, design for Singapore.Recovery Services Vaults: Your First Line of DefenceAzure Backup lives in Recovery Services vaults, and 2026's Resiliency in Azure experience manages them at scale — vault security levels, immutability, soft delete, and multi-user authorisation, all in one pane.Provisioning is simple with the Azure CLI: # Primary vault — Malaysia West az backup vault create \ --resource-group rg-prod-bcdr \ --name rsv-malaysia-west-primary \ --location malaysiawest \ --immutability-state Disabled # Independent regional DR vault — Southeast Asia (Singapore) az backup vault create \ --resource-group rg-prod-bcdr-dr \ --name rsv-southeastasia-secondary \ --location southeastasia Three hardening steps I insist on for every vault, because ransomware increasingly targets backups themselves: # 1. Zone-redundant backup storage (protects vault data against single-AZ loss) az backup vault update \ --resource-group rg-prod-bcdr \ --name rsv-malaysia-west-primary \ --backup-storage-redundancy ZoneRedundant # 2. Immutability — lock once you are confident (irreversible once Locked) az backup vault update \ --resource-group rg-prod-bcdr \ --name rsv-malaysia-west-primary \ --immutability-state Unlocked # 3. Restrict public network access where private endpoints exist az backup vault update \ --resource-group rg-prod-bcdr \ --name rsv-malaysia-west-primary \ --public-network-access Disable Soft delete is enabled by default with a 14-day retention window — verify it has not been disabled, and keep it that way. For the secondary Singapore vault, replicate only what your tiering justifies: Tier 1 databases and configuration state, not everything.Site Recovery: Regional Failover That You Can Actually TestFor workloads that truly need regional DR, Azure Site Recovery replicates VMs to a target region — for Malaysia West, that means Southeast Asia. Replication policies control recovery point history (effectively your RPO) and application-consistent snapshot frequency.The infrastructure is clean to express in Bicep: param location string = 'malaysiawest' param drLocation string = 'southeastasia' resource rsv 'Microsoft.RecoveryServices/vaults@2024-04-01' = { name: 'rsv-dr-malaysia-west' location: location sku: { name: 'RS0' tier: 'Standard' } } // Azure-to-Azure replication policy: 240-minute recovery point history resource replicationPolicy 'Microsoft.RecoveryServices/vaults/replicationPolicies@2024-04-01' = { parent: rsv name: 'policy-tier2-4hr-rpo' properties: { providerSpecificInput: { instanceType: 'A2A' recoveryPointHistory: 240 // minutes of recovery point retention appConsistentFrequencyInMinutes: 120 multiVmSyncStatus: 'Enable' } } } Recovery plans define the ordered failover sequence across VMs. The CLI's site-recovery extension covers creation — fabric IDs come from az site-recovery fabric list after replication is configured: az site-recovery recovery-plan create \ --resource-group rg-prod-bcdr \ --vault-name rsv-dr-malaysia-west \ --recovery-plan-name plan-tier1-core \ --primary-fabric-id <malaysia-west-fabric-id> \ --recovery-fabric-id <southeastasia-fabric-id> \ --failover-deployment-model ResourceManager \ --groups '[{"group-type":"Boot","replication-protected-items":[{"id":"<protected-item-id>","virtual-machine-id":"<vm-resource-id>"}]}]' The part everyone skips: test failover. Site Recovery test failover spins up your replicated VMs in an isolated network in the DR region without touching production replication. Run it at least quarterly for Tier 1, and write down what breaks. That document — not the architecture diagram — is your real DR capability.Resiliency in Azure: The 2026 Step ChangeThis is what makes 2026 different from every previous year. In June 2026, Microsoft shipped three capabilities under Infrastructure Resiliency Manager (currently in preview) inside the Resiliency in Azure experience:1. Availability Zone Down Drills. These simulate a complete AZ outage for a Service Group — a logical grouping of the resources that make up one application. Fault injection covers VMs, VM Scale Sets, AKS node pools, and forced database failovers, with custom faults via Azure Automation runbooks. For resources with active-passive protection, integrated recovery plans handle failover and reprotection during the drill. Critically, this validates the zone-redundant architecture you already paid for. Many Malaysia West deployments claimed zonal resilience on paper and never proved it. The drill lifecycle runs fault injection, failover, reprotection, and reverse failback — with real-time health monitoring throughout.2. Recovery Orchestration Plans. Ordered, group-based recovery across an entire application — VMs, VMSS, SQL databases, SQL Managed Instance, Cosmos DB, PostgreSQL, AKS, and storage accounts. Resources in the same group fail over in parallel; groups run sequentially. Automated readiness checks run every 24 hours to detect configuration drift before you need the plan, and Azure Automation runbooks plug in as pre/post-scripts. One important scope note from the documentation: orchestration plans support zonal recovery, not regional recovery — regional failover stays with Site Recovery recovery plans.3. Zonal resiliency goals and recommendations. The platform continuously evaluates your resiliency posture, flags resource-level gaps, tracks adherence against goals you set, and attaches qualitative cost indicators (High, Medium, Low, or No cost impact) to each recommendation. A resiliency agent can generate the IaC templates to close gaps, and Resiliency Copilot answers natural-language questions like "which data sources have no recovery points in the last seven days?"All of this is portal-driven today — there is no az resiliency CLI yet — and Infrastructure Resiliency Manager is in preview, so expect to enrol for access. Treat preview behaviour as evolving, but the direction is clear: DR validation is becoming a first-class, continuous activity rather than an annual fire drill.The Testing Cadence That Actually WorksPrint this section. DR without testing is just documentation.Monthly:Run an AZ down drill against each Tier 1 Service GroupRestore one random backup to an isolated network and verify it worksConfirm measured RPO/RTO against your documented targetsQuarterly:Execute a full Site Recovery test failover to Southeast Asia, end to endRun communication drills — can the recovery team actually reach each other at 2 AM?Update runbooks for any infrastructure changes since the last testAnnually:Full failover exercise (not test failover) for at least one Tier 2 applicationRe-tier workloads; review cost optimisation opportunitiesBoard-level DR readiness reportAfter every major change:New workload deployed? Add it to a recovery plan and a Service GroupResiliency recommendations flagged a gap? Remediate or formally accept it within 14 daysPitfalls I See RepeatedlyNetworking dependencies. The plan assumes DNS updates instantly, but the TTL is 24 hours. Users hit dead IPs for a day. Lower TTLs ahead of planned failovers.Secrets and certificates. Backups contain encrypted secrets, but the recovery key lives in the primary region's Key Vault. You are locked out of your own recovery. Keep a recovery copy in the DR region.Third-party integrations. Payment gateways whitelist your Malaysian egress IPs. After failover to Singapore, the new source IPs get rejected. Register DR egress ranges before you need them.Human factors. The recovery lead resigned last quarter. The new owner has never seen the runbook. At least two people must be able to execute every recovery step.Automation decay. Failover scripts that worked six months ago now reference deleted resource groups. Nobody noticed because nobody ran them. This is exactly what quarterly test failovers catch.ConclusionBuilding a resilient recovery plan for a Malaysian enterprise in 2026 does not require reinventing your architecture. It requires honesty about the gap between documented and tested DR — then closing that gap with zone-redundant foundations, explicit regional DR where it matters, and the new Resiliency in Azure drills that finally make testing cheap and repeatable.Remember the Malaysia West reality: no paired region means no free safety net. Design zonal resilience deliberately, replicate to Singapore explicitly, and let the June 2026 Infrastructure Resiliency Manager capabilities prove your architecture before a real outage does it for you.Key TakeawaysDefine RPO/RTO tiers before choosing tools. Every workload maps to a tier; that mapping drives backup frequency, replication strategy, and budget.Malaysia West is non-paired — design regional DR explicitly. Cross-Region Restore is unavailable there. Use Availability Zones first, and replicate to Southeast Asia (Singapore) for regional failover until a second Malaysian region reaches GA.Harden your vaults against ransomware. Immutability, zone-redundant backup storage, soft delete, and restricted public access are table stakes in 2026, not optional extras.Test monthly with AZ down drills. Resiliency in Azure's Infrastructure Resiliency Manager makes zone-outage simulation a controlled, repeatable exercise. If you only test annually, you are hoping, not testing.Close the human gap. Two trained people per recovery step, current contact lists, quarterly communication drills. The best architecture fails when the only person with the credentials is unreachable. --- ## Azure Backup Cost Trap: Why You Overpay 2x on Storage Redundancy (and How to Fix It) URL: https://wenfeng.my/azure-backup-cost-trap-why-80-of-tenants-overpay-2x/ Author: admin Published: 2026-08-12 06:30:11 Modified: 2026-10-04 15:28:54 Every few months I sit down with a customer's Azure Cost Management export and go hunting for the line items nobody watches. Compute gets reviewed. Networking gets questioned. But backup storage quietly grows month after month, because it was configured exactly once — on day one, by whoever stood up the landing zone — and never touched again.The most common finding is boring and expensive: most vaults I audit are still running on Geo-Redundant Storage (GRS), the default. That one checkbox the team never changed is doubling their backup storage bill, and in most cases buying them protection they cannot even use.This is the most repeatable cost optimization I know of in Azure Backup. It requires no architecture change, no new tooling, and no migration project — just an audit and a handful of deliberate decisions. Here is how the trap works and how to fix it.The GRS Default TrapWhen you create a Recovery Services vault and start protecting workloads, Azure sets the vault's storage redundancy to GRS unless you explicitly choose otherwise. GRS replicates every byte of backup data three times within the primary region and three more times in a secondary location hundreds of kilometers away.That sounds like exactly what backup should do. The problem is the price, and the fact that the geographic copy is largely decorative for most tenants.What you are actually paying forAzure Backup storage is billed per GB-month at rates that depend on the redundancy option. The figures below are verified against the Azure Retail Prices API for Southeast Asia in August 2026 (GRS is only offered in paired regions — more on that below): RedundancyRate (per GB/month)10 TB per month10 TB per year LRS$0.0224~$229~$2,750 ZRS$0.0280~$287~$3,440 GRS (default)$0.0448~$459~$5,500 GRS costs exactly 2x LRS. At 10 TB that is roughly $2,750 a year of pure overhead. At 100 TB — a typical number once VM backups, SQL backups, and file shares accumulate — the overpayment is around $27,500 a year. Every year. For a copy of the data that most organizations will never restore from.Why GRS is wrong for most backup vaultsThe pitch for GRS is geographic survival: if the entire primary region fails, your backups exist somewhere else. Analyze that claim before paying for it:Azure region-wide outages are exceptionally rare. Regions fail over individual services occasionally; total region loss is a once-in-a-blue-moon event. The failure modes you actually experience — disk failure, datacenter failure, accidental deletion, ransomware — happen inside the region.Cross-region restore is not a failover. Restoring from the secondary copy is a manual, slow operation: items can take up to 48 hours to become available in the secondary region, and you restore into a region where your production network, identity configuration, and dependent services do not exist.Your production is probably not cross-region either. If your workloads run in one region and only the backups replicate elsewhere, a regional disaster leaves you with restored VMs in a region that has nothing to connect to. Real cross-region DR is a workload replication decision (Site Recovery, traffic manager, multi-region design) — not a backup storage decision.Within-region redundancy is where the real protection lives. LRS keeps three synchronous copies in one datacenter. ZRS spreads copies across availability zones in the region. For the failures that actually occur, ZRS at 1.25x the LRS price is usually the sensible middle ground — not GRS at 2x.The Malaysia West twistHere is the detail that changes the story for Malaysian tenants. Cross-region restore — the feature that justifies paying for GRS — restores into your region's Azure paired region. But Malaysia West is a nonpaired region: Microsoft's official region list shows Malaysia West with no paired region at all.It goes further than that. I checked the Azure Retail Prices API (August 2026), and there is no GRS backup storage meter for Malaysia West at all — the backup storage SKUs offered there are LRS, ZRS, and archive. GRS backup storage is simply not sold in nonpaired regions, because there is no pair to replicate to.So Malaysia West tenants are protected from this trap by the platform itself: you could not pay the GRS premium on a Malaysia West vault even if you wanted to. The trap reappears the moment your organization runs vaults in paired regions — and most Malaysian enterprises do, because Southeast Asia (Singapore) and East Asia (Hong Kong) were the default landing zones for years before Malaysia West opened. In those regions GRS is available at exactly the 2x price, and cross-region restore genuinely works — restoring into the pair (Southeast Asia ↔ East Asia). If your production sits in Southeast Asia and your vault is still on GRS, you are paying double for a copy that restores into a region where your network, identity, and applications do not live.The audit to run: list every vault, note its region, and check whether the region is paired. Nonpaired-region vaults can only be LRS or ZRS — the question there is whether you need zone redundancy. Paired-region vaults are where the GRS decision actually applies.When GRS is actually rightTo be fair, GRS has legitimate use cases:Regulatory or contractual requirements that mandate geographically separated backup copies.Paired-region DR designs where you already run cross-region infrastructure and cross-region restore genuinely slots into your recovery plan.Irreplaceable data with no other protection — the last copy of something that must survive a regional event at any cost.If none of those apply to a given workload, LRS or ZRS protects it at half the price.The Fix: A Six-Lever Backup Cost AuditStorage redundancy is the biggest lever, but it is not the only one. When I audit a tenant's backup estate, I work through six.1. Set vault redundancy deliberatelyFirst, find out what you have. This bash sweep lists every Recovery Services vault in the subscription with its redundancy and cross-region restore state: #!/bin/bash # Sweep: every Recovery Services vault and its storage redundancy az backup vault list --query "[].{name:name, rg:resourceGroup}" -o tsv | while read -r name rg; do az backup vault show --name "$name" --resource-group "$rg" \ --query "properties" -o json | python3 -c " import sys, json p = json.load(sys.stdin) rd = p.get('redundancyDetails') or p.get('redundancySettings') or {} print('$name | $rg | redundancy=%s | crossRegionRestore=%s' % ( rd.get('redundancy', 'unknown'), rd.get('crossRegionRestore', 'n/a'))) " done For vaults that do not yet protect anything, you can set the redundancy directly: az backup vault update \ --name rsv-backup-prod-myw \ --resource-group rg-backup \ --backup-storage-redundancy LRS \ --cross-region-restore Disabled (Flags evolve — run az backup vault update -h against your CLI version. The constraint that matters is constant: this only works before items are protected. See the pitfalls section.)If you deploy vaults as code, make the decision explicit so nobody inherits the default by accident. Terraform: resource "azurerm_recovery_services_vault" "backup" { name = "rsv-backup-prod-myw" resource_group_name = azurerm_resource_group.core.name location = "malaysiawest" sku = "Standard" # Deliberate choice, not the GRS default storage_mode_type = "LocallyRedundant" cross_region_restore_enabled = false } Or Bicep: resource backupVault 'Microsoft.RecoveryServices/vaults@2024-04-01' = { name: 'rsv-backup-prod-myw' location: 'malaysiawest' sku: { name: 'Standard' } properties: { redundancySettings: { redundancy: 'LocallyRedundant' crossRegionRestore: 'Disabled' } } } 2. Tier retention by workload criticalityOne-size-fits-all retention is the second silent cost. The default Azure VM policy keeps daily recovery points for 30 days and weekly points for 4 weeks, and many teams layer on monthly and yearly schedules at maximum retention for every workload — including dev/test VMs nobody would ever restore.A simple three-tier model usually cuts retained data by a third or more: TierWorkloadsDaily retentionLong-term retention CriticalProduction databases, domain controllers30 daysMonthly 12 months, yearly 3 years StandardProduction app servers14 daysMonthly 6 months BasicDev/test, ephemeral workloads7 daysNone The practical workflow: export the policy, edit retention, reapply. # Export, edit, reapply a backup policy az backup policy show \ --vault-name rsv-backup-prod-myw \ --resource-group rg-backup \ --name StandardVMs > policy.json # Edit policy.json: reduce daily retentionDuration, # trim or remove yearly schedule for this tier az backup policy set \ --vault-name rsv-backup-prod-myw \ --resource-group rg-backup \ --name StandardVMs \ --policy @policy.json 3. Archive long-term recovery pointsFor the yearly and compliance copies you must keep but will almost never touch, the archive tier is the biggest price drop in Azure Backup: roughly $0.0027 per GB-month versus $0.0224 for standard LRS — about an 8x reduction. The trade-offs are a 180-day minimum retention and rehydration time measured in hours when you do restore, which makes it exactly right for regulatory long-term copies and exactly wrong for operational recovery points.One interaction to plan for: archive tier is not supported on vaults configured with ZRS — it requires LRS or GRS. Redundancy and archive decisions are not independent knobs; make them together per vault. A vault you already moved to ZRS cannot use archive tier, and a vault you plan to archive heavily may be better kept on LRS than pushed to ZRS.4. Trim instant-restore retentionInstant restore keeps local snapshots alongside vaulted backups so recent restores are fast. The retention window is 1 to 5 days and defaults to 2. Every extra day is extra snapshot storage billed for every protected VM. For non-critical workloads, 1 day is usually enough; keep the longer window only for the systems where same-day restore speed genuinely matters.5. Right-size Site Recovery scopeAzure Site Recovery (ASR) is DR replication, not backup — and it is billed per protected instance (around $25 per month each, plus the storage the replicas consume). The trap is enabling it for every VM because "DR" appeared in a requirements document. Replicate only the workloads with a real cross-region RPO/RTO commitment. A dev/test VM with a backup and no replica is not a DR gap — it is correct scoping.6. Make backup cost visibleCosts you cannot see do not get managed. Pull backup spend per resource with the Cost Management API so it shows up in your regular reporting: from azure.identity import DefaultAzureCredential from azure.mgmt.costmanagement import CostManagementClient cred = DefaultAzureCredential() client = CostManagementClient(cred) query = client.query.usage( "/subscriptions/<subscription-id>", { "type": "Usage", "timeframe": "MonthToDate", "dataset": { "granularity": "None", "aggregation": {"totalCost": {"name": "Cost", "function": "Sum"}}, "grouping": [{"type": "Dimension", "name": "ResourceName"}], "filter": { "dimension": { "name": "ServiceName", "operator": "In", "values": ["Backup"], } }, }, }, ) for row in sorted(query.rows, key=lambda r: r[1], reverse=True): print(f"{row[2]:>10.2f} USD {row[0]}") Pair this with a monthly budget alert on the backup cost line so growth gets noticed when it is 5 percent, not 50.Pitfalls That Bite During the CleanupRedundancy is locked once items are protected. You cannot switch a vault from GRS to LRS after backup items exist. The fix is a new vault with the correct redundancy and a cutover of protection — plan it as a small migration, not a setting change.Cross-region restore cannot be reverted. Once enabled, CRR stays enabled. Treat it as a one-way decision and get sign-off before flipping it on.Nonpaired regions have no GRS at all. Malaysia West is nonpaired, and the Retail Prices API confirms no GRS backup storage meter exists there — only LRS, ZRS, and archive. The GRS trap applies to vaults in paired regions (for Malaysian organizations, typically Southeast Asia ↔ East Asia), where CRR restores into the pair.Archive tier and ZRS are mutually exclusive. Archive requires an LRS or GRS vault; it is unavailable on ZRS vaults. Decide redundancy and archive together per vault.Archive tier is not for operational recovery. The 180-day minimum retention and hours-long rehydration mean archiving the wrong tier of recovery points can hurt you during a real incident.Do not double-pay for protection. Backup plus ASR on the same VM is correct only when the workload needs both point-in-time recovery and cross-region failover. Everywhere else it is duplicate billing.Prices vary by region. The rates above are verified against the Azure Retail Prices API for Southeast Asia in August 2026. Confirm your region's rates in the Azure Pricing Calculator before quoting savings to your CFO.Key TakeawaysThe GRS default doubles backup storage cost. If nobody has ever touched your vault settings, you are likely paying it. LRS or ZRS covers the failure modes that actually happen at half the price.Malaysia West tenants are protected by the platform — their paired-region vaults are not. GRS backup storage is not even offered in nonpaired Malaysia West. The trap lives in the paired-region vaults (Southeast Asia, East Asia) that most Malaysian enterprises have been running for years.Cross-region restore is not disaster recovery. Real DR replicates workloads and networking; a geo-redundant backup copy restored into an empty region is a migration, not a failover.Retention tiering plus archive tier cuts another 30-50%. Match retention to workload criticality, and push compliance-only yearly copies to the archive tier at roughly 8x less than standard storage — on LRS/GRS vaults, since archive and ZRS do not mix.Audit quarterly and govern by policy. Backup data only grows. Enforce redundancy and retention decisions in Terraform or Bicep, and put backup cost on a monthly alert before it becomes another surprise line item.None of this is architecturally exciting. That is precisely why it works: the savings come from reversing a default, not from a project. Find your vaults, make the redundancy decision deliberately, tier the retention, and put a number on it. Most tenants I have walked through this audit cut backup storage spend roughly in half — and the whole exercise takes less time than one incident review. --- ## Azure Monitor SLIs and Metrics Export Go GA — SLO-Driven Observability for Enterprise Teams URL: https://wenfeng.my/azure-monitor-slis-and-metrics-export-go-ga-slo-driven-observability-for-enterprise-teams/ Author: admin Published: 2026-08-11 23:30:55 Modified: 2026-10-04 15:29:05 If your operations team measures success by whether servers are "up," you're measuring the wrong thing. Users don't care if your VM is running — they care whether they can log in, submit orders, and get responses in under two seconds.On June 11, 2026, Azure Monitor made two capabilities generally available that make this shift practical: Service Level Indicators (SLIs) and Metrics Export. Together they close an observability gap that has been forcing enterprises to bolt on third-party platforms just to run SLO-driven operations natively.This article is a practical guide: what these features actually do, how to implement them with real code (Azure CLI, Bicep, Terraform, Python, bash), and the pitfalls I'd warn you about before you start.The Problem: Infrastructure Metrics Don't Tell You What MattersAzure Monitor has always been good at infrastructure metrics. CPU usage, disk I/O, network throughput, memory pressure — all available out of the box for every Azure resource. But these are supply-side metrics. They tell you what your infrastructure is doing, not what your users are experiencing.Consider a scenario I've seen play out in production more than once: your VMs show 40% CPU utilization (healthy), your database responds in 10ms (excellent), and network latency is 2ms (perfect). Meanwhile, users are reporting the application feels slow and broken.The infrastructure metrics are fine. The user experience is not. Something in the middle — application logic, session handling, client-side rendering, a downstream dependency — is degrading the experience, and infrastructure metrics alone can't see it.SRE teams have known this since Google's SRE book defined SLIs and SLOs as the foundation of reliable service delivery back in 2016. But implementing them in Azure historically meant either custom code (Application Insights custom metrics, hand-built Log Analytics queries) or third-party platforms (Datadog, New Relic, Dynatrace).The native Azure Monitor SLI capability removes that excuse.What Azure Monitor SLIs Actually DoDefining what mattersAn SLI is a quantitative measure of your service's behavior from the user's perspective. In Azure Monitor's GA implementation, you define an SLI through a handful of building blocks:Service group — the application or workload boundary the SLI represents.SLI type — whether you measure availability or latency.Evaluation method — whether Azure Monitor evaluates individual requests or time windows.Baseline target — the SLO the measured result is compared against.Common SLIs you'll define this way:Availability: What percentage of requests complete successfully?Latency: What percentage of requests complete within 200ms?Error rate: What percentage of requests return errors?The SLI runs continuously against incoming telemetry and produces a running score you can track over time — not a one-off dashboard query someone has to remember to run.Setting targets (SLOs)An SLO is the target you set for your SLI: "99.9% of requests complete within 200ms over a rolling 30-day window." In Azure Monitor you set the baseline target percentage and the evaluation period, and the platform computes compliance for you.The most important derived number is the error budget — how much unreliability you're allowed. For a 99.9% SLO over 30 days, your error budget is roughly 43 minutes of bad behavior. That converts an abstract reliability target into a concrete number the whole team can work with: "We have 31 minutes of error budget left this month."Alerting on burn rate, not thresholdsThe most powerful feature is burn-rate alerting. Instead of alerting when a metric crosses a static threshold (CPU > 90%), you alert when your error budget is being consumed faster than sustainable.Azure Monitor SLIs support three alert types you wire to action groups:Baseline alert — fires when the SLI falls below target over the evaluation period.Fast burn rate — detects rapid error budget consumption over a short lookback, e.g. 14.4x the sustainable rate over the last hour. At 14.4x, a 30-day error budget is gone in about two days. This is your page-the-on-call signal.Slow burn rate — detects sustained, slower consumption that will still miss the SLO if nothing changes. This is your create-a-ticket signal.This multi-window approach, straight from Google's SRE workbook, dramatically reduces alert noise while catching real reliability issues early. You stop waking people up for transient CPU spikes and start waking them up only when users are actually about to be impacted.What Metrics Export DoesMetrics Export solves the second half of the problem: getting Azure Monitor metrics out of Azure Monitor and into whatever platform your organization actually uses.Before this, exporting metrics meant Log Analytics queries, custom connectors, or accepting multi-minute delays. Now you can stream platform metrics to:Azure Event Hubs — for custom processing pipelines and external systems (Datadog, Splunk, your own data lake).Azure Storage accounts — for long retention, archiving, and data-lake pipelines.Log Analytics — for query-driven analysis alongside logs.Two mechanisms matter, and choosing the wrong one is a common mistake:Diagnostic settings — the classic per-resource export. Simple, but multi-dimensional metrics get flattened into single-dimensional aggregates.Data collection rule (DCR) based metric export — supports multi-dimensional metrics intact. If you need per-queue, per-endpoint, or per-instance breakdowns downstream, this is the one.The native Prometheus/Grafana stackCombine SLIs + Metrics Export + Azure Monitor workspace + Managed Grafana and you get a complete open-source-style observability stack running natively in Azure: Azure resources ──► Azure Monitor (SLIs + SLOs + burn-rate alerts) │ ├── Metrics Export (DCR → Event Hubs / Log Analytics / Storage) │ └── Managed Prometheus collection ──► Azure Monitor workspace (Prometheus-compatible) │ ▼ Managed Grafana — dashboards + alerting Enterprise governance of Azure Monitor, flexibility and ecosystem of Prometheus/Grafana, no third-party license, and no telemetry leaving your tenant.Practical ImplementationHere's the implementation path I'd follow for a production web API.Step 1: Define the SLI and SLOIn the portal: Azure Monitor → Service Level Indicators → create a service group for your application, add an availability SLI evaluated per request, set baseline target 99.9% over a rolling 30-day window, and enable fast/slow burn-rate alerts against your on-call action group.Under the hood, an availability SLI over Application Insights data is computing something like this — worth knowing because you can sanity-check your SLI's numbers with the equivalent Log Analytics query: requests | where timestamp > ago(30d) | summarize total = count(), good = countif(success == true and duration < 2000) | extend availability_pct = round(100.0 * good / total, 3) If your SLI score and this query disagree materially, your SLI's evaluation method or scope is misconfigured — fix that before you trust alerts.Step 2: Export metrics via Azure CLIQuick export of an App Service's platform metrics to Event Hubs: SUB_ID="00000000-0000-0000-0000-000000000000" az monitor diagnostic-settings create \ --name "orders-api-metrics-export" \ --resource "/subscriptions/$SUB_ID/resourceGroups/rg-prod/providers/Microsoft.Web/sites/orders-api" \ --metrics '[{"category":"AllMetrics","enabled":true}]' \ --event-hub "insights-metrics-prod" \ --event-hub-rule "/subscriptions/$SUB_ID/resourceGroups/rg-observability/providers/Microsoft.EventHub/namespaces/eh-observability-prod/authorizationrules/CaptureSender" And a quick bash check of raw request metrics while you wait for the pipeline to warm up: az monitor metrics list \ --resource "/subscriptions/$SUB_ID/resourceGroups/rg-prod/providers/Microsoft.Web/sites/orders-api" \ --metric "Http5xxx" "Requests" \ --interval PT1H \ --output json \ | jq -r '.value[] | .name.value as $m | .timeseries[0].data[] | "\($m)\t\(.timeStamp)\t\(.total)"' \ | tail -24 Step 3: Bicep — the full observability baselineFor repeatable deployments, define the Monitor workspace and the export in Bicep: param location string = resourceGroup().location // Azure Monitor workspace (Prometheus-compatible store for Managed Grafana) resource amw 'Microsoft.Monitor/accounts@2023-04-03' = { name: 'amw-prod-observability' location: location } resource appService 'Microsoft.Web/sites@2024-04-01' existing = { name: 'orders-api' } resource ehNamespace 'Microsoft.EventHub/namespaces@2024-01-01' existing = { name: 'eh-observability-prod' } resource authRule 'Microsoft.EventHub/namespaces/authorizationRules@2024-01-01' existing = { parent: ehNamespace name: 'CaptureSender' } // Metrics export to Event Hubs resource metricsExport 'Microsoft.Insights/diagnosticSettings@2021-05-01-preview' = { name: 'orders-api-metrics-export' scope: appService properties: { eventHubAuthorizationRuleId: authRule.id eventHubName: 'insights-metrics-prod' metrics: [ { category: 'AllMetrics' enabled: true } ] } } Step 4: Terraform — same export, Terraform shopsIf your estate is Terraform, the same export maps cleanly: resource "azurerm_monitor_diagnostic_setting" "orders_api_metrics" { name = "orders-api-metrics-export" target_resource_id = azurerm_linux_web_app.orders_api.id eventhub_name = azurerm_eventhub.metrics.name eventhub_authorization_rule_id = azurerm_eventhub_namespace_authorization_rule.capture.id metric { category = "AllMetrics" enabled = true } } Check the azurerm provider changelog for native SLI/SLO resource support before you wire it into Terraform — until it lands there, define SLIs through the portal, ARM, or Bicep and keep them in a documented runbook.Step 5: Python — consume the exported streamOnce metrics flow into Event Hubs, downstream processing is a small consumer. This one computes a rolling good/total ratio per metric — the raw material of an external SLO engine: import json from azure.eventhub import EventHubConsumerClient CONN_STR = "<event-hub-connection-string>" # from Key Vault, never in code def on_event(partition_ctx, event): body = json.loads(event.body_as_str()) for record in body.get("records", []): metric = record.get("properties", {}) name = record.get("metricName") total = record.get("total") print(f"{metric.get('resourceId')} {name} total={total}") partition_ctx.update_checkpoint(event) client = EventHubConsumerClient.from_connection_string( conn_str=CONN_STR, consumer_group="$Default", eventhub_name="insights-metrics-prod", ) with client: client.receive(on_event=on_event, starting_position="-1") From here you can push to Datadog, persist to a data lake for long retention, or feed your own compliance reporting.Pitfalls to AvoidFlattened dimensions. Diagnostic settings export multi-dimensional metrics as single-dimensional aggregates — you'll lose per-queue, per-endpoint, per-instance breakdowns silently. If downstream tools need dimensions, use DCR-based metric export instead. Test this before you commit to a dashboard design.Event Hub firewall blocks. Diagnostic settings can't reach Event Hubs behind a virtual network firewall unless you enable the trusted Microsoft services bypass. The failure mode is a diagnostic setting that looks healthy but delivers nothing.Same-region requirement. For regional resources, the Event Hub must sit in the same region as the monitored resource. Multi-region estates need one hub per region or a hub in the right region per resource group.Setting SLO targets before you have baseline data. If you set 99.9% on day one with no idea what the service actually delivers, you'll either drown in burn-rate alerts or set a target so loose it's meaningless. Run the SLI for two to four weeks first, observe the real distribution, then set the target deliberately.One giant SLI for everything. An SLI scoped to an entire subscription tells you nothing actionable. Define one SLI per critical user journey — checkout, login, API search — so a burn-rate alert points at a specific service owner.Alert routing not split by burn speed. Fast burn should page a human; slow burn should open a ticket. Wiring both to the same notification channel recreates exactly the alert fatigue SLO alerting is supposed to kill.What This Means for Enterprise TeamsEliminating the third-party tax. Many enterprises pay $50–200K per year for Datadog or New Relic primarily for SLO tracking and cross-platform visibility. Native SLIs + Metrics Export + Managed Grafana credibly covers 80% of those use cases inside your existing Azure spend. Advanced code-level APM may still justify a specialist tool — but the SLO layer no longer does.SRE adoption loses its tooling excuse. The hard part of adopting SRE was never the philosophy; it was building the platform. Azure now ships the platform.Cross-team visibility. Metrics Export to Grafana means business stakeholders can see error budgets without Log Analytics access, and incident responders get burn-rate alerts that tell them exactly how urgent a situation really is.Key TakeawaysSLIs measure what users experience, not what servers do — the GA release shifts Azure Monitor from infrastructure dashboards to user-experience measurement with service groups, availability/latency SLI types, and request or time-window evaluation.Burn-rate alerting replaces threshold noise — baseline, fast-burn, and slow-burn alerts mapped to action groups give you the Google SRE workbook multi-window pattern natively.Metrics Export ends observability lock-in — stream metrics to Event Hubs, Azure Monitor workspace, or Log Analytics; just remember DCR export for multi-dimensional metrics.The native stack covers most enterprise needs — SLIs + Metrics Export + Managed Grafana can retire a large slice of third-party observability spend.Start with one critical user journey — run its SLI for a few weeks, set the SLO from real data, add burn-rate alerts, then expand. Don't instrument everything at once.If your team is still running your Azure estate on CPU dashboards and "is it up?" checks, this GA release is the right moment to make the shift — the tooling finally exists, it's native, and the migration path is a few CLI commands away. --- ## Claude Opus 4.8, Gemini 3.5 Flash GA, and the LLM Cost-Performance Frontier URL: https://wenfeng.my/llm-cost-performance-in-2026-claude-opus-4-8-gemini-3-5-flash-and-the-enterprise-decision-2/ Author: admin Published: 2026-08-11 06:30:33 Modified: 2026-10-04 15:31:16 The LLM landscape shifted again in mid-2026, and the implications for enterprise AI strategy are bigger than the headlines suggest. Claude Opus 4.8, Gemini 3.5 Flash GA, and a repositioned OpenAI are not just model upgrades — they represent a fundamental restructuring of the cost-performance frontier that every architect needs to understand.I work with Malaysian enterprises evaluating Azure OpenAI, self-hosted inference, and everything in between. The cost arithmetic is usually the deciding factor in whether an AI project survives its first year. If you are evaluating LLM deployment right now, this is the article that will save you six months of wrong turns.The Three Headlines That MatterClaude Opus 4.8: Parallel Subagent WorkflowsAnthropic shipped Claude Opus 4.8 on May 28, 2026, with a specific architectural bet: parallel subagent workflows. The model achieves 88.6% on SWE-bench Verified (up from 87.6% on Opus 4.7), but the real story is how it gets there — not raw reasoning power alone, but orchestration patterns built into the model's behavior.Key numbers:2.5x faster output speed in fast mode (which now costs 3x less than fast mode on prior Opus models)$5 / $25 per 1M tokens (input / output) — unchanged from Opus 4.7Parallel subagent decomposition at scale via Dynamic Workflows in Claude Code, launched alongside the model1M token context window with adaptive thinking supportFor enterprises, this means an Opus 4.8-powered agent can decompose a complex request into subtasks, execute them in parallel, and synthesize results — without you building that orchestration layer yourself. Dynamic Workflows is currently a research preview on Claude Code Enterprise, Team, and Max plans, but the direction is clear: orchestration is moving into the platform. That changes the architecture of agent systems fundamentally.Gemini 3.5 Flash GA: Frontier Intelligence at 4x SpeedGoogle shipped Gemini 3.5 Flash to GA at Google I/O on May 19, 2026, with a value proposition that is hard to ignore: frontier-class intelligence at 4x the output speed of comparable frontier models, at $1.50 / $9 per 1M tokens.Why this matters for production:Latency drops from ~2s to ~500ms for typical enterprise queries (directional; varies by workload and region)Cost per effective token is 3-5x lower than Opus 4.8 for equivalent-quality tasks1M token context window means entire codebases fit in a single promptNative multimodal (text, image, video, audio) without separate modelsFor cost-sensitive enterprise workloads — customer service, document processing, code review — Flash GA is the new default. The question is no longer "can a small model do this?" but "which workload deserves the premium model?"OpenAI's Superapp PivotOpenAI declared "chat is dead" and is rebuilding ChatGPT as a superapp. While this makes waves in consumer AI, the enterprise implications are more subtle:Model routing is becoming the default — ChatGPT increasingly auto-routes between models based on query complexityStructured outputs are first-class across all models, reducing the need for custom parsingMCP integration means ChatGPT connectors work with your enterprise tools directlyThe risk for enterprises: vendor lock-in deepens when your AI layer becomes an app platform, not just a model API.The Two-Tier MarketThe market has crystallized into two distinct tiers, and the gap between them is not just performance — it is cost.Tier 1: Frontier Reasoning Models — Claude Opus 4.8, GPT-5.x, and their peers. You reach for these when accuracy and complex reasoning matter more than throughput: multi-step agentic workflows, code generation, architectural analysis, and any task where a wrong answer costs more than an expensive inference call.Tier 2: High-Throughput Workhorses — Gemini 3.5 Flash, DeepSeek V4 Pro ($0.435/$0.87 per 1M tokens, MIT licensed), Qwen 3.5 (200+ languages, Apache 2.0). These achieve near-frontier intelligence at dramatically lower cost and are the right choice for classification, summarization, RAG retrieval, and high-volume API workloads.Running a classification pipeline on Opus 4.8 when Gemini Flash handles it in a single pass costs you roughly 3.3x more per million input tokens (and 2.8x more on output) — and against DeepSeek V4 Pro the gap widens to over 10x. That single sentence is the entire argument for this article.The Real Decision MatrixStop comparing benchmark percentages. Here is the decision framework that actually matters for production deployment: FactorClaude Opus 4.8Gemini 3.5 FlashGPT-5.4 (Azure OpenAI) Cost per 1M tokens$5 / $25$1.50 / $9$2.50 / $15 Best forComplex reasoning, code gen, agent orchestrationHigh-volume, low-latency, multimodalGeneral purpose, structured output Latency~3s (complex tasks)~500ms~1.5s Context window1M1M272K Malaysia regionVia APIVia APIAzure OpenAI — verify current GPT model availability in Malaysia West before committing Self-host optionNoNo (Gemma for open)No SOC 2 / HIPAAYesYesYes (Azure) The Malaysian Enterprise RealityIf you are operating in Malaysia, three factors override the global benchmarks:1. Data residency requirements are tightening. PDPA compliance does not strictly require onshore processing today, but BNM (Bank Negara Malaysia) and SC (Securities Commission) guidance increasingly points toward data sovereignty for financial services. If your workload touches financial or healthcare data, an Azure deployment with onshore data residency — Malaysia West where your target model is available, otherwise an APAC regional or data-zone deployment — is the safe choice, regardless of model quality rankings.2. Latency to users matters more than latency to benchmarks. Malaysian enterprises serving regional (ASEAN) customers need sub-200ms response times for customer-facing applications. Gemini Flash's raw speed advantage shrinks when you add network latency from Singapore or US endpoints. The practical gap is smaller than the benchmarks suggest.3. The cost math is different at scale. At 10M input tokens/month, the difference between Flash (~$15) and Opus (~$50) is meaningful but not project-breaking. At 1B input tokens/month — a realistic number for enterprise document processing — the gap widens to roughly $18K vs $60K per year on input alone (more once output tokens are counted). At that scale, the model choice is the cost architecture.Cost-Per-Quality: Three Real ScenariosLet me ground this in real enterprise scenarios.Scenario 1: RAG-Based Document Q&A (1,000 queries/day) ModelInput $/1MOutput $/1MDaily Cost (est.)Quality Claude Opus 4.8$5.00$25.00~$45Excellent Gemini 3.5 Flash$1.50$9.00~$15Very Good DeepSeek V4 Pro$0.435$0.87~$1.40Good For a straightforward RAG pipeline with document retrieval and answer generation, Gemini Flash delivers 90%+ of Opus quality at one-third the cost. DeepSeek V4 Pro is compelling for high-volume, cost-sensitive workloads where "good enough" accuracy is acceptable.Scenario 2: Autonomous Coding Agent (Architecture Migration) ModelTokens/TaskCost/TaskSWE-bench Verified Claude Opus 4.8~50K~$0.6588.6% GPT-5.x~55K~$0.80~85% (varies by model and harness) DeepSeek V4 Pro~60K~$0.06~72% (varies by harness) For agentic coding tasks — where a wrong answer means a failed deployment or a security vulnerability — Opus 4.8's higher cost is justified by its higher success rate. The gap translates directly into rework cycles, human intervention costs, and deployment delays.Scenario 3: Content Generation Pipeline (50 articles/day)This is where I have direct experience running a multi-agent content pipeline:Orchestration layer (planning, routing, quality checks): Opus 4.8 or Claude Sonnet — accuracy matters, volume is lowDraft generation (article writing): Gemini Flash or DeepSeek V4 — high volume, acceptable error rateClassification/tagging (topic routing, sentiment): DeepSeek V4 Pro — extreme volume, simple taskThe cost reduction from intelligent model routing is not incremental. It is 10-20x compared to running everything on a single frontier model.Practical Architecture: A Tiered Model StrategyThe right approach for most Malaysian enterprises is a tiered model strategy, not a single-model bet:Customer-Facing API (low latency, high volume) └── Gemini 3.5 Flash ($1.50/$9) - Customer service chatbot - Document classification - Simple Q&A over knowledge base Internal Developer Tools (moderate complexity) └── Gemini Pro / GPT-4.1-class ($3-5/$15) - Code review and generation - Test case generation - Technical documentation Complex Reasoning & Agents (highest quality) └── Claude Opus 4.8 ($5/$25) - Multi-step agent orchestration - Architecture decision support - Complex codebase analysis Compliance-Sensitive Workloads (data residency) └── Azure OpenAI (Malaysia West where available; otherwise APAC region) - Financial document processing - Healthcare record analysis - Any workload with PDPA/BNM constraintsImplementation: Routing at the Orchestration LayerThe key insight is that model routing should happen at the orchestration layer, not the infrastructure layer. Your agent framework — LangGraph, CrewAI, MAF, or a custom pipeline — should make the routing decision based on task type. Here is the pattern in Python:import openai from azure.identity import DefaultAzureCredential MODEL_ROUTING = { "planning": "claude-opus-4-8", # complex reasoning — Anthropic API / Bedrock / Foundry "generation": "gemini-3-5-flash", # high-throughput — Gemini API / Vertex "classification": "deepseek-v4-pro", # cost-optimized — DeepSeek API "compliance": "gpt-5.4", # Azure OpenAI deployment (data residency) } def route_task(task_type: str, prompt: str, context: str = "") -> str: model = MODEL_ROUTING.get(task_type, "gemini-3-5-flash") if model.startswith("gpt"): # Azure OpenAI: Azure AD auth, only OpenAI models credential = DefaultAzureCredential() client = openai.AzureOpenAI( azure_ad_token=credential.get_token( "https://cognitiveservices.azure.com/.default").token, api_version="2026-05-15", azure_endpoint="https://your-resource.openai.azure.com", ) else: # Non-OpenAI models: route through an OpenAI-compatible gateway # (LiteLLM, Azure API Management, or a thin proxy) client = openai.OpenAI( base_url="https://your-ai-gateway.internal/v1", api_key="gateway-key", ) response = client.chat.completions.create( model=model, messages=[ {"role": "system", "content": f"You are processing a {task_type} task."}, {"role": "user", "content": f"{context}\n\n{prompt}"}, ], temperature=0.3 if task_type == "classification" else 0.7, ) return response.choices[0].message.contentFor Azure-first enterprises that want the router to sit in front of multiple backends, Azure API Management works as the model gateway:// apim-model-router.bicep — route by X-Model-Tier header resource apim 'Microsoft.ApiManagement/service@2024-06-01-preview' = { name: 'apim-ai-gateway' location: resourceGroup().location sku: { name: 'Standard', capacity: 1 } properties: { publisherEmail: 'platform@contoso.com.my' publisherName: 'Contoso Malaysia' } }The inbound policy reads an X-Model-Tier header and uses <choose> to set the backend to Anthropic, Google, or your Azure OpenAI instance. LiteLLM or a simple Python proxy are lighter-weight alternatives if you do not already run APIM.Cost Optimization Beyond Model SelectionModel selection is the biggest lever, but three more strategies deliver measurable savings in production:1. Prompt caching: 60-85% savings. All three providers now support prefix caching. For RAG workloads where the system prompt and context are stable across requests, this is the single biggest remaining cost lever. Actual numbers from a mid-volume deployment: a 2K-token system prompt repeated 10K times/day drops from ~$15/day to ~$3/day; total monthly savings around $1,300.2. Structured output enforcement. Constrained decoding eliminates the 20-30% cost overhead of re-prompting for malformed JSON:response = client.chat.completions.create( model="gpt-4.1", response_format={"type": "json_schema", "json_schema": invoice_schema}, messages=[{"role": "user", "content": "Extract invoice data"}] ) # Guaranteed valid JSON — no re-prompting needed3. Batch processing for non-real-time workloads. Azure OpenAI's Batch API processes requests asynchronously at 50% of real-time pricing. For nightly content generation, bulk classification, or scheduled analysis, this is a free 2x cost reduction.Pitfalls I See RepeatedlyBenchmark chasing. SWE-bench scores do not predict your workload's performance. Build a small evaluation set from your own data and measure cost-per-successful-task, not cost-per-token.Using premium models for classification. In practice, 70% of enterprise queries are "simple" and do not need frontier models. Audit your usage — the answer is usually 60-70% mis-routed.Forgetting prompt caching. If you are paying for the same system prompt 10,000 times a day, you are burning money that every major provider already lets you reclaim.Skipping the data residency conversation. For financial and healthcare workloads, compliance constraints often force onshore or regional Azure deployments regardless of benchmark rankings. Decide this before architecture, not after.Ignoring lock-in. The superapp pivot deepens vendor lock-in. Evaluate MCP-based tool integration carefully before committing your AI layer to a single provider's platform.Key TakeawaysThe market split into two tiers. Frontier models (Opus 4.8, GPT-5.x) for critical reasoning; high-throughput workhorses (Gemini Flash, DeepSeek V4) for volume. Match the tier to the workload.Model routing delivers 10-20x cost reduction. Intelligent routing across a heterogeneous model stack costs a fraction of running everything on one frontier model. This is the single highest-impact optimization for enterprise AI cost.Cost-per-quality beats cost-per-token. A model that costs 10x more but succeeds on the first attempt is cheaper than a cheap model that needs three retries and human intervention.For Malaysian enterprises, compliance drives architecture. PDPA, BNM, and RMiT requirements often favor onshore Azure deployments or self-hosted inference over provider-managed APIs — even when the API is cheaper on paper.Prompt caching is the biggest remaining lever. 60-85% savings on repeated system prompts, available on all major providers today. Turn it on.The enterprises that get this right will spend 3-5x less for the same AI capability. The ones that do not will wonder why their AI budget tripled while their competitors' stayed flat. --- ## GitHub Copilot Cloud Agent Goes Fully Autonomous: Enterprise Dev Teams Need a New Review Process URL: https://wenfeng.my/github-copilot-cloud-agent-goes-fully-autonomous-enterprise-dev-teams-need-a-new-review-process-2/ Author: admin Published: 2026-08-10 23:30:22 Modified: 2026-10-04 15:31:05 A practical governance guide for AI-generated code at scaleGitHub Copilot's cloud agent now operates fully autonomously. Assign it an issue — from the issue itself, from Copilot Chat, or through the API — and it provisions a cloud environment, creates a branch, writes the code, runs the tests, iterates on failures, and opens a pull request. No human touches the keyboard until the review stage.That is a genuinely impressive capability, and enterprise adoption is accelerating. But there is a widening gap nobody wants to talk about in the rush to enable it: the governance gap. Most teams have not updated their PR review process, branch protection rules, or CI/CD gates to account for AI-generated code at scale — and the coding agent market (Cursor reportedly crossing $4B ARR as of mid-2026, Claude Code, Copilot agent mode) is pushing enterprises to move faster than their review processes can absorb.I run autonomous agents that write and commit code daily, so I have hit this problem firsthand. This article is the practical guide I wish I had: what actually needs to change in your GitHub governance before you point a Copilot cloud agent at an enterprise codebase.The Governance GapBefore AI agentsDeveloper writes code → Opens PR → Senior reviewer checks logic, security, architecture → CI runs tests → Approved and merged Review cadence: 1–3 PRs per developer per day. Each reviewer comfortably handles 2–5 PRs per day. The system balances because production and review capacity grow together — you never hire a developer without budgeting review time.With AI agentsAgent receives issue → Creates branch → Writes code (often across multiple files) → Runs tests and fixes failures → Opens PR → Waits for review One agent can produce 5–20 PRs per day. Reviewer capacity is still 2–5 PRs per day — except AI-generated code frequently spans more files and demands more scrutiny, not less.The arithmetic is brutal: Team shape PR inflow/day Review capacity/day Result 8 humans, no agents ~12 15 Balanced + 1 Copilot agent ~22 15 Queue grows + 3 Copilot agents ~42 15 Review collapse The math does not work. If one Copilot agent produces 10 PRs/day and you have three reviewers each handling 5 PRs/day, you can support about 1.5 agents. Most enterprises plan to deploy many more. Something in the pipeline must change — and "review faster" is not a durable answer.What Changes When the Author Is Not HumanHuman code review checks for logic errors, security vulnerabilities, architecture alignment, naming, and test coverage. AI-generated code needs all of that plus a different set of checks:Intent alignment — Does the code actually solve the stated problem? Agents are excellent at producing code that addresses the issue title while quietly solving a different problem.Plausible correctness — AI code often looks right while carrying subtle bugs. It reads fluently, which makes reviewers skim.Over-engineering — Agents love adding abstractions, configuration layers, and helper functions nobody asked for.Dependency safety — Agents may import packages that do not exist, are unmaintained, or are malicious lookalikes of real libraries.Context coherence — The agent may not know your codebase conventions, error-handling style, or existing utilities, and will cheerfully reinvent them.The review question shifts from "is this code correct?" to "is this code correct, necessary, and aligned with what we actually asked for?" That is a harder question, and it is why agent PRs need their own gates.Six Governance Changes to Make Now1. Tighten branch protection for agent PRsUpdate GitHub branch protection so AI-generated PRs face stricter requirements than human PRs:# .github/settings.yml (Probot Settings app or equivalent) branches: main: required_pull_request_reviews: required_approving_review_count: 2 # raise from 1 for agent PRs dismiss_stale_reviews: true require_code_owner_reviews: true required_status_checks: strict: true contexts: - ci/tests - ci/security-scan - ci/agent-pr-audit # automated gate, defined below enforce_admins: true Why two reviewers for agent PRs? Human code and AI code fail differently. Humans mostly make mistakes in logic; agents mostly make mistakes in intent alignment. One reviewer frequently catches only one of the two failure classes.2. Route agent PRs through CODEOWNERSForce agent PRs to the team that owns the affected code — and add security review for sensitive paths:# CODEOWNERS infrastructure/** @cloud-team @security-team src/api/** @backend-team @security-team migrations/** @backend-team @dba-team .github/workflows/** @platform-team @security-team Agents will happily open PRs against any path they can read. CODEOWNERS is how you make sure the right humans are the ones approving them.3. Add CI gates that detect and audit agent PRsDetect agent-authored PRs by the PR author (Copilot agent PRs come from the Copilot bot identity) and run extra checks on them:# .github/workflows/agent-pr-gates.yml name: Agent PR Quality Gates on: pull_request: types: [opened, synchronize] jobs: agent-audit: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: fetch-depth: 0 - name: Detect agent-authored PR id: detect run: | AUTHOR="${{ github.event.pull_request.user.login }}" if [[ "$AUTHOR" == *"copilot"* || "$AUTHOR" == *"[bot]" ]]; then echo "is_agent=true" >> $GITHUB_OUTPUT else echo "is_agent=false" >> $GITHUB_OUTPUT fi - name: Security scan (all PRs) run: semgrep --config=auto src/ - name: Agent-specific audit if: steps.detect.outputs.is_agent == 'true' run: | git diff origin/${{ github.base_ref }}...HEAD > /tmp/pr.diff python3 scripts/agent-pr-audit.py --diff /tmp/pr.diff - name: Dependency verification (agent PRs) if: steps.detect.outputs.is_agent == 'true' run: python3 scripts/verify-dependencies.py src/ The audit script flags the failure modes agents actually produce:#!/usr/bin/env python3 """agent-pr-audit.py — flag common AI-generated code issues in a PR diff.""" import re, sys SUSPICIOUS = [ (r'eval\(', "eval() — code execution risk"), (r'exec\(', "exec() — code execution risk"), (r'subprocess\.\w+\(.*shell=True', "shell=True — injection risk"), (r'except\s*:', "bare except — hides errors"), (r'(password|secret|token)\s*=\s*["\'][^"\']+["\']', "hardcoded credential"), (r'TODO.{0,20}implement', "unfinished implementation"), ] def audit(diff: str): findings = [] for n, line in enumerate(diff.splitlines(), 1): if not line.startswith('+') or line.startswith('+++'): continue for pattern, message in SUSPICIOUS: if re.search(pattern, line): findings.append(f"line {n}: {message}: {line.strip()[:80]}") return findings if __name__ == '__main__': diff = open(sys.argv[sys.argv.index('--diff') + 1]).read() findings = audit(diff) for f in findings: print("FINDING:", f) sys.exit(1 if findings else 0) 4. Lock down the agent's environmentThe Copilot cloud agent prepares its sandbox from a workflow you control: .github/workflows/copilot-setup-steps.yml. This file is your control point for what the agent can install, download, and touch:# .github/workflows/copilot-setup-steps.yml name: "Copilot Setup Steps" on: workflow_dispatch jobs: copilot-setup-steps: runs-on: ubuntu-latest permissions: contents: read # minimum the setup steps need; do not add more steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: "22" - run: npm ci # lockfile-only installs, never npm install - run: npm run build Treat this file like production infrastructure: review every change to it, pin action versions by SHA, use lockfile-only installs, and grant no more permissions than the agent needs. An over-privileged setup workflow is an over-privileged agent.5. Decide what agents may touch — and gate infrastructure hardNot every issue belongs to an agent. Write the criteria down before someone enables the agent on the wrong repository:✅ Good agent candidates: - Bug fixes with clear reproduction steps - Test additions for existing code - Documentation updates - Dependency bumps with CI verification - Refactors with explicit before/after criteria ❌ Keep human-owned: - Auth, encryption, access control changes - Database schema migrations - Architecture decisions - Performance-critical paths - Cross-service changes For infrastructure-as-code, add a hard gate so any agent PR touching IaC must produce a verified plan before a human even reviews it: - name: IaC what-if gate (agent PRs) if: steps.detect.outputs.is_agent == 'true' run: | if git diff --name-only origin/main...HEAD | grep -Eq '^(infra|bicep|terraform)/'; then echo "Agent touched infrastructure paths — requiring what-if" az deployment group what-if \ --resource-group rg-prod-weu \ --template-file infra/main.bicep \ --parameters infra/prod.parameters.json fi The equivalent for Terraform is a required terraform plan status check with plan output attached to the PR. The principle is the same: agent infrastructure changes get machine-verified consequences before human attention.6. Make agent work auditableYou cannot govern what you cannot see. Use the GitHub CLI to keep a live picture of agent activity:# Open agent PRs and their review state gh pr list --repo your-org/your-repo --state open \ --author "app/copilot-swe-agent" \ --json number,title,createdAt,reviewDecision # Agent vs human merge volume, last 200 merged PRs gh pr list --repo your-org/your-repo --state merged --limit 200 \ --json author,mergedAt \ --jq 'group_by(.author.login) | map({author: .[0].author.login, count: length})' Pair that with a PR template that forces the agent to declare what it did and tells human reviewers where to focus:## Agent-Generated PR **Agent:** Copilot cloud agent **Issue:** #1234 ### Reviewer focus areas - [ ] Intent: does this solve the stated problem? - [ ] Edge cases the agent could not see - [ ] Fit with existing patterns and utilities - [ ] Security implications of any new dependency Track a small set of metrics continuously: PRs per day by source, review time by source, merge-without-changes rate, and post-merge revert rate. Healthy targets: agent PRs should not take more than roughly twice as long to review as human PRs, should clear review without rework at least ~60% of the time, and should revert at under 5%. If those numbers drift, your governance is not keeping up with your agent fleet.Pitfalls to Avoid1. Rubber-stamping fluent code. Agent code reads well, and reviewers relax when prose is smooth. The dangerous bugs are the plausible ones — the off-by-one, the swallowed exception, the subtly wrong default. Require reviewers to name at least one thing they verified against the issue's intent, not just "looks good."2. Letting agents hold standing write access to sensitive paths. Branch protection and CODEOWNERS only work if they are enforced for everyone, including bots. Verify your rules actually apply to the agent identity — several teams have discovered their admin bypass also exempted the agent.3. Over-privileged agent environments. Every secret, token, and permission available in copilot-setup-steps.yml is attack surface. Agents can be steered by malicious issue text; assume anything the agent can reach, a prompt injection will eventually try to reach. Least privilege is not optional here.4. No owner for agent merges. When an agent's PR breaks production at 2 a.m., someone has to own the revert. Require every agent PR to name a human sponsor — the person who assigned the issue — and hold that sponsor accountable for merges.5. Measuring velocity instead of quality. Agent adoption makes PRs-per-week soar, which looks like winning right up until the revert rate soars with it. Track reverts, review backlog, and post-merge defects by source from day one.6. Waiting for an incident to update the process. Every team that retrofitted governance after a bad agent merge says the same thing: the controls were obvious in hindsight and cheap to add in advance. Add them before enabling the agent, not after.ConclusionThe Copilot cloud agent going fully autonomous is not a feature to block — it is a capability that relocates where the human work happens. Writing code gets cheaper; judging code gets more valuable. The enterprises that benefit most will be the ones that treat review capacity, branch protection, CI gates, and agent permissions as first-class architecture decisions rather than afterthoughts.Start with the arithmetic: count your review capacity, count the agents you plan to run, and close the gap with automation and explicit rules before you scale the fleet. The goal is not to stop AI-generated code. It is to make sure a human can still vouch for every line that reaches production.Key TakeawaysFix branch protection first. Two reviewers, dismissed stale reviews, CODEOWNERS enforcement, and required security checks for agent-authored PRs are the minimum viable baseline.Add agent-specific CI gates. Detect agent PRs by author, then run pattern audits, dependency verification, and test-coverage requirements before a human spends a minute on them.Lock down copilot-setup-steps.yml. The agent's environment definition is your security perimeter — pin versions, use lockfile installs, grant least privilege.Write the assignment criteria. Bug fixes, tests, docs, and dependency bumps go to agents; auth, schema, architecture, and performance paths stay human-owned.Measure by source. Track review time, merge-without-changes rate, and revert rate separately for agent and human PRs — drift in these numbers is your early warning system.Law Wen Feng is Principal Solution Architect at Cloud Catalyst, where he helps enterprise clients in Malaysia and Southeast Asia build governed AI-assisted development workflows. He runs AI agents that write and commit code daily and has firsthand experience with the governance challenges of autonomous coding agents. --- ## LLM Inference Optimization — Quantization, Speculative Decoding, and the 10x Cost Reduction Playbook URL: https://wenfeng.my/llm-inference-optimization-quantization-speculative-decoding-and-the-10x-cost-reduction-playbook/ Author: admin Published: 2026-08-10 06:30:08 Modified: 2026-10-04 15:31:26 Here's a number that should change how you think about AI infrastructure: 80% of AI GPU spend is now inference, not training. If your team is running LLMs in production — whether through Azure OpenAI APIs, self-hosted vLLM, or a hybrid of both — the overwhelming majority of your compute bill goes to serving models, not building them.I've been tracking this shift across Malaysian enterprises since late 2025, and the pattern is consistent. Organisations that deployed their first LLM proof-of-concept for RM5,000–10,000 in API costs are now looking at RM50,000–200,000/month bills as they scale to production. The good news: the inference optimization stack has matured dramatically. The techniques that were research papers 18 months ago are now production-ready, and they stack together to deliver genuinely transformative cost reductions.This article walks through the complete optimization layer cake — from quantization at the model level to prompt caching at the API level — with real numbers, real code, and a decision framework specifically tuned for the constraints Malaysian enterprises face.The Cost Problem: Why Inference Optimization Is Now a Board-Level ConcernBefore we dive into solutions, let's ground the problem in real numbers. Using current Azure OpenAI pricing (June 2026): ModelInput (per 1M tokens)Cached InputOutput (per 1M tokens) GPT-5.4 Global$2.50$0.25$15.00 GPT-5.4 mini Global$0.75$0.08$4.50 GPT-5.1 Global$1.25$0.13$10.00 GPT-4.1 Global$2.00$0.50$8.00 GPT-4.1 mini Global$0.40$0.10$1.60 Now consider a typical enterprise use case: a document processing pipeline that ingests 10,000 documents/day, each requiring ~2,000 input tokens and generating ~500 output tokens. At GPT-5.4 pricing, that's roughly $75/day in output tokens plus $50/day on the input side — about RM550/day or RM16,500/month. For a mid-sized Malaysian enterprise, that's a non-trivial budget line item that demands optimization.The playbook I'm about to walk through has allowed teams I work with to cut these bills by 5-10x while maintaining or improving quality. It works through five optimization layers, each compounding the gains of the ones beneath it.Layer 1: Quantization — Shrinking Models Without Shrinking CapabilitiesQuantization is the foundational layer. It reduces model weight precision from 16-bit floating point (FP16/BF16) to 8-bit, 4-bit, or even lower, cutting memory requirements by 2-4x with surprisingly minimal quality degradation.The Quantization SpectrumHere's what the benchmarks actually show for a Llama 3.1 70B model (illustrative figures from community GGUF testing): QuantizationModel SizeQuality (MMLU)Inference SpeedVRAM Required FP16 (baseline)140 GB82.11.0x2x A100 80GB Q8_070 GB81.81.3x1x A100 80GB Q4_K_M40 GB81.21.8x1x A100 80GB Q4_038 GB80.51.9x1x A100 80GB Q2_K26 GB77.32.1x1× RTX 5090 32GB (partial offload) The key insight: Q4_K_M gives you near-FP16 quality at half the model size and nearly 2x the inference speed. For most enterprise tasks — RAG, summarization, classification — the quality difference between Q8 and Q4 is practically undetectable.Practical Quantization with llama.cpp # Quantize a HuggingFace model to GGUF Q4_K_M # Requires: llama.cpp build with CUDA support # Step 1: Convert to GGUF python convert_hf_to_gguf.py \ meta-llama/Llama-3.1-70B-Instruct \ --outfile llama-3.1-70b-f16.gguf \ --outtype f16 # Step 2: Quantize to Q4_K_M ./llama-quantize \ llama-3.1-70b-f16.gguf \ llama-3.1-70b-q4_k_m.gguf \ Q4_K_M # Step 3: Serve with llama.cpp ./llama-server \ -m llama-3.1-70b-q4_k_m.gguf \ --host 0.0.0.0 \ --port 8080 \ -ngl 99 \ # offload all layers to GPU -c 8192 \ # context length -t 8 # CPU threads When to Use WhatProduction RAG pipelines: Q4_K_M on a single A100 — best cost/quality ratioCustomer-facing chat: Q8_0 for maximum quality retentionEdge deployment / on-premise: Q4_0 on RTX 4090s or even smaller quantizations on laptop GPUs for internal toolsMaximum throughput, accept some degradation: Q3_K_M or Q2_K for high-volume classification tasksFor Malaysian enterprises with on-premise GPU infrastructure (common in banking and government), quantization is often the single highest-impact optimization. Going from FP16 to Q4 lets you serve a 70B model on a single GPU instead of two, halving your infrastructure cost immediately.Layer 2: KV Cache Compression — Unlocking Longer ContextsThe KV (Key-Value) cache is the memory that LLMs use to maintain attention over conversation history. For a 70B model with a 4K context window, the KV cache alone consumes ~2GB of VRAM. At 128K context, that balloons to ~64GB — more than most single GPUs can handle.Grouped Query Attention (GQA)Modern models like Llama 3.1 and Qwen 2.5 use GQA instead of Multi-Head Attention (MHA). Instead of each attention head having its own key and value projection, groups of query heads share a single key-value pair.The impact: 4-8x reduction in KV cache size with negligible quality loss. Llama 3.1 70B uses 8 KV heads (vs. 64 query heads), giving an immediate 8x reduction.PagedAttention with vLLMvLLM's PagedAttention brings OS-style virtual memory management to the KV cache. Instead of pre-allocating contiguous memory blocks for each sequence (which wastes memory on variable-length outputs), it allocates small "pages" on demand. # vLLM with PagedAttention (enabled by default) from vllm import LLM, SamplingParams # Serve a Q4-quantized model with automatic KV cache management llm = LLM( model="meta-llama/Llama-3.1-70B-Instruct", quantization="awq", # or "gptq" tensor_parallel_size=2, # split across 2 GPUs max_model_len=32768, # 32K context gpu_memory_utilization=0.90, # 90% of GPU memory to KV cache enable_prefix_caching=True, # cache common prefixes ) # Process multiple requests with efficient KV cache sharing sampling_params = SamplingParams( temperature=0.7, top_p=0.9, max_tokens=2048, ) prompts = ["Summarize this document: ..."] * 100 # batch of requests outputs = llm.generate(prompts, sampling_params) The practical result: vLLM achieves up to 24x higher throughput than HuggingFace Transformers (and 2.2-3.5x higher than HuggingFace TGI) under high-concurrency workloads, primarily through efficient KV cache management. For a Malaysian fintech processing thousands of customer queries per minute, the difference between naive serving and vLLM is the difference between needing 8 GPUs and needing 1.Layer 3: Continuous Batching — Maximizing GPU UtilisationTraditional "static batching" waits for all requests in a batch to complete before processing the next batch. If one request generates 50 tokens and another generates 500, the GPU sits idle for the duration of the shorter request while waiting for the longer one.Continuous batching (also called iteration-level scheduling) inserts and removes requests at every decoding step, keeping the GPU fully utilised.The Throughput ImpactRepresentative community benchmarks illustrate the difference clearly (exact figures vary by model, hardware, and workload): Batching MethodThroughput (tok/s)GPU Utilisationp99 Latency Static batching (HF TGI)4235%12.4s Continuous batching (vLLM v0.6)28589%4.1s That's a 6.8x throughput improvement from batching alone, with latency actually decreasing because requests aren't queuing behind each other.vLLM Continuous Batching in Practice # vLLM server with continuous batching optimized for throughput # Run as a service # vllm serve meta-llama/Llama-3.1-70B-Instruct \ # --tensor-parallel-size 2 \ # --max-num-seqs 64 \ # --max-num-batched-tokens 8192 \ # --enable-prefix-caching # Client code import openai client = openai.OpenAI( base_url="http://your-vllm-server:8000/v1", api_key="not-needed", ) # vLLM serves as an OpenAI-compatible endpoint # All requests benefit from continuous batching automatically response = client.chat.completions.create( model="meta-llama/Llama-3.1-70B-Instruct", messages=[{"role": "user", "content": "Analyse this financial report..."}], temperature=0.7, max_tokens=1024, ) Layer 4: Speculative Decoding — The Biggest Latency BreakthroughSpeculative decoding is the technique that made me stop dismissing "academic" inference optimizations. A small, fast "draft" model proposes multiple tokens simultaneously, then the large "target" model verifies them in a single forward pass. Correct proposals are accepted; incorrect ones trigger a rollback to the draft model's output at that position.The critical property: speculative decoding is lossless. The output is mathematically identical to running the large model alone — you're just getting there faster.Real Benchmark NumbersSnowflake's Arctic Inference team published the most comprehensive benchmarks on vLLM (Llama 3.1 70B, 8&times;H100 node): WorkloadNo SpeculationN-gramEAGLELSTM + Suffix (Arctic) ShareGPT (chat)76 tok/s91 tok/s102 tok/s179 tok/s HumanEval (code)77 tok/s100 tok/s112 tok/s217 tok/s SWE-Bench (agents)76 tok/s175 tok/s—302 tok/s That's a 2.3x-4x speedup depending on workload, with zero quality degradation. For agentic workloads (the fastest-growing inference category), the gains are the largest because agent tasks contain highly repetitive patterns that draft models exploit efficiently.Deploying Speculative Decoding with vLLM # Deploy Llama 3.1 70B with Arctic LSTM speculative decoding pip install "git+https://github.com/snowflakedb/ArcticInference.git#egg=arctic-inference[vllm]" vllm serve \ meta-llama/Llama-3.1-70B-Instruct \ --quantization "fp8" \ --tensor-parallel-size 2 \ --speculative-config '{ "method": "arctic", "model": "Snowflake/Arctic-LSTM-Speculator-Llama-3.1-70B-Instruct", "num_speculative_tokens": 3, "enable_suffix_decoding": true }' The draft model is tiny — typically 0.5-2 billion parameters — so it adds negligible memory overhead while dramatically accelerating the target model. For enterprises running inference at scale, this is essentially free performance.N-gram Speculation for Quick WinsIf you don't want to train or deploy a draft model, vLLM's N-gram speculative decoding works out of the box — it looks for repeating token patterns in the input prompt and uses them as draft proposals: # Zero-config N-gram speculation # Just add to your vLLM config: # --speculative-config '{"method": "ngram", "num_speculative_tokens": 4}' Even this lightweight approach yields 1.2-2.3x speedups on code-heavy workloads where variable names and patterns repeat frequently.Layer 5: Prompt Caching — The API-Level Cost KillerIf you're using managed APIs (Azure OpenAI, OpenAI, Anthropic), prompt caching is the single easiest cost reduction available. It requires zero infrastructure changes and delivers 40-85% input cost reductions for workloads with repeated prompt prefixes.How It WorksWhen a request contains a prefix identical to a previously cached request (same model, same first 1024+ tokens), the provider charges only 10% of the normal input token rate for the cached portion. On Azure OpenAI, cached input tokens for GPT-5.4 cost $0.25/1M instead of $2.50/1M.Real Cost ImpactConsider a RAG pipeline with a 3,000-token system prompt (instructions + retrieved context) and a 200-token user query: ScenarioInput TokensStandard CostCached CostSavings 10K requests/day, 50% cache hit3,200 × 10K$80.00/day$44.00/day45% 10K requests/day, 80% cache hit3,200 × 10K$80.00/day$22.40/day72% 50K requests/day, 80% cache hit3,200 × 50K$400.00/day$128.00/day72% That last row represents a RM11,520/month saving — from changing zero lines of code, just ensuring your prompts share a common prefix.Azure OpenAI Prompt Caching Implementation from openai import AzureOpenAI client = AzureOpenAI( api_key="your-key", api_version="2024-12-01-preview", azure_endpoint="https://your-resource.openai.azure.com/", ) def process_with_caching(user_query: str, context: str) -> str: # The system prompt and context MUST be identical prefix for caching # Place static content FIRST — caching works left-to-right response = client.chat.completions.create( model="gpt-5.4-global", messages=[ { "role": "system", "content": f"""You are a compliance assistant for Malaysian banking regulations. Standards: {context} Follow these rules exactly: 1. Always cite the specific regulation section 2. Flag any conflicts with BNM Guidelines 3. Rate compliance risk as Low/Medium/High/Critical""" }, {"role": "user", "content": user_query} ], temperature=0.3, max_tokens=1024, ) # Check cache hit rate in response usage = response.usage if hasattr(usage, 'prompt_tokens_details'): cached = usage.prompt_tokens_details.cached_tokens or 0 print(f"Cache hit: {cached}/{usage.prompt_tokens} tokens") return response.choices[0].message.content The critical implementation detail: place your long, static content at the beginning of the prompt. Caching works left-to-right through the token stream. If your system prompt varies per request, you won't get cache hits.The Decision Framework: Self-Host vs. API for Malaysian EnterprisesThis is where it gets practical. The right choice depends on your scale, data sensitivity, and team capabilities.Cost Comparison: Azure OpenAI vs. Self-Hosted vLLM FactorAzure OpenAI (GPT-5.4 mini)Self-Hosted vLLM (Llama 3.1 70B Q4) Input cost (per 1M tokens)$0.75~$0.09* Output cost (per 1M tokens)$4.50~$0.24* With prompt caching$0.08 inputN/A (local) Infrastructure cost$0~$2.50/hr (H100) or ~$1.00/hr (A100) Setup complexityMinutesDays to weeks Data residencyAzure Malaysia West (✅ PDPA)Full control (✅) MaintenanceZeroOngoing (model updates, GPU failures) Break-evenBest <50M tokens/monthBest >50M tokens/month *Self-hosted costs assume 2&times;A100 80GB at $2/hr combined serving ~274K output tokens/hr at a 76 tok/s baseline (&asymp;$0.09/1M input, &asymp;$0.24/1M output); speculative decoding roughly doubles throughput at similar cost. Raw GPU throughput alone is not the whole equation &mdash; factor in engineering, monitoring, model updates, and failure handling when comparing paths.When to Choose Each PathChoose Azure OpenAI when:Monthly token volume is under 50M tokensYou need rapid deployment (days, not weeks)Your team lacks ML infrastructure expertiseYou need guaranteed SLAs and no ops burdenData can leave your premise (Malaysia West region covers PDPA requirements)Choose self-hosted vLLM/llama.cpp when:Monthly volume exceeds 100M tokens (the economics flip decisively)Data sovereignty is non-negotiable (some government and financial sectors)You need model customisation (fine-tuning, custom system prompts baked into weights)You're running high-frequency, low-latency workloads (sub-200ms p99)You already have GPU infrastructure for trainingThe hybrid approach (increasingly common in Malaysian enterprises): Use Azure OpenAI for customer-facing applications that need SLAs and rapid scaling, self-host open models for internal high-volume processing pipelines where you control the data.Stacking the Layers: The 10x Cost Reduction in PracticeThe real power comes from combining all five layers. Here's a real scenario I helped architect for a Malaysian financial services company processing KYC documents:Baseline: Azure OpenAI GPT-5.4, 100K documents/month, ~30K input and ~10K output tokens per document across multiple review passes.Monthly cost: ~$7,500 (input) + ~$15,000 (output) = $22,500/monthAfter optimization stack:Self-hosted Llama 3.1 70B Q4 on 2x A100 (on-premise, existing GPUs): $0 marginal computevLLM continuous batching: 6.8x throughput improvement → handled same loadSpeculative decoding (EAGLE): 2.3x latency reduction → same throughput with 1 GPUPrompt caching (system prompt reuse): 45% input reduction on remaining API fallback trafficQuantized model quality maintained at Q4_K_M (verified on internal eval set)Optimised monthly cost: ~$350 (GPU electricity) + $500 (Azure fallback for edge cases) = ~$850/monthThat's a 26x reduction — from $22,500 to $850. Even in less aggressive scenarios where you keep Azure OpenAI as primary and just add prompt caching + batching, you're looking at 3-5x reductions from day one.Key TakeawaysQuantization is table stakes, not a compromise. Q4_K_M delivers 98%+ of FP16 quality on enterprise workloads. If you're still serving unquantized models in production, you're paying 2-4x more than necessary for no measurable quality gain.Speculative decoding is the most underutilised optimization. The technology is lossless, adds negligible overhead, and delivers 2-4x speedups — especially on the agentic workloads that are driving the next wave of enterprise AI. Every production inference deployment should evaluate it.Prompt caching is free money for API users. Restructure your prompts to put static content first, and you'll immediately save 40-85% on input costs with zero code changes. This is the first thing every Azure OpenAI customer should implement.Self-hosting becomes cost-effective at scale, but don't underestimate the ops burden. For most Malaysian enterprises under 100M tokens/month, Azure OpenAI with prompt caching and Batch API pricing (50% discount) is the pragmatic choice. Go self-hosted when data sovereignty demands it or volume makes it economically compelling.The optimization layers compound. Each layer builds on the previous one. A quantized model with KV cache optimization, continuous batching, speculative decoding, and prompt caching on API fallback can deliver 10-25x cost reductions. The playbook exists — the gap is adoption. --- ## Why Most Enterprise AI Agent Pilots Fail in 2026 — and the Orchestration Patterns That Actually Work URL: https://wenfeng.my/why-60-percent-enterprise-ai-agent-pilots-fail-2026-orchestration-patterns-2/ Author: admin Published: 2026-08-09 23:30:42 Modified: 2026-10-04 15:31:05 Here is a number that should make every enterprise CTO uncomfortable: the vast majority of agentic AI pilots never reach production. Gartner predicts over 40% of agentic AI projects will be canceled entirely by the end of 2027, due to escalating costs, unclear business value, or inadequate risk controls. LangChain's 2026 State of Agent Engineering survey of 1,300+ practitioners found that only 57% of organizations have agents running in production — meaning roughly 4 in 10 do not. Multiple industry analyses cite even starker figures, with some estimating that up to 88% of agent pilots stall before reaching production. The question has shifted from "should we use AI agents?" to "why do our agents keep failing, and how do we coordinate hundreds of them without losing control?"I run a multi-agent content pipeline in production. Every day, autonomous agents research topics, write articles, optimize SEO, schedule social posts, and publish to a Ghost blog — all coordinated through an orchestration layer with circuit breakers, cost caps, and human-in-the-loop gates. I have also watched friends and clients attempt agent deployments that crashed spectacularly. The patterns of failure are remarkably consistent. So are the patterns that survive.The Five Failure Modes That Kill Agent Pilots1. Bolting Agents Onto Workflows Designed for HumansThis is the most common and most fatal mistake. An enterprise identifies a process — customer support triage, document review, report generation — and wraps an LLM agent around it with a few function-calling tools. The agent is expected to behave like a human employee but faster.The problem is structural. Human workflows assume judgment, escalation, and context-switching that agents cannot replicate without explicit engineering. A human support agent knows when to escalate to a manager. An LLM agent with generic instructions will either try to resolve everything itself (causing hallucinated actions) or refuse to act at all (causing timeouts and deadlocks).What works instead: Decompose the workflow into discrete, well-defined steps. Each step should have clear inputs, outputs, and failure criteria. The agent's job is to execute one step well, not to own the entire workflow. This is the pipeline pattern, and it is the most reliable orchestration approach for enterprise use.2. Runaway Loops and Token Cost ExplosionsAn agent is given a task with a vague success criterion. It tries, fails, retries, tries a different approach, fails again, and keeps looping — consuming tokens at $5-25 per million while producing nothing useful. In one real case I observed, a code-generation agent burned through $400 in API costs on a single task that a human developer could have completed in 20 minutes.Without explicit loop guards — maximum iteration counts, time budgets, and cost caps — agents will consume resources indefinitely. Most enterprise pilot budgets are consumed by runaway agents before the team realizes what happened.What works instead: Every agent invocation needs three hard limits, enforced in code rather than in the prompt:import time class AgentBudget: """Hard limits for a single agent invocation.""" def __init__(self, max_iterations=8, max_seconds=240, max_cost_usd=1.50): self.max_iterations = max_iterations self.max_seconds = max_seconds self.max_cost_usd = max_cost_usd self.iterations = 0 self.cost_usd = 0.0 self.started = time.monotonic() def check(self): self.iterations += 1 if self.iterations > self.max_iterations: raise BudgetExceeded(f"iteration cap {self.max_iterations}") if time.monotonic() - self.started > self.max_seconds: raise BudgetExceeded(f"time cap {self.max_seconds}s") if self.cost_usd > self.max_cost_usd: raise BudgetExceeded(f"cost cap ${self.max_cost_usd}") def add_cost(self, tokens_in, tokens_out, rate_in=3.0, rate_out=15.0): # rates per 1M tokens — adjust to your model pricing self.cost_usd += (tokens_in * rate_in + tokens_out * rate_out) / 1_000_000 The prompt can suggest discipline; only the runtime can enforce it.3. Context Window Exhaustion and Memory CollapseAgents that accumulate conversation history eventually hit context window limits. When they do, behavior degrades unpredictably — earlier instructions get forgotten, critical context gets dropped, and the agent starts repeating itself or contradicting its own earlier decisions.This is especially dangerous in multi-step workflows where an agent needs to remember decisions from step 1 while executing step 5. Without explicit memory management, the agent loses track.What works instead: Use structured memory patterns. For pipeline workflows, each step receives only the inputs it needs — not the full conversation history. For persistent agents, implement rolling context windows with explicit summarization checkpoints. Tools like vector stores (Azure AI Search, Cosmos DB with vector search) provide durable memory that survives context window resets.4. Tool Misuse and Hallucinated ActionsAgents with function-calling access can invoke tools incorrectly — calling APIs with wrong parameters, accessing resources they should not touch, or fabricating tool calls that do not match any available function. In production, tool misuse is not just an error — it is a security incident.A common pattern: the agent hallucinates a tool name that does not exist, the runtime returns an error, and the agent retries with a slightly different hallucinated name. This creates noisy logs, wastes tokens, and can trigger unintended side effects if the runtime is not strict about validation.What works instead: Implement strict tool schemas with runtime validation. Every function call must be validated against the actual tool registry before execution. Use least-privilege access — agents should only have access to the specific tools they need for their assigned step, not the full toolset. On Azure, that means a dedicated managed identity per agent role with narrowly scoped RBAC, not a shared contributor account.5. No Observability Into Agent Decision-MakingWhen an agent fails in production, the team often cannot determine why. The agent's reasoning is opaque — it received a prompt, made some decisions, called some tools, and produced output. Without structured logging of each decision point, debugging becomes guesswork.This is the observability gap that separates production-grade agent systems from demos. In a demo, you watch the agent work and everything looks impressive. In production, you need to know exactly what the agent decided at each step, what tools it called, what data it saw, and where it went wrong.What works instead: Log every agent decision point as structured JSON, then ship it to Application Insights or a Log Analytics workspace:import json import logging import time def log_agent_decision(agent, step, tool, params, result): logging.info(json.dumps({ "ts": round(time.time()), "agent": agent, "step": step, "tool": tool, "params": params, "status": result.get("status"), "tokens": result.get("tokens_used"), "cost_usd": result.get("cost_usd"), })) If you cannot reconstruct a failed agent run from your logs in 15 minutes, your observability is insufficient.The Orchestration Patterns That Actually WorkHaving seen what fails, here are the patterns that survive production deployment.Pattern 1: Sequential PipelineThe simplest and most reliable pattern. Tasks flow through a chain of specialized agents, each handling one well-defined step. No agent has visibility into the full workflow — each receives inputs from the previous step and produces outputs for the next.When to use: Content creation, data processing, ETL workflows, report generation.Why it works: Each agent has a narrow, testable responsibility. Failures are isolated to individual steps. Cost is predictable because each step has bounded inputs.PIPELINE = [research_scout, strategise, write_draft, seo_review, social_pack] def run_pipeline(topic: dict) -> dict: state = {"topic": topic} for step in PIPELINE: try: state = step(state, AgentBudget()) except (BudgetExceeded, StepError) as exc: log_agent_decision(step.__name__, "run", "-", {}, {"status": str(exc)}) break # partial state is preserved and retried on the next run return state Real example: My content pipeline uses exactly this pattern: Research Scout → Content Strategist → Blog Writer → SEO Optimizer → Social Writer → Buffer Poster. Each agent runs independently. If the SEO Optimizer fails, the blog post still exists — it just gets optimized on the next pass.Pattern 2: Supervisor (Router) PatternA central orchestrator agent evaluates incoming requests and routes them to specialized worker agents. The supervisor handles coordination, error recovery, and result aggregation.When to use: Customer support routing, task classification, multi-domain assistants.Why it works: The supervisor provides a single point of control while workers remain specialized. Error handling is centralized — if a worker fails, the supervisor can retry, re-route, or escalate.Trade-off: The supervisor itself becomes a single point of failure and a cost center. Keep the supervisor's logic simple — classification and routing, not complex reasoning.Pattern 3: Parallel Fan-OutFor independent subtasks that can run concurrently. A coordinator splits a request into parallel workstreams, each handled by a dedicated agent, then aggregates results.When to use: Multi-source research, parallel code review, batch processing.Why it works: Dramatically reduces latency. Five parallel research agents finishing in 2 minutes beats one sequential agent taking 10 minutes.Trade-off: Requires careful result aggregation. Conflicting outputs from parallel agents need resolution logic.Pattern 4: Circuit Breaker PatternNot an orchestration pattern per se, but a critical resilience pattern. Every agent invocation is wrapped in a circuit breaker that monitors failure rates, cost accumulation, and response quality. When thresholds are exceeded, the circuit opens — the agent is taken offline and the workflow degrades gracefully.Configuration example:circuit_breaker: max_failures: 3 max_cost_usd: 2.00 max_duration_seconds: 300 half_open_after_seconds: 600 Why this is essential: Without circuit breakers, a single misbehaving agent can cascade failures through the entire pipeline, consuming budget and time while producing nothing. Circuit breakers convert hard failures into graceful degradation.Cost Governance: Enforce Budgets in Infrastructure, Not Just CodeCode-level guards protect a single invocation. Infrastructure-level budgets protect the whole pilot — including the bugs you did not anticipate. Two complementary controls:1. Azure consumption budget with alerting (Bicep):resource agentBudget 'Microsoft.Consumption/budgets@2024-08-01' = { name: 'agent-pilot-monthly' scope: resourceGroup() properties: { amount: 500 category: 'Cost' timeGrain: 'Monthly' timePeriod: { startDate: '2026-08-01T00:00:00Z' } notifications: { AlertAt80Percent: { enabled: true operator: 'GreaterThan' threshold: 80 contactEmails: ['platform-team@example.com'] } } } } Or the equivalent Azure CLI one-liner:az consumption budget create \ --budget-name agent-pilot-monthly \ --category cost \ --amount 500 \ --time-grain monthly \ --start-date 2026-08-01 \ --end-date 2026-12-31 \ --resource-group rg-ai-agents 2. A daily tripwire that pauses the pipeline:#!/bin/bash # Fail fast if yesterday's agent spend crossed the daily cap DAILY_CAP=20 SPEND=$(az costmanagement query \ --type ActualCost \ --scope "/subscriptions/$(az account show --query id -o tsv)/resourceGroups/rg-ai-agents" \ --timeframe Custom \ --time-period '{"from":"'"$(date -d yesterday +%Y-%m-%d)"'T00:00:00Z","to":"'"$(date -d yesterday +%Y-%m-%d)"'T23:59:59Z"}' \ --dataset '{"granularity":"None","aggregation":{"totalCost":{"name":"PreTaxCost","function":"Sum"}}}' \ --query "properties.rows[0][0]" -o tsv) SPEND=${SPEND:-0} if (( $(echo "$SPEND > $DAILY_CAP" | bc -l) )); then echo "ALERT: agent spend $SPEND exceeded daily cap $DAILY_CAP" >&2 exit 1 # scheduler treats non-zero as "do not start tonight's run" fi An agent pilot that costs $500/month is sustainable. One that silently costs $5,000/month is a budget crisis — and the crisis is what gets pilots cancelled.A Practical Checklist for Malaysian EnterprisesBefore deploying AI agents in production, verify these prerequisites:Decompose first, agent second. Map your workflow into discrete steps before assigning agents. If a step cannot be defined with clear inputs and outputs, it is not ready for an agent.Start with the pipeline pattern. Sequential pipelines are the lowest-risk entry point. Each step is testable, debuggable, and replaceable.Implement cost controls from day one. Per-task guards in code, consumption budgets in infrastructure, and a daily tripwire in your scheduler.Require structured logging. Every agent must log its decisions, tool calls, and outputs in a structured format. If you cannot debug an agent failure in 15 minutes, your observability is insufficient.Test failure scenarios. What happens when an agent returns garbage? When a tool call times out? When the LLM API is rate-limited? If you have not tested these scenarios, you have not tested your system.PDPA compliance for agent data access. Malaysian enterprises must ensure agents handling personal data comply with PDPA. This means least-privilege tool access, data masking in agent context, and audit trails for all data access.Human-in-the-loop for critical actions. Any agent action with side effects — sending emails, modifying data, making financial decisions — should require human approval in the first deployment phase. Automate later, once confidence is established.The Real Cost of Getting It WrongThe vast failure rate is not just a technology problem. Failed agent pilots waste engineering time, consume API budgets, erode stakeholder confidence, and create organizational resistance to future AI initiatives. A failed pilot costs more than no pilot at all, because it poisons the well.The enterprises succeeding with AI agents in 2026 share three traits:They treat agent deployment as an engineering discipline, not an experimentation exercise.They invest in orchestration infrastructure before scaling agent count.They measure agent ROI relentlessly and kill underperforming agents quickly.Key TakeawaysMost enterprise agent pilots fail — Gartner predicts over 40% of agentic AI projects will be canceled by end of 2027, and industry surveys find only about 57% of organizations have agents in production. The root cause: bolting agents onto human-designed workflows instead of decomposing tasks into agent-compatible steps.The pipeline pattern is the safest entry point — sequential, testable, debuggable steps with clear handoffs between agents.Circuit breakers and budget guards are not optional — enforce iteration limits, time budgets, and cost caps in code, and back them with infrastructure-level consumption budgets.Observability separates production from demos — without structured logging of agent decisions, debugging is guesswork.Start small, measure relentlessly, and automate incrementally — the enterprises succeeding with agents treat deployment as engineering, not experimentation.This article is part of the AI Agents & Multi-Agent Systems series on wenfeng.my/. For practical architecture guidance on building production agent systems, follow the blog for weekly updates. --- ## Multi-Agent AI Systems in 2026 — Architecture Patterns (MCP, A2A, Swarm) for Production URL: https://wenfeng.my/multi-agent-ai-systems-in-2026-architecture-patterns-for-production/ Author: admin Published: 2026-08-09 06:30:14 Modified: 2026-10-04 15:31:05 The AI agent market is projected to reach roughly $8.5 billion by 2026 (Deloitte TMT Predictions 2026), and 57% of organizations surveyed report already running agents in production (LangChain State of Agent Engineering, June 2026). Yet most of these deployments are fragile, hard to debug, and expensive to operate. The gap between a working demo and a production system is architectural — and that gap is where most teams get stuck. After running a multi-agent content pipeline in production for months, I have a clear view of what works, what breaks, and which architecture patterns are actually production-ready in 2026. This article provides a practical comparison of the three dominant patterns — sequential pipeline, supervisor/orchestrator, and swarm — plus the protocols (MCP, A2A) that connect them. Why Multi-Agent Demos Fail in Production Demos succeed because the happy path is scripted. Production fails on everything around it: Unbounded context. Agents accumulate conversation history until costs spike or the model starts dropping instructions. Silent handoff failures. One agent emits slightly malformed output, the next agent improvises around it, and the error surfaces three steps later. No cost ceiling. A retry loop or negotiation cycle can burn through a month's token budget in one night. No audit trail. When something goes wrong, nobody can reconstruct which agent decided what, with which input. Every architecture decision below is really a decision about how you manage these four failure modes. The Three Production Architecture Patterns Pattern 1: Sequential Pipeline How it works: Tasks flow through a fixed sequence of specialized agents. Each agent completes its work, passes the result to the next, and the chain terminates with a final output. Research Scout → Content Strategist → Writer → SEO Review → Social Writer When to use it: Workflows with a clear, linear dependency chain Tasks where each step has well-defined inputs and outputs Production systems that need predictable cost and latency Teams that want easy debugging (you can inspect any step's output) Real-world example: A content pipeline where research feeds strategy, strategy feeds writing, writing feeds SEO review, and SEO feeds social content creation. Each agent has a single responsibility and a clear handoff contract. Advantages: Predictable token consumption (each step's context is bounded) Easy to debug (inspect any intermediate output) Simple to scale (parallelize independent chains) Cost is linear and auditable Disadvantages: No adaptive behavior — if step 3 fails, the whole chain fails Fixed topology — adding a new step requires redesigning the chain No feedback loops — earlier steps cannot learn from later steps Production tip: Enforce structured JSON handoff contracts between stages. Validate at every boundary so format drift is caught where it happens, not three steps downstream: from pydantic import BaseModel, ValidationError class HandoffContract(BaseModel): topic_id: str title: str word_count: int output_path: str def handoff(step_name: str, output: dict, next_agent: str) -> dict: try: return HandoffContract(**output).model_dump() except ValidationError as exc: raise RuntimeError( f"{step_name} produced invalid handoff for {next_agent}: {exc}" ) from exc Pattern 2: Supervisor/Orchestrator How it works: A central "supervisor" agent receives user requests, decomposes them into subtasks, assigns subtasks to specialist agents, collects results, and synthesizes a final answer. ┌→ Research Agent ─┐ User Request → Supervisor ─→ Writing Agent ─┼→ Final Output └→ Review Agent ──┘ When to use it: Complex tasks requiring dynamic routing and decision-making Workloads where the optimal agent mix varies per request Systems that need to handle diverse input types (text, code, data) Scenarios where the supervisor leverages frontier models for planning while delegating execution to cheaper models Real-world example: An enterprise assistant that routes user questions to different specialist agents — a SQL agent for data queries, a code agent for programming tasks, a documentation agent for policy lookups — based on intent classification. Advantages: Dynamic routing — the supervisor picks the right agent for each task Easy to add new specialists without changing the core architecture Natural fit for heterogeneous model tiers (supervisor uses frontier, specialists use mid-tier) Supports fallback and retry with different agents Disadvantages: Supervisor becomes a bottleneck and single point of failure Inter-agent communication overhead grows with agent count Debugging is harder — you need to trace the supervisor's routing decisions Supervisor context window can balloon with complex multi-agent conversations Production tip: Implement routing as an explicit policy table, not free-form agent selection. Deterministic routing is debuggable, testable, and gives you a clean escalation path: ROUTING_POLICY = { "sql-query": {"agent": "sql-agent", "model": "mid-tier", "max_tokens": 4000}, "code-change": {"agent": "code-agent", "model": "frontier", "max_tokens": 8000}, "policy-lookup": {"agent": "doc-agent", "model": "mid-tier", "max_tokens": 2000}, } def route(request): intent = classify_intent(request) # cheap classifier call policy = ROUTING_POLICY.get(intent) if policy is None: return escalate_to_human(request) # confidence below threshold return dispatch(policy["agent"], request, model=policy["model"], max_tokens=policy["max_tokens"]) Pattern 3: Swarm/Decentralized How it works: Multiple peer agents collaborate without a central coordinator. Agents discover each other's capabilities, negotiate task allocation, and resolve conflicts through established protocols. When to use it: Highly dynamic environments where task decomposition is not known in advance Scenarios requiring agent-to-agent negotiation (e.g., resource allocation, scheduling) Research and exploration tasks where multiple perspectives improve outcomes Systems where the supervisor pattern creates unacceptable latency Real-world example: A multi-agent coding system where a planning agent, a coding agent, a testing agent, and a review agent collaborate in a loop until the code passes all quality gates. No single agent controls the flow — they negotiate through message passing. Advantages: No single point of failure Naturally parallelizable Can discover novel solutions through agent negotiation Scales well with agent count Disadvantages: Hardest to debug — emergent behavior is difficult to trace Token consumption is unpredictable (agents may loop or negotiate extensively) Requires robust error handling and deadlock detection Most protocols (A2A) are still maturing Production tip: Never run a swarm without hard iteration and token budgets. This single guard prevents the most expensive swarm failure mode — negotiation loops that never converge: MAX_ITERATIONS = 10 MAX_TOKENS = 50_000 def run_swarm(task, agents): tokens_used, iterations = 0, 0 state = {"task": task, "status": "open"} while state["status"] == "open": if iterations >= MAX_ITERATIONS or tokens_used >= MAX_TOKENS: return {"status": "escalated", "reason": "budget exhausted"} message = agents[iterations % len(agents)].act(state) tokens_used += message["tokens"] iterations += 1 state = merge(state, message) return state The Protocol Layer: MCP and A2A Architecture patterns define how agents interact within a system. Protocols define how agents interact across systems — with tools, with data sources, and with other agents. MCP (Model Context Protocol) MCP is now Linux Foundation-governed with multimodal support. It standardizes how agents discover and invoke tools and data sources. What MCP solves: Tool discovery — agents can find available tools without hardcoded integrations Tool invocation — standardized request/response format for tool calls Resource access — agents can access files, databases, and APIs through a common interface Prompt management — reusable prompt templates that agents can reference MCP in production means declaring tool servers once and letting any MCP-capable agent use them. A typical client configuration looks like this: { "mcpServers": { "knowledge-base": { "command": "python3", "args": ["-m", "mcp_kb_server", "--index", "internal-docs"], "env": { "KB_READ_ONLY": "true" } }, "sql-warehouse": { "command": "npx", "args": ["-y", "@internal/mcp-sql", "--dsn-env", "WAREHOUSE_DSN"] } } } MCP is ideal for connecting agents to existing enterprise tools: databases, APIs, file systems, and SaaS platforms. It does not solve agent-to-agent communication. When to use MCP: Your agents need to access shared tools and data sources You want to build a reusable tool catalog that multiple agents can use You need standardized tool interfaces across different agent frameworks You are integrating with enterprise systems (ERP, CRM, data platforms) A2A (Agent-to-Agent Protocol) A2A, backed by Google and 50+ partners, standardizes how agents communicate with other agents — including agents running on different platforms or owned by different organizations. What A2A solves: Agent discovery — find agents by capability, not by name or endpoint Task delegation — request work from another agent with structured task descriptions Result exchange — standardize how agent outputs are returned Authentication — verify agent identity and authorization Discovery works through an Agent Card — a JSON document each agent publishes describing its capabilities: { "name": "Financial Analysis Agent", "description": "Analyzes sales and finance CSV datasets", "url": "https://agents.example.com/finance-analyst", "version": "1.2.0", "capabilities": { "streaming": true, "pushNotifications": false }, "skills": [ { "id": "analyze-financial-data", "inputModes": ["application/json"], "outputModes": ["application/json"] } ] } Once discovered, tasks are delegated with a structured payload: { "task": { "id": "analyze-financial-data", "type": "analysis", "input": {"data_source": "sales_q2_2026.csv"}, "expected_output": {"format": "json", "schema": "analysis_result"}, "timeout_seconds": 300 } } When to use A2A: You need agents to delegate work to other agents across system boundaries You are building a multi-agent marketplace or federation You want to enable cross-organization agent collaboration You need agent authentication and audit trails MCP + A2A Together The two protocols are complementary, not competing: ConcernProtocol Agent → ToolMCP Agent → Agent (same system)Direct function calls or MCP Agent → Agent (cross-system)A2A Agent → Data sourceMCP Authentication (agent identity)A2A Authentication (tool access)MCP + OAuth Most production systems will use both: MCP for tool and data access within the system, A2A for cross-system agent delegation. Governance: The Missing Layer The 2026 data is clear: 54% of organizations are actively deploying AI agents (KPMG Q1 2026 AI Pulse Survey), yet governance gaps threaten the payoff — Gartner predicts over 40% of agentic AI projects will be canceled by the end of 2027 due to escalating costs, unclear business value, or inadequate risk controls. The three architecture patterns each have distinct governance requirements. Pipeline Governance Easiest to govern — each step has a defined input, output, and model Implement per-step cost budgets and token limits Log every intermediate output for audit trails Use human-in-the-loop gates at critical decision points Supervisor Governance Moderate complexity — the supervisor's routing decisions need logging and override capability Implement routing policies (which tasks go to which agents, under what conditions) Track supervisor decision quality — is it routing to the right specialist? Set escalation rules when confidence is below threshold Swarm Governance Hardest to govern — emergent behavior requires comprehensive logging Implement hard iteration limits and total token budgets per request Log all inter-agent messages for post-hoc analysis Use governance agents that monitor other agents for policy compliance Universal Governance Principles Regardless of architecture pattern: Least-privilege tool access. Each agent should only access the tools and data it needs. An SEO agent should not have database write access. Audit trails. Log every LLM call, tool invocation, and inter-agent message as structured JSONL. You need this for debugging, compliance, and cost attribution — a few lines of bash are enough to see where the tokens go: # Aggregate token spend per agent from the audit log jq -r '[.agent, .tokens_used] | @tsv' /var/log/agents/llm-calls.jsonl \ | awk -F'\t' '{spend[$1]+=$2} END {for (a in spend) print a, spend[a]}' \ | sort -k2 -nr Human-in-the-loop for critical actions. Publishing content, modifying data, or making financial decisions should require human approval in the initial deployment phase. Cost budgets. Set per-agent and per-request token limits. An agent in a loop should not consume unlimited tokens. Failure isolation. One agent's failure should not cascade to the entire system. Implement circuit breakers and fallback paths. Decision Framework: Which Pattern for Which Workload FactorPipelineSupervisorSwarm Task predictabilityHigh (fixed steps)Medium (dynamic routing)Low (emergent) Cost predictabilityHighMediumLow Debugging easeEasyMediumHard AdaptabilityLowHighVery high ScalabilityLinearModerateHigh Governance maturityMatureDevelopingEarly Best forContent workflows, ETL, report generationEnterprise assistants, complex Q&AResearch, coding, creative collaboration Practical Recommendation For most enterprises starting with multi-agent systems in 2026: Start with a pipeline. It is the easiest to build, debug, and govern. Use it for any workflow with a clear sequential dependency chain. Upgrade to supervisor when needed. When your workload requires dynamic routing (different inputs need different processing), add a supervisor. Keep the individual agents as pipeline stages. Reserve swarm for specific use cases. Only use swarm patterns when you genuinely need decentralized collaboration — and be prepared for higher operational complexity. Key Takeaways Sequential pipelines are the most production-ready pattern for 2026 — predictable cost, easy to debug, simple to govern. Supervisor/orchestrator patterns are ideal for dynamic routing but introduce a single point of failure and require careful context management. Swarm patterns offer maximum flexibility but are the hardest to operate and govern — use them selectively, always with hard budgets. MCP and A2A are complementary protocols — MCP for agent-to-tool access, A2A for agent-to-agent delegation across system boundaries. Governance is not optional — implement least-privilege access, audit trails, cost budgets, and human-in-the-loop gates from day one. The enterprises that succeed with multi-agent systems will be the ones that choose the right architecture for their workload complexity, invest in governance from the start, and optimize costs through heterogeneous model deployment and context management. --- ## Azure AI Landing Zones + Hypervelocity Engineering: The Operating Model for Enterprise AI Platforms URL: https://wenfeng.my/azure-ai-landing-zones-hypervelocity-engineering/ Author: admin Published: 2026-08-08 23:30:01 Modified: 2026-10-04 15:29:16 Azure AI Landing Zones + Hypervelocity Engineering: The Operating Model for Enterprise AI Platforms Every enterprise I work with has the same problem: AI projects are built independently across business units, each one reinventing platform governance from scratch. One team sets up Azure OpenAI with its own VNet, its own monitoring, its own cost tracking. Another team deploys Azure AI Foundry with completely different patterns. A third runs open-source models on AKS with no governance at all. The result is AI sprawl — multiple parallel AI platforms with inconsistent security, duplicate infrastructure, uncoordinated costs, and zero shared learning. On July 17, 2026, Microsoft published a reference architecture that attempts to solve exactly this: combining Azure AI Landing Zones with Hypervelocity Engineering (HVE). It is the most concrete guidance Microsoft has released on enterprise AI platform governance, and it deserves a careful look — both for what it gets right and where enterprises will struggle to apply it. The Problem: AI Sprawl Is an Operating Model Failure The symptoms are consistent across almost every large organization I advise: AI projects are developed independently across business units Platform capabilities evolve slower than AI innovation Governance and security become reactive rather than proactive Infrastructure is treated as a one-time deployment instead of a continuously evolving product Engineering teams spend more time provisioning environments than delivering business value Notice what this list is not. It is not a technology problem. Azure OpenAI works. Azure AI Foundry works. AKS works. The failure is organizational: nobody owns the AI platform as a product, so every team builds its own private version of one. Traditional cloud engineering practices were designed for application modernization — relatively stable workloads with predictable lifecycles. Enterprise AI introduces fundamentally different demands: rapid experimentation, scalable model deployment, secure data access, and continuous compliance in a space where model capabilities and pricing change monthly. Meeting those demands with per-project infrastructure guarantees sprawl. What Hypervelocity Engineering Actually Is HVE is Microsoft's internal engineering operating model — the system Microsoft uses to build and operate Azure services at scale. The important clarification from Microsoft's own announcement: HVE is not another architecture framework or software product. It defines how engineering organizations operate to deliver secure, governed, and continuously improving solutions. Its core philosophy in one sentence: engineer platforms, automate everything practical, measure continuously, and improve through rapid feedback while keeping humans accountable for critical decisions. HVE combines several disciplines that most enterprises already recognize — platform engineering, Infrastructure as Code, Policy as Code, security by design, DevSecOps, continuous observability — into a unified operating model, with AI-assisted engineering layered on top. Six principles anchor it: Outcome-driven engineering — every engineering decision maps to a measurable business outcome, not technology adoption. Platform engineering — reusable, self-service platforms instead of per-project infrastructure. Automation everywhere — provisioning, governance, security validation, testing, and operations all automated where practical. Security by design — identity, network, and deployment pipeline security built in, not bolted on. AI-assisted engineering — AI generates code, templates, and documentation; humans validate and govern. Continuous observability and feedback — telemetry, cost analysis, and user feedback drive the next engineering decision. The shift from traditional delivery is stark: Traditional Engineering Hypervelocity Engineering Project-based delivery Product and platform engineering Manual provisioning Infrastructure as Code Governance after deployment Governance by design Static architecture documents Continuous architecture evolution Infrastructure owned by IT Self-service engineering platforms Periodic releases Continuous delivery Manual operations AI-assisted engineering The mechanism that keeps this moving is RPIR — Research, Plan, Implement, Review — run as a continuous cycle rather than a one-time waterfall. Architecture is never "done"; the Review phase feeds operational reality back into the next Research cycle. The Azure AI Landing Zone as the Reference Implementation The AI Landing Zone is the tangible half of the pattern. It builds on the Cloud Adoption Framework landing zone architecture and adds AI-specific governance layers: Microsoft Entra ID — identity and access for every AI resource Management groups — AI workloads get their own hierarchy, separate from general workloads Azure Policy — AI-specific guardrails enforced as code Hub-and-spoke networking with private endpoints — Azure OpenAI, AI Foundry, and AI Search accessed privately, never over public endpoints Azure Firewall and API Management — controlled egress and API governance Shared AI services — AKS for custom model hosting, Azure AI Foundry and AI Search as shared platform services Azure Monitor and Defender for Cloud — centralized observability and threat protection Infrastructure as Code — Bicep or Terraform as the only sanctioned deployment path The key idea: business units consume platform services through governed self-service. They do not build their own Azure OpenAI deployment; they request quota on the centrally managed one. Here is how RPIR maps onto the architecture activities enterprises already run: RPIR Phase Activities Azure Capabilities Outcome Research Requirements, standards, security baselines, model evaluation Azure AI Search, CAF, Well-Architected Framework Evidence-based decisions Plan Target architecture, ADRs, roadmap Landing zones, Azure Policy, management groups Governed blueprint Implement Deploy infrastructure, policies, AI services, automation Bicep, Terraform, GitHub Actions Repeatable automated deployment Review Validate security, compliance, cost, operations Defender for Cloud, Azure Monitor, Azure Advisor Continuous improvement The difference from classic CAF delivery: Research never stops. In AI, a model generation that did not exist at project kickoff may be the right choice six months later. What This Looks Like in Practice Here is what a Malaysian enterprise implementing this pattern would actually build, with the real artifacts. Step 1: Scaffold the management hierarchy AI workloads get isolated subscriptions for cost attribution, RBAC boundaries, and policy scoping: # AI platform management group under the tenant root az account management-group create \ --name mg-ai-platform \ --display-name "AI Platform" \ --parent <tenant-root-mg-id> az account management-group create \ --name mg-ai-workloads \ --display-name "AI Workloads" \ --parent mg-ai-platform # Hub subscription (platform services) — created via alias az account alias create --name sub-ai-hub \ --display-name "AI Platform Hub" \ --billing-scope "<billing-account-id>" \ --workload Production Step 2: Deploy the AI hub with network isolation Azure OpenAI lives in the hub subscription, locked down with private endpoints and diagnostic logging from day one. This is the architecture-as-code principle in action — the Bicep file is the architecture: param openAiName string param location string = resourceGroup().location param subnetId string param logAnalyticsWorkspaceId string resource openAi 'Microsoft.CognitiveServices/accounts@2024-10-01' = { name: openAiName location: location kind: 'OpenAI' sku: { name: 'S0' } properties: { publicNetworkAccess: 'Disabled' disableLocalAuth: true networkAcls: { defaultAction: 'Deny' } } } resource openAiEndpoint 'Microsoft.Network/privateEndpoints@2024-01-01' = { name: 'pe-${openAiName}' location: location properties: { subnet: { id: subnetId } privateLinkServiceConnections: [ { name: 'openai-connection' properties: { privateLinkServiceId: openAi.id groupIds: [ 'account' ] } } ] } } resource openAiDiag 'Microsoft.Insights/diagnosticSettings@2021-05-01-preview' = { scope: openAi name: 'to-log-analytics' properties: { workspaceId: logAnalyticsWorkspaceId logs: [ { categoryGroup: 'allLogs', enabled: true } ] metrics: [ { category: 'AllMetrics', enabled: true } ] } } Note disableLocalAuth: true — key-based access is off; only Entra ID tokens work. That single property eliminates an entire class of leaked-key incidents. Step 3: Encode governance as Azure Policy The governance layer is where platform-level thinking becomes enforceable. For example, restricting which models business units may deploy — following Microsoft's published policy pattern for Azure OpenAI and Foundry model deployments, with the allow-list keyed on model name,version pairs: { "mode": "All", "policyRule": { "if": { "allOf": [ { "field": "type", "equals": "Microsoft.CognitiveServices/accounts/deployments" }, { "not": { "value": "[concat(field('Microsoft.CognitiveServices/accounts/deployments/model.name'), ',', field('Microsoft.CognitiveServices/accounts/deployments/model.version'))]", "in": "[parameters('allowedModels')]" }} ] }, "then": { "effect": "deny" } }, "parameters": { "allowedModels": { "type": "Array", "defaultValue": [ "gpt-4o,2024-11-20", "gpt-4.1,2025-04-14", "text-embedding-3-large,1" ] } } } Note mode: All — Azure OpenAI deployment sub-resources carry no tags or location, so an Indexed-mode policy would never evaluate them. Assign it at the AI workloads management group so every workload subscription inherits it: az policy definition create \ --name deny-unapproved-openai-models \ --display-name "Deny unapproved Azure OpenAI model deployments" \ --rules model-policy.json az policy assignment create \ --name ai-models-guardrail \ --scope "/providers/Microsoft.Management/managementGroups/mg-ai-workloads" \ --policy deny-unapproved-openai-models \ --params '{"allowedModels":{"value":["gpt-4o,2024-11-20","gpt-4.1,2025-04-14","text-embedding-3-large,1"]}}' Cost control gets the same treatment — a monthly budget on the hub subscription is one CLI call: az consumption budget create \ --budget-name ai-platform-monthly \ --category cost \ --amount 15000 \ --time-grain Monthly \ --start-date 2026-08-01 \ --end-date 2027-07-31 Step 4: Automate the deployment pipeline HVE's "automation everywhere" means the landing zone itself ships through CI/CD. A minimal GitHub Actions workflow with what-if review before apply: name: ai-platform-iac on: push: branches: [ main ] paths: [ 'platform/**' ] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: azure/login@v2 with: client-id: ${{ secrets.AZURE_CLIENT_ID }} tenant-id: ${{ secrets.AZURE_TENANT_ID }} subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }} - name: What-if run: | az deployment group what-if \ --resource-group rg-ai-hub \ --template-file platform/ai-hub/main.bicep - name: Deploy run: | az deployment group create \ --resource-group rg-ai-hub \ --template-file platform/ai-hub/main.bicep Step 5: Close the loop with the Review phase This is the part most enterprises skip — and the part that makes HVE more than a landing zone project. Operational data flows back into architecture decisions: // Token consumption by resource over the last 30 days AzureMetrics | where TimeGenerated > ago(30d) | where ResourceProvider == 'MICROSOFT.COGNITIVESERVICES' | where MetricName in ('GeneratedTokens', 'ProcessedPromptTokens') | summarize Tokens = sum(Total) by Resource, bin(TimeGenerated, 1d) A team whose token spend grows 20% month-over-month while latency degrades triggers a Research phase: move to a cheaper model, add a caching layer, or renegotiate quota. The platform evolves because the data demands it, not because someone scheduled an annual architecture review. Realistic timeline for this build: roughly 8 weeks to a functional platform — two weeks for the management hierarchy and policy baseline, two for hub services and networking, two for governance automation, two for onboarding the first business units. Compare that with the typical 6–12 months of uncoordinated per-project infrastructure builds. Pitfalls: An Honest Assessment What HVE gets right is real: platform-level governance prevents sprawl, architecture-as-code eliminates design-to-deployment drift, and RPIR gives AI platforms a structured cadence for evolution. But implement this in a real enterprise and you will hit these problems: HVE works at Microsoft because Microsoft owns the entire stack. Enterprises have legacy identity estates, hybrid footprints, and matrixed accountability. The reference architecture needs adaptation, not copy-paste. Treat it as a target state, not a migration plan. Continuous validation assumes mature DevOps. If your organization has no CI/CD pipelines, no Infrastructure as Code discipline, and manual change approval boards, the Implement and Review phases of RPIR will stall. Fix the engineering fundamentals first — HVE amplifies whatever maturity you already have. The platform team is a real cost. A centralized AI platform needs 2–4 dedicated engineers for ongoing operations, plus governance tooling overhead. The reference architecture never addresses this. Budget it explicitly or the platform dies a slow death of neglect. Deny-first policy kills self-service. Flip every policy to Deny on day one and business units route around the platform — back to the sprawl you were trying to solve. Start with Audit effect, watch what teams actually try to deploy for two or three cycles, then tighten the deny list based on evidence. The reference architecture is silent on regulated industries. Malaysian financial services, healthcare, and government have residency, sectoral, and audit requirements the guidance does not cover. Data classification rules — which data may be processed by which model — must come from your compliance team, not from a Microsoft template. Do not skip Review. Without the feedback loop, this is just another landing zone project with a fancier name. The RPIR cycle is the entire point: if Research-to-Review takes longer than four weeks, your feedback loop is too slow for the pace AI actually moves at. Key Takeaways Microsoft's AI Landing Zones + Hypervelocity Engineering combines platform governance (the landing zone) with a continuous architecture lifecycle (RPIR) — one without the other is incomplete. HVE's core innovation is architecture-as-code: the design artifact IS the deployment artifact, eliminating design-to-deployment drift. Governance must be enforced as code — Azure Policy definitions, budgets, and diagnostic settings deployed through CI/CD — or it does not exist at scale. The pattern compresses platform foundations from 6–12 months of per-project builds to roughly 8 weeks, but only if DevOps fundamentals are already in place. Start with Audit-mode policies and a funded platform team; start with deny-everything and no owner, and you will recreate the sprawl you set out to fix. If you are building an enterprise AI platform on Azure, this is now the reference pattern to beat. The question worth asking your own leadership is simple: who owns your AI platform as a product — and do they have the mandate, the budget, and the feedback loops to keep it evolving? --- ## Microsoft Fabric Becomes an Agentic AI Platform: What the Build 2026 Announcements Actually Mean URL: https://wenfeng.my/microsoft-fabric-at-build-2026-what-the-agentic-ai-announcements-actually-mean/ Author: admin Published: 2026-08-08 06:30:35 Modified: 2026-10-04 15:29:27 Microsoft Fabric Becomes an Agentic AI Platform — Build 2026 Announcements Microsoft Build 2026 was not a normal feature-drop for Microsoft Fabric. The announcements — Fabric IQ reaching general availability, operations agents going GA, GPU acceleration built directly into Fabric Data Warehouse, a brand-new PostgreSQL-compatible database called Azure HorizonDB, and an open-source SDK called Rayfin for deploying application backends straight onto Fabric — add up to one clear strategic move: Fabric is being repositioned from a data platform into the infrastructure layer for agentic AI on enterprise data. The momentum backs the ambition. Two and a half years after GA, Fabric serves more than 31,000 customers and is the fastest-growing data platform in Microsoft's history. FabCon Europe is already scheduled for Barcelona later in 2026. But if you are an enterprise architect deciding what to do with this, the useful question is not "what did Microsoft announce?" It is "what changes my deployment plan this quarter, and what should I watch from a distance until it matures?" This article is my honest read of the Build 2026 announcements — separating production-ready capability from preview promise, with practical starting points for each. The Problem: Every Agent Starts From Zero Microsoft's framing at Build was blunt: the bottleneck for enterprise AI is no longer model capability. It is context. Every new agent an organization deploys starts from zero — relearning how the business works, where data lives, which definitions are canonical, and what rules apply. A "revenue" metric computed one way in the CRM dashboard and another way in the ERP export is a human annoyance today; it becomes an agent failure mode tomorrow, because agents coordinate and act on that data without a human double-checking. This is the gap the Build 2026 announcements target. The 2026 Microsoft Work Trend Index shows workers moving from asking AI questions to handing off entire tasks and orchestrating multi-agent systems. For that to work at enterprise scale, agents need a shared, governed understanding of the business — not another siloed vector store per project. The Solution: Five Announcements That Matter 1. Fabric IQ, Now Generally Available Fabric IQ is the semantic and operational context layer. It sits on top of OneLake and Power BI's semantic model technology and provides three integrated layers of business context: Unified data — OneLake unifies analytical and operational data into a single accessible layer. Business intelligence — governed semantic models provide the trusted definitions agents should reason over. Operational intelligence — ontologies define business entities, relationships, properties, rules, and actions, and can connect to live signals from Fabric Real-Time Intelligence. Fabric IQ is part of the broader Microsoft IQ family (Work IQ, Fabric IQ, Foundry IQ, and the newly announced Web IQ), and it now extends beyond Fabric itself: ontologies are accessible from Microsoft Foundry as knowledge sources (preview), Fabric IQ integrates with Microsoft Agent 365 as a first-party MCP tool (preview), and graph in Fabric reached GA to support the relationship-first queries this requires. Why it matters: if you run multi-agent systems, Fabric IQ is Microsoft's answer to "how do all my agents share one definition of the business?" That is an architecture-level capability, not a dashboard feature. Maturity note: Fabric IQ core is GA, but ontologies are only expected to reach GA in the coming months. Plan around that. 2. Operations Agents Go GA — Completing the Agent Pair This is the announcement that literally makes Fabric an agentic platform: Operations agents (GA at Build) continuously monitor real-time data, detect patterns or anomalies, and act based on predefined business logic — detect, decide, act, inside governed guardrails. Fabric data agents (GA since FabCon in March 2026) behave like virtual analysts grounded on specific domain data, supporting deeper analysis and delivering insights. Both can be used inside Fabric or as foundational knowledge sources in Microsoft Foundry, Copilot Studio, or Microsoft 365 Copilot. Why it matters: operations agents move Fabric from "platform you query" to "platform that watches and acts." For organizations running dozens of pipelines and real-time feeds, this is a genuine reduction in operational burden — with the usual caveat that autonomous remediation needs tight policy boundaries on day one. 3. GPU-Accelerated Data Warehouse — Decoding the 7x Claim Microsoft built GPU acceleration directly into Fabric Data Warehouse using NVIDIA accelerated computing and custom CUDA kernels — making it the first fully managed data warehouse to offer GPU acceleration. The underlying research won ACM SIGMOD's Best Industry Paper of 2026. The headline number: in Microsoft's internal benchmarking (May 2026), the GPU-accelerated Fabric Data Warehouse delivered up to 7x faster performance relative to three comparable external vendors for reporting and application workloads at 64-user concurrency. UNC Health reports up to 5x query speed improvements in production. Two things make this architecturally interesting: It is automatic. Query acceleration works within Fabric without query rewrites. No new syntax, no re-platforming. It targets concurrency. The gains are for many simultaneous users running reporting and analytical queries — exactly the profile of dashboards and agent-driven analysis, not single-user ETL. Maturity note: early access preview opens in the weeks after Build. Treat the 7x figure as workload-specific marketing until you benchmark your own queries — but the UNC Health field result suggests real gains for concurrent reporting workloads. 4. Azure HorizonDB — PostgreSQL, Built for AI Applications HorizonDB is the new entrant and, for long-term architecture decisions, possibly the most important announcement. It is a fully managed, PostgreSQL-compatible database in public preview with: Elastic storage scaling to 128 TB Scale-out compute up to 3,072 vCores Zone resilience by default, with sub-millisecond multi-zone commit latency Built-in vector search and integrated AI model management Direct connectivity to Microsoft Foundry and Fabric NASDAQ's engineering team put their finger on the value: instead of stitching together transactional storage, a vector database, and AI plumbing as separate components, HorizonDB consolidates them into one platform. My read: HorizonDB is designed for AI-powered applications, not as a drop-in replacement for your OLTP PostgreSQL estate. If you are starting a new RAG or agent-memory project and would otherwise deploy PostgreSQL plus a separate vector store, HorizonDB is now architecturally cleaner and operationally simpler. If your existing Azure Database for PostgreSQL handles application transactions, keep it — and note the new Microsoft Defender for Cloud integration and migration assessment tooling announced for that service instead. 5. Rayfin, Database Hub, and the Developer Surface Three more announcements round out the platform story: Rayfin is a new open-source SDK and CLI that lets developers (and coding agents) describe an application backend — database, authentication, access policies — in code and deploy it directly onto Fabric, with application data landing in OneLake. Microsoft's partnership with Replit signals where this is headed: agents write the app, Fabric ships the backend. It is early, but it positions Fabric as an application runtime, which is new territory. Database Hub (early access) unifies management of Azure SQL, Cosmos DB, Azure Database for PostgreSQL, SQL Server via Azure Arc, Azure Database for MySQL, and Fabric Databases in one control plane — agent-assisted, human-in-the-loop, with Copilot-powered insights. The value is not the unified UI; it is one governance and observability model across the whole database estate. Developer tooling matured fast: Fabric local MCP is GA, Fabric remote MCP is in public preview, Agent Skills for Fabric and Fabric Jumpstart launched as open source, and migration assistants for Azure Data Factory, Synapse Analytics, and Azure SQL entered public preview. Practical Examples: What To Do This Quarter Here is how I would sequence adoption if I were planning a Fabric estate today. Step 1 — Get agents talking to Fabric now (free, low risk). Fabric local MCP is GA and open source. Connect GitHub Copilot or any MCP-capable client to your Fabric tenant: { "mcpServers": { "fabric-mcp-server": { "command": "npx", "args": [ "-y", "@microsoft/fabric-mcp@latest", "server", "start", "--mode", "all" ] } } } Run it via Node.js (npx @microsoft/fabric-mcp, Node 20 LTS or later) or build from source from Microsoft's open-source MCP repository (microsoft/mcp on GitHub, under servers/Fabric.Mcp.Server), authenticate against your tenant, and your coding assistant gains Fabric context — API specifications, item schemas, best practices, and OneLake catalog discovery. Step 2 — Put Fabric skills into your terminal workflow. Agent Skills for Fabric is an open-source set of purpose-built plugins for natural-language Fabric operations: # Clone the open-source skills repo git clone https://github.com/microsoft/skills-for-fabric.git # Skills are designed for GitHub Copilot CLI / agent runtimes — # point your agent at the skills directory, then ask things like: # "show me the top CU-consuming pipelines in workspace prod-analytics" # "list semantic models refreshed in the last 24 hours" This is the fastest way to evaluate whether Fabric IQ-grounded, natural-language operations actually fit your team's workflow before you commit governance effort to ontologies. Step 3 — Plan migrations with the new assistants, not spreadsheets. If you run Azure Data Factory, Synapse Analytics, or Azure SQL workloads, the migration assistants (public preview) convert pipelines, notebooks, Spark pools, and DACPAC-based schemas into Fabric artifacts with AI-assisted compatibility resolution. Even if you do not migrate, run the assessment — it produces a concrete modernization inventory your architecture board can actually review. Step 4 — Size capacity before you sign anything. GPU-accelerated warehouses, ontologies, and operations agents all consume Fabric capacity units. Use the official Fabric SKU Estimator and model your peak concurrency explicitly — the GPU acceleration story is specifically about concurrent users, so your concurrency profile is exactly the variable that determines whether you benefit: # Capacity planning inputs to gather before choosing an F-SKU: # 1. Peak concurrent report/dashboard users (GPU DW benefit driver) # 2. Daily pipeline CU consumption (current ADF/Synapse metered usage) # 3. Real-time event volumes (operations agents + RTI workloads) # 4. Semantic model sizes (Fabric IQ / Direct Lake memory footprint) # # Estimator: https://www.microsoft.com/en-us/microsoft-fabric/capacity-estimator # Trial: https://www.microsoft.com/en-us/microsoft-fabric/getting-started az login az account set --subscription "<your-subscription>" # Fabric capacities are managed via the Fabric admin portal / Power Platform admin; # use the portal to create an F-SKU capacity and attach workspaces to it. Step 5 — Pilot HorizonDB for one new AI application. If any new RAG, agent-memory, or AI-native application project is on your roadmap, stand HorizonDB up in preview and compare it against your default "PostgreSQL plus vector DB" pattern. Judge it on three axes: operational simplicity, latency under your real query mix, and how cleanly it hands data to Fabric and Foundry. Pitfalls: Where This Goes Wrong Pitfall 1 — Confusing preview with GA. Fabric IQ core, operations agents, data agents, graph, and local MCP are GA. Ontologies, Fabric remote MCP, HorizonDB, Rayfin, Database Hub, GPU query acceleration, and the migration assistants are preview or early access. Building a production architecture on the preview list without fallback plans is how Fabric projects stall. Pitfall 2 — Generalizing the 7x benchmark. The 7x figure comes from Microsoft's internal benchmarking against three unnamed vendors, at 64-user concurrency, for reporting and application workloads. If your workload is single-user ETL, nightly batch, or simple lookups, the headline number does not describe your gain. Benchmark your own queries in the early access preview before promising numbers to leadership. Pitfall 3 — Treating HorizonDB as a PostgreSQL migration target. It is PostgreSQL-compatible and built for AI application scale, but migrating an existing OLTP estate is a different exercise than adopting it for a new AI workload. Microsoft even shipped separate Defender and migration tooling for Azure Database for PostgreSQL — a signal that both services will coexist for years. Pitfall 4 — Enabling autonomous agents without policy boundaries. Operations agents act on predefined business logic — the guardrails are only as good as the policies you write. Start with detection and recommendation, add automated remediation only for actions you can roll back, and log every agent action to your audit trail. Pitfall 5 — Skipping the capacity model. Every agentic capability here consumes CU. GPU-accelerated concurrency, ontology reasoning, and always-on operations agents are not free riders on an idle SKU. Organizations that bolt agents onto an undersized capacity hit throttling and cost surprises simultaneously. Model first, then enable. Conclusion Build 2026 marks the point where Microsoft's Fabric story stops being "one lakehouse for analytics" and becomes "the shared data and context foundation for your agent estate." Fabric IQ provides the governed business context, operations and data agents act on it, GPU acceleration keeps concurrent analytics responsive, HorizonDB extends the platform into AI application backends, and Rayfin plus the MCP tooling bring developers and coding agents into the loop. None of this requires a platform migration tomorrow. What it does require is a decision about sequencing: adopt the GA pieces now (Fabric IQ, operations agents, local MCP), pilot the previews against real workloads (GPU acceleration, HorizonDB, migration assistants), and keep your capacity model honest throughout. The enterprises that benefit most will not be the ones that deploy the most agents — they will be the ones that give every agent the same governed understanding of the business first. Key Takeaways Fabric is now positioned as agentic infrastructure, not just analytics. Fabric IQ (GA), operations agents (GA), and data agents (GA) form a closed loop: shared business context, live monitoring, governed action. Read the 7x GPU claim correctly. It is a 64-user-concurrency benchmark for reporting and application workloads with automatic acceleration — powerful for dashboards and agent-driven analytics, irrelevant to batch ETL. Validate it in early access preview with your own queries. HorizonDB belongs in your AI application shortlist, not your OLTP migration plan. 128 TB elastic storage, 3,072 vCores scale-out, built-in vector search, and native Fabric/Foundry connectivity make it compelling for new RAG and agent projects in preview. Developer tooling is the quiet enabler. Fabric local MCP (GA), remote MCP (preview), Agent Skills for Fabric, and migration assistants for ADF/Synapse/Azure SQL lower the cost of evaluating and adopting the whole platform. Sequence adoption by maturity, not excitement. Adopt GA capabilities now, pilot previews on real workloads, write agent policy boundaries before enabling remediation, and model capacity units before scaling anything. --- ## AI Agent Governance in Enterprise — What 2026 Data Reveals About Production Deployments URL: https://wenfeng.my/ai-agent-governance-in-enterprise-what-2026-data-reveals-about-production-deployments/ Author: admin Published: 2026-08-08 02:00:03 Modified: 2026-10-04 15:31:15 AI Agent Governance in Enterprise — What 2026 Data Reveals About Production DeploymentsIf you're reading this from a Kuala Lumpur or Singapore office, chances are your organisation is already experimenting with AI agents — or about to. The latest data confirms what many of us have been seeing on the ground: AI agents have moved decisively from proof-of-concept to production. But here's the uncomfortable truth most vendor presentations skip: nearly half of these deployments are quietly threatening their own success through inadequate governance.I've been building Azure solutions for Malaysian enterprises for over a decade, and the pace of AI agent adoption this year has been unlike anything I've seen. It's also exposing a pattern I keep encountering — brilliant technical implementations that crumble when they hit real-world compliance, security, and operational constraints.Let me walk through what the 2026 data actually tells us, where the governance gaps are, and a practical framework you can implement tomorrow.The Numbers: We're Past the Pilot PhaseThe Ampcome mid-year 2026 report is the one everyone should be reading. The headline number: 54% of enterprises now run AI agents in production — up from just 11% in 2024. That's not a gradual climb. That's a step change.The implications are significant:• 88% of organisations use AI in at least one business function.• 80% report measurable economic impact from AI agents.• 327% growth in multi-agent architectures, where manager agents orchestrate specialist agents.• $207 million is the average projected AI budget for the next 12 months among enterprises.PwC's 2025 survey supports the same trajectory — 88% of executives plan to increase AI budgets due to agentic AI, and 66% report increased productivity from current deployments.Here in Malaysia, the picture is evolving rapidly. Microsoft's AI Diffusion report shows AI adoption rose to 21.8% in Q1 2026, up from 19.7% in H2 2025. The launch of the Malaysia West cloud region has made in-country data residency a reality, and organisations like LHDNM (with the MyInvois platform) and PETRONAS are already leveraging Azure AI at scale.But there's a catch that the headline numbers obscure.The Governance Gap: Nearly Half of Projects Are at RiskHere's the number that should make every architect pause: nearly half of all AI agent projects face governance challenges that threaten their viability.The Ampcome report identifies governance and compliance gaps — missing audit trails, absent permission frameworks, and lack of explainability — as the top blockers for procurement approval. These aren't theoretical concerns. They're the reason projects stall in security reviews and never make it to production.Let me break down the specific risks I see most often in enterprise environments:1. Hallucinations in Business ContextAgent hallucinations aren't just about generating wrong answers. In production, they mean an agent might fabricate a customer's billing history, invent a compliance deadline, or generate a fictional product specification. When an agent has tool access, hallucinations become actions on bad data — invoices sent to wrong recipients, reports built on fabricated numbers.The Ampcome data shows that in tender document processing alone, agents achieve 95% accuracy on standard formats — but that remaining 5% can be catastrophic if it's the one tender that goes to procurement without verification.2. Tool Misuse and Over-Privileged AccessThis is the one I see most frequently. Teams give an agent broad access to APIs "just in case" because LLMs are unpredictable — they might need to call anything. The result is an agent that can read and write to systems far beyond its intended scope.The Microsoft Security Community blog puts it bluntly: "Authorization is not a reasoning problem. It is an identity enforcement problem." Prompt-level instructions for access control are not access control. They're suggestions that a determined prompt injection can override.3. Prompt Injection AttacksIn a multi-agent setup, prompt injection becomes a supply chain attack. Agent A receives a document from Agent B that contains embedded malicious instructions. Agent A trusts Agent B, so it processes the instruction as legitimate. Now Agent A is executing actions the original developer never intended.For Malaysian financial services firms processing documents across systems, this isn't theoretical. It's a compliance violation waiting to happen.4. Cascading Failures in Multi-Agent SystemsThe 327% growth in multi-agent architectures brings a new failure mode: cascading errors. When Agent A makes a wrong decision and passes it to Agent B, which compounds the error and passes it to Agent C, you get a chain reaction that's nearly impossible to trace after the fact. Without audit trails, you can't even identify where the failure started.The Practical Governance FrameworkI'm not going to tell you to "implement a governance framework" and leave it at that. Here's what actually works when you're building on Azure with Azure OpenAI and Copilot Studio — with code you can take back to your team.Layer 1: Identity-First Access ControlStop letting the LLM decide who gets access. Every agent should operate under a managed identity with narrowly scoped permissions.First, create a dedicated managed identity for your agent workload:# Create a user-assigned managed identity for the agent az identity create \ --resource-group rg-ai-agents-prod \ --name id-agent-invoice-processor # Get the principal ID for RBAC assignments az identity show \ --resource-group rg-ai-agents-prod \ --name id-agent-invoice-processor \ --query principalId -o tsvNow assign least-privilege RBAC. The agent that processes invoices should only be able to read from the invoice storage account and write to the processed queue — nothing else:# Grant read-only access to the invoice storage account az role assignment create \ --assignee <principal-id> \ --role "Storage Blob Data Reader" \ --scope "/subscriptions/<sub-id>/resourceGroups/rg-finance/providers/Microsoft.Storage/storageAccounts/stinvoices" # Grant write access ONLY to the processing queue az role assignment create \ --assignee <principal-id> \ --role "Storage Queue Data Message Sender" \ --scope "/subscriptions/<sub-id>/resourceGroups/rg-finance/providers/Microsoft.Storage/storageAccounts/stagentqueues/queues/invoice-processed"Notice what's missing: no Contributor role. No "in case it needs it" broad access. If the agent needs something else, that's a separate deployment with its own identity and its own scope.Layer 2: Azure OpenAI Content Safety ConfigurationConfigure content filtering at the resource level, not the application level. This ensures every request is filtered regardless of which team or application is calling it:# Deploy Azure OpenAI with content filtering az cognitiveservices account create \ --name aoai-agent-prod \ --resource-group rg-ai-agents-prod \ --kind OpenAI \ --sku S0 \ --location southeastasia \ --custom-domain aoai-agent-prod # Configure content filters via REST (severity thresholds) az rest --method PUT \ --uri "https://management.azure.com/subscriptions/<sub-id>/resourceGroups/rg-ai-agents-prod/providers/Microsoft.CognitiveServices/accounts/aoai-agent-prod/contentFilters/custom?api-version=2024-10-01" \ --body '{ "name": "strict-enterprise-filter", "properties": { "contentFilters": [ {"name": "Hate", "severity": 2, "enabled": true}, {"name": "Sexual", "severity": 2, "enabled": true}, {"name": "Violence", "severity": 2, "enabled": true}, {"name": "SelfHarm", "severity": 2, "enabled": true} ], "protectedMaterialText": {"enabled": true}, "blockedKeywords": ["confidential", "classified"] } }'The severity setting of 2 means medium-and-above severity content gets blocked. Tune this to your risk appetite, but start conservative and loosen based on monitoring data.Layer 3: Authorization-Aware Agent Pattern with Entra IDFor Copilot Studio agents, the Microsoft-recommended pattern is to keep authorization logic outside the LLM entirely. Power Automate acts as the enforcement layer:User Request → Copilot Studio Agent → Power Automate Flow → ↓ (validates via Microsoft Graph) ↓ Checks Entra ID group membership ↓ Normalizes group names ↓ Compares against approved RBAC groups → Authorized? Execute action → Denied? Terminate with audit logHere's a simplified Power Automate expression for the authorization check:// After retrieving user groups via Microsoft Graph @contains( join( toLower( body('Get_User_Groups')?['value'] ), ',' ), 'ai-invoice-processor-users' )The key insight: the LLM never sees the authorization decision logic. It can't be prompt-injected into granting itself access. The identity system makes the call.Layer 4: Audit Trails That Actually Survive ScrutinyEvery agent action should generate an immutable audit record. For Azure-based deployments, use Azure Monitor with Log Analytics:# Create a Log Analytics workspace for agent audit logs az monitor log-analytics workspace create \ --resource-group rg-ai-agents-prod \ --workspace-name law-agent-audit \ --location southeastasia \ --retention-time 2555 # Enable diagnostic logging for the OpenAI resource az monitor diagnostic-settings create \ --name "agent-audit-logs" \ --resource "/subscriptions/<sub-id>/resourceGroups/rg-ai-agents-prod/providers/Microsoft.CognitiveServices/accounts/aoai-agent-prod" \ --workspace "/subscriptions/<sub-id>/resourceGroups/rg-ai-agents-prod/providers/Microsoft.OperationalInsights/workspaces/law-agent-audit" \ --logs '[ {"category": "RequestResponse", "enabled": true, "retentionPolicy": {"enabled": true, "days": 730}}, {"category": "Trace", "enabled": true, "retentionPolicy": {"enabled": true, "days": 730}}, {"category": "Audit", "enabled": true, "retentionPolicy": {"enabled": true, "days": 2555}} ]' \ --metrics '[ {"category": "AllMetrics", "enabled": true, "retentionPolicy": {"enabled": true, "days": 30}} ]'The 2555-day (7-year) retention on Audit logs aligns with Malaysia's PDPA requirements and most financial services regulatory mandates. The RequestResponse logs at 2 years give you the traceability you need for incident investigation.Layer 5: Policy-as-Code with Azure PolicyThis is where governance scales. Define your constraints as policy and enforce them automatically:{ "properties": { "displayName": "AI Agent Managed Identity Required", "policyType": "Custom", "mode": "All", "parameters": {}, "policyRule": { "if": { "allOf": [ { "field": "type", "equals": "Microsoft.CognitiveServices/accounts" }, { "field": "tags.environment", "equals": "production" } ] }, "then": { "effect": "auditIfNotExists", "details": { "type": "Microsoft.ManagedIdentity/userAssignedIdentities", "existenceCondition": { "field": "tags.workload", "equals": "ai-agent" } } } } } }Deploy this policy to enforce that every production AI resource must have a user-assigned managed identity tagged as an AI agent workload. Resources without proper identity assignment get flagged in compliance reports before they can be used.# Deploy the policy definition az policy definition create \ --name "require-agent-identity" \ --display-name "Require Managed Identity for AI Agent Resources" \ --rules @policy-agent-identity.json \ --mode All # Assign to the AI agents resource group az policy assignment create \ --name "enforce-agent-identity" \ --policy "require-agent-identity" \ --scope "/subscriptions/<sub-id>/resourceGroups/rg-ai-agents-prod" \ --enforcement-mode DefaultCommon Pitfalls I've Seen in Malaysian EnterprisesAfter working with several organisations on Azure AI deployments, here are the patterns that trip people up:Pitfall 1: Using one "super agent" instead of scoped specialists. Teams build a single agent that can do everything — query databases, send emails, update records. This is the fastest path to a governance nightmare. Build narrow, purpose-built agents and let a coordinator agent orchestrate them. Yes, it's more work upfront, but each agent is independently auditable and independently permissioned.Pitfall 2: Relying on prompts for security controls. I've seen agents with system prompts that say "you must not access customer data." That's not security. That's a suggestion. Enforce access boundaries at the identity and network layer.Pitfall 3: Skipping the audit trail because "we'll add it later." You won't. And when your compliance team or PDPA auditor asks for logs of what the agent did, you'll wish you had them. Build audit logging from day one — it's a few lines of Azure CLI as shown above.Pitfall 4: Treating multi-agent systems as a single trust boundary. When Agent A delegates to Agent B, that delegation should carry the original user's identity and permissions — not Agent A's elevated privileges. This is the principle of "delegated identity" and it's non-negotiable in production.Pitfall 5: Ignoring the Malaysia-specific regulatory landscape. Malaysia's AI governance framework is expected to be submitted to Cabinet in June 2026. The PDPA already applies to AI-processed personal data. Organisations that wait for the framework to be finalised before starting governance work will be playing catch-up. Start now with the fundamentals: data residency, consent management, and audit trails.Conclusion: Governance Is Not the Opposite of SpeedI hear it all the time: "Governance slows us down." In my experience, it's the opposite. The organisations that have clear governance frameworks are the ones scaling fastest because they've already solved the blocking issues — security review, compliance approval, audit readiness — before they become bottlenecks.The Ampcome report confirms this: organisations that built governance before scaling report faster time-to-value (median ≤6 months) than those that bolt it on after.For Malaysian enterprises running Azure OpenAI and Copilot Studio, the tooling is already there. Managed identities, RBAC, content filtering, audit logs, policy-as-code — these aren't exotic capabilities. They're built-in features waiting to be configured correctly.The question isn't whether you can afford to implement governance. It's whether you can afford the 50% failure rate that comes without it.---Key Takeaways1. 54% of enterprises are in production, but governance is the #1 blocker — audit trails, permission frameworks, and explainability are prerequisites for procurement approval, not nice-to-haves.2. Identity enforcement beats prompt-level security every time — use managed identities, Entra ID RBAC, and Power Automate authorization flows to keep access decisions outside the LLM.3. Least-privilege is non-negotiable for agents — scope each agent's access to exactly the resources it needs, and nothing more. One agent per workload, one identity per agent.4. Build audit trails from day one — configure Azure Monitor diagnostic settings with 7-year retention on audit logs. PDPA and financial regulators will ask for this data; don't be the team that can't produce it.5. Start with one workflow, then scale — prove governance works on a single high-validated use case (invoice processing, customer support tier-1) before expanding to multi-agent architectures. --- ## Malaysia PDPA Compliance and Data Sovereignty: A Practical Cloud Architecture Guide for 2026 URL: https://wenfeng.my/malaysia-pdpa-compliance-data-sovereignty-cloud-architecture-2026/ Author: admin Published: 2026-08-05 06:30:35 Modified: 2026-10-04 15:31:16 Malaysia PDPA Compliance and Data Sovereignty: A Practical Cloud Architecture Guide for 2026Malaysia's Personal Data Protection Act (PDPA) is entering a new enforcement phase in 2026. Cross-border data transfer restrictions are tightening. Mandatory breach notifications are no longer theoretical — the PDPC (Personal Data Protection Commissioner) has issued its first enforcement notices. And enterprises running multi-cloud environments are struggling to answer a simple question: where is our data, and does it stay where it should?I have been designing cloud architectures for Malaysian enterprises for the last decade. The compliance conversation has shifted from "should we worry about PDPA?" to "how do we prove to auditors that we comply?" Here is the practical architecture.The 2026 PDPA RequirementsThe key requirements that affect cloud architecture:1. Data ResidencyPDPA does not mandate all data stay in Malaysia. The actual requirement is more nuanced:Sensitive personal data (health, financial, biometric) should be processed and stored in Malaysia or in jurisdictions with equivalent protectionNon-sensitive personal data can be processed cross-border, but the data controller must ensure equivalent protectionGovernment-linked data must stay in-country (per MyDIGITAL guidelines)2. Cross-Border Transfer RestrictionsBefore transferring personal data overseas, you must ensure the receiving country provides comparable protection. In practice, this means:Azure/Microsoft: Malaysia Central and Malaysia East regions satisfy residency requirementsAWS: AWS Asia Pacific (Kuala Lumpur) Region satisfies residencyGoogle Cloud: No local region yet — reliance on Singapore for Southeast Asia3. Mandatory Breach NotificationOrganizations must notify the PDPC within 72 hours of discovering a personal data breach. This requires: - Breach detection infrastructure - Assessment workflows - Notification templates and processes - Evidence preservationCloud Provider Data Residency Options Provider Malaysia Region Status Compliance Certifications Azure Malaysia Central (Johor) GA PDPA, ISO 27001, SOC 2 Azure Malaysia East (Kuala Lumpur) GA PDPA, ISO 27001, SOC 2 AWS Asia Pacific (Kuala Lumpur) GA PDPA, ISO 27001 Google Cloud No Malaysia region — Singapore fallback Oracle Malaysia (planned) Preview — My recommendation: Azure or AWS for primary workloads requiring data residency. Use Google Cloud only for non-personal-data workloads or with explicit cross-border transfer mechanisms.The ArchitectureHere is a production architecture for a Malaysian enterprise with PDPA compliance requirements:┌─────────────────────────────────────────────────────────┐ │ Azure Malaysia Central │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ Data Zone │ │ App Zone │ │ DMZ Zone │ │ │ │ (VNet 10.0) │ │ (VNet 10.1) │ │ (VNet 10.2) │ │ │ │ │ │ │ │ │ │ │ │ Azure SQL │ │ App Service │ │ API Gateway │ │ │ │ Blob Storage │ │ Functions │ │ Front Door │ │ │ │ Key Vault │ │ Container │ │ │ │ │ │ Purview │ │ Apps │ │ │ │ │ └──────────────┘ └──────────────┘ └──────────────┘ │ │ │ │ │ │ │ └────────────────┼──────────────────┘ │ │ │ │ │ Azure Policy + Blueprints │ │ (region enforcement) │ └─────────────────────────────────────────────────────────┘ │ │ VPN/ExpressRoute (encrypted) │ ┌─────────────────────────────────────────────────────────┐ │ Cross-Border Transfer Layer │ │ ┌────────────────────────────────────────────────────┐ │ │ │ Azure Data Factory / Integration Account │ │ │ │ • Data classification via Purview │ │ │ │ • Transfer approval workflow │ │ │ │ • Encryption in transit and at rest │ │ │ │ • Audit trail for every transfer │ │ │ └────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ Global/Azure East US (non-sensitive analytics) │ └─────────────────────────────────────────────────────────┘ Zone DesignData Zone (VNet 10.0.x.x): All personal data processing and storage. Azure SQL, Cosmos DB, Blob Storage, Key Vault. Peered VNet with no direct internet access.App Zone (VNet 10.1.x.x): Application compute. App Service, Container Apps, Functions. Accesses Data Zone through Private Endpoints only.DMZ Zone (VNet 10.2.x.x): External-facing services. API Gateway, Azure Front Door, WAF. No personal data stored here.Azure Policy for Residency Enforcement{ "if": { "allOf": [ { "field": "type", "equals": "Microsoft.Sql/servers/databases" }, { "field": "location", "notIn": ["malaysiacentral", "malaysiaeast"] } ] }, "then": { "effect": "deny", "details": { "message": "PDPA compliance: SQL databases must be deployed to Malaysia Central or Malaysia East." } } } Apply this as an Azure Policy initiative across your management group. Every database deployment to a non-Malaysia region is automatically blocked.# Deploy the policy initiative az policy initiative create \ --name "pdpa-data-residency" \ --display-name "PDPA Data Residency Requirements" \ --description "Enforce data residency for PDPA-regulated workloads" \ --policy-definition-file pdpa-policy-definition.json \ --params pdpa-policy-params.json # Assign at the subscription level az policy assignment create \ --name "pdpa-malaysia-central" \ --policy-initiative "pdpa-data-residency" \ --scope "/subscriptions/<subscription-id>" \ --params '{"allowedLocations": {"value": ["malaysiacentral", "malaysiaeast"]}}' Data Classification PipelineYou cannot protect data you do not know about. Azure Purview (now Microsoft Purview) classifies and labels personal data automatically:# Enable Purview on the storage account az purview account create \ --name purview-compliance \ --resource-group rg-compliance \ --location malaysiacentral \ --managed-resource-group purview-managed-rg # Configure classification rules for PDPA-sensitive data az purview classification-rule create \ --account-name purview-compliance \ --resource-group rg-compliance \ --rule-name "PDPA-Personal-Data" \ --rule-type system \ --classifiers '[{ "name": "Malaysian IC Number", "pattern": "\\d{6}-\\d{2}-\\d{4}", "sensitivityLabel": "Highly Confidential" }, { "name": "Malaysian Phone Number", "pattern": "\\+60\\d{9,10}", "sensitivityLabel": "Confidential" }]' Breach Notification ArchitectureWhen a breach occurs, you need to detect, assess, and notify within 72 hours. Build this into your security infrastructure:# Azure Sentinel (Microsoft Sentinel) alert rule for data exfiltration az monitor log-analytics workspace alert-rule create \ --workspace-name workspace-security \ --resource-group rg-security \ --name "pdpa-breach-detection" \ --display-name "PDPA Breach Detection" \ --description "Detects potential personal data exfiltration" \ --severity 1 \ --enabled true \ --query ' AzureDiagnostics | where Category == "StorageBlobLogs" | where operationName_s == "GetBlob" or operationName_s == "DownloadBlob" | where userAgent_s !~ "Microsoft.Azure.Search" | summarize count() by caller_ip_s, bin(TimeGenerated, 1h) | where count_ > 100 ' The 72-hour workflow:Hour 0-4: Detection → Alert fires → SOC triages Hour 4-12: Assessment → Scope of breach → Affected individuals Hour 12-24: Containment → Isolate affected systems → Preserve evidence Hour 24-48: Notification draft → Legal review → PDPC notification form Hour 48-72: Submit notification → Notify affected individuals → Document The Gap Between PDPA and Cloud Provider GuaranteesHere is what cloud providers actually guarantee versus what PDPA requires: PDPA Requirement Azure Guarantee Gap Data stays in Malaysia Azure Policy region locks You must configure and enforce Breach notification support Microsoft notifies for their breaches You must notify for YOUR breaches Data subject access requests No built-in DSAR tool Build custom tooling Consent management No built-in consent platform Third-party or custom build Data minimization No enforcement Architectural discipline The critical gap: Cloud providers protect their infrastructure. They do not protect you from your own misconfigurations. A storage account with public access in Malaysia Central is still a PDPA violation — the data is in the right region, but it is exposed.Cost Considerations Component Monthly Cost (MYR) Notes Azure SQL (General Purpose, 4 vCores) ~RM 4,500 Malaysia Central pricing Azure Policy + Blueprints Free Governance tools Microsoft Purview ~RM 2,000 Based on data catalog size Microsoft Sentinel ~RM 3,500 100GB/day ingestion ExpressRoute (1 Gbps) ~RM 3,800 For hybrid connectivity Total baseline ~RM 13,800/month Does not include compute Key TakeawaysPDPA does not mandate all data in Malaysia, but sensitive data and government-linked data must stay in-country. Design your architecture with explicit data classification and residency zones.Azure Policy is your enforcement mechanism. Do not rely on process alone — use policy-as-code to block non-compliant deployments automatically.Microsoft Purview classifies personal data automatically. You cannot protect what you do not know about. Enable classification before enabling encryption and access controls.The 72-hour breach notification requires pre-built infrastructure. Detection, assessment, containment, and notification workflows must be operational before a breach occurs, not built during one.Cloud providers protect their infrastructure, not your misconfigurations. Public storage accounts, over-privileged IAM roles, and unencrypted data are your responsibility regardless of region.ResourcesPDPA Malaysia Official TextAzure Malaysia RegionsMicrosoft Purview Data ClassificationAzure Policy for Data Residency --- ## Microsoft Fabric Real-Time Intelligence: Eventhouse, KQL, and Streaming Analytics Without the Infrastructure URL: https://wenfeng.my/microsoft-fabric-real-time-intelligence-eventhouse-kql-streaming-analytics/ Author: admin Published: 2026-08-04 23:30:48 Modified: 2026-10-04 15:29:37 Microsoft Fabric Real-Time Intelligence: Eventhouse, KQL, and Streaming Analytics Without the InfrastructureFabric Real-Time Intelligence went GA with a clear promise: run a full streaming analytics pipeline — ingestion, storage, query, and alerting — without managing Kafka, Flink, or dedicated Kusto clusters. Eventhouses store billions of events with KQL query performance, CDC replication support, and native integration with Fabric Lakehouses for historical analysis.The question is not whether it works. It does. The question is when to use it versus the alternatives you may already have.What Fabric Real-Time Intelligence Actually Is┌─────────────────────────────────────────────────┐ │ Fabric Real-Time Intelligence │ │ │ │ ┌────────────┐ ┌──────────────────────┐ │ │ │ Eventhouse │───▶│ KQL Database │ │ │ │ (storage) │ │ (query engine) │ │ │ └────────────┘ └──────────────────────┘ │ │ │ │ │ │ │ ▼ │ │ │ ┌──────────────────────┐ │ │ │ │ KQL Queries │ │ │ │ │ (real-time dashboards)│ │ │ │ └──────────────────────┘ │ │ │ │ │ ▼ │ │ ┌────────────────────────────────────────┐ │ │ │ Activator (alerting & automation) │ │ │ │ → Trigger Power Automate │ │ │ │ → Send Teams/Email alerts │ │ │ │ → Execute Logic Apps │ │ │ └────────────────────────────────────────┘ │ │ │ │ ┌────────────────────────────────────────┐ │ │ │ Fabric Lakehouse (historical) │ │ │ │ ← CDC replication from Eventhouse │ │ │ └────────────────────────────────────────┘ │ └─────────────────────────────────────────────────┘ Eventhouse is the storage and compute engine. It stores events in a columnar format optimized for time-series queries. Think of it as Azure Data Explorer (Kusto) integrated into Fabric.KQL Database provides the query language. Kusto Query Language is purpose-built for log analytics, telemetry, and time-series data. It is faster than SQL for these workloads because it is designed for them.Activator is the alerting layer. It monitors query results and triggers actions when conditions are met — no custom polling code required.Lakehouse Integration means Eventhouse data can be replicated to a Fabric Lakehouse for historical analysis, combining real-time and batch data in one platform.When to Use Fabric Real-Time IntelligenceUse Fabric RTI when:You are already on Fabric. The integration with Lakehouse, Power BI, and the rest of the Fabric ecosystem is the primary value. No ETL between streaming and batch layers.You need real-time dashboards on operational data. KQL queries on Eventhouse return in milliseconds for billions of rows. Power BI DirectQuery on KQL makes this instant.You want managed alerting without building infrastructure. Activator handles threshold monitoring, anomaly detection, and automated response without custom code.Your data is already in KQL or can be ingested via standard connectors. Eventhouse supports Event Hubs, IoT Hub, OPC UA, and REST API ingestion.Do NOT use Fabric RTI when:You need complex event processing (CEP). Eventhouse stores and queries events but does not implement stream processing topologies (windowing, joins across streams, stateful processing). Use Azure Stream Analytics or Flink for that.You need exactly-once processing guarantees. Eventhouse provides at-least-once delivery. For financial transactions requiring exactly-once semantics, use Event Hubs + custom processing.You need multi-region write. Eventhouse is single-region. For globally distributed streaming, use Cosmos DB or Event Hubs with Geo-DR.Fabric RTI vs Azure Data Explorer vs Event Hubs + Stream Analytics Criteria Fabric RTI Azure Data Explorer Event Hubs + Stream Analytics Infrastructure management None (Fabric-managed) Cluster provisioning Multiple services Query language KQL KQL SQL-like (Stream Analytics) Storage model Eventhouse (columnar) ADX tables Custom (you manage) Cost model Fabric CU-based ADX compute units EH throughput + SA streaming units Integration Fabric native Standalone + Kusto Query Azure-native Real-time dashboards Power BI native Power BI + ADX connector Power BI + custom Alerting Activator (built-in) Custom logic apps Built-in alerts CDC support GA (April 2026) Supported Custom Best for Fabric shops Independent streaming analytics Complex event processing My decision rule:def choose_streaming_platform( on_fabric: bool, needs_cep: bool, needs_exactly_once: bool, budget_conscious: bool ) -> str: """Select the streaming analytics platform.""" if needs_cep: return "Event Hubs + Stream Analytics" if needs_exactly_once: return "Event Hubs + Custom Processing" if on_fabric: return "Fabric Real-Time Intelligence" if budget_conscious: return "Azure Data Explorer" return "Fabric Real-Time Intelligence" KQL Query PatternsHere are the most common KQL patterns for production workloads:// Time-series aggregation: requests per minute over last hour requests | where timestamp > ago(1h) | summarize request_count = count() by bin(timestamp, 1m), status_code | render timechart // Anomaly detection: find unusual error spikes requests | where timestamp > ago(24h) | where status_code >= 500 | summarize error_count = count() by bin(timestamp, 5m) | extend anomalies = series_decompose_anomalies(error_count, 1.5, -1, 'linefit') | where arraylength(anomalies) > 0 // CDC replication: track changes in source database changes | where source_table == "customer_orders" | where operation == "insert" or operation == "update" | project timestamp, operation, primary_key, changed_column, new_value // Top-N with percentile: slowest endpoints requests | where timestamp > ago(1h) | summarize avg_duration = avg(duration_ms), p95_duration = percentile(duration_ms, 95), request_count = count() by endpoint | top 10 by p95_duration desc Activator Alerting Patterns// Alert: Error rate exceeds 5% in any 5-minute window requests | where timestamp > ago(5m) | summarize total = count(), errors = countif(status_code >= 500) | extend error_rate = errors * 100.0 / total | where error_rate > 5.0 Configure Activator to: 1. Monitor this query every 5 minutes 2. When results are non-empty (threshold exceeded): - Send Teams alert to #ops-alerts channel - Create incident in ServiceNow - Execute runbook to scale up computeThis replaces custom polling, monitoring dashboards, and alert scripts.Cost ModelingFabric Real-Time Intelligence costs are based on Fabric Capacity Units (CUs): Workload Estimated CU/Hour Monthly Cost (RM) Light ingestion (1K events/sec) 2-4 CU ~RM 800-1,600 Medium ingestion (10K events/sec) 8-16 CU ~RM 3,200-6,400 Heavy ingestion (100K events/sec) 32-64 CU ~RM 12,800-25,600 Query-heavy (complex KQL dashboards) 4-8 CU additional ~RM 1,600-3,200 Compare with Azure Data Explorer: | Workload | ADX Cost (RM/month) | |----------|---------------------| | Light (D11 node) | ~RM 4,500 | | Medium (D12 node) | ~RM 9,000 | | Heavy (D13 node) | ~RM 18,000 |Fabric RTI is cost-competitive for light-to-medium workloads, especially when you factor in the eliminated infrastructure management.Migration from Azure Data ExplorerIf you are already running ADX and considering migration to Fabric RTI:# 1. Export ADX table data to ADLS Gen2 az kusto data-connection event-hub create \ --cluster-name adx-cluster \ --database-name telemetry \ --name adls-export \ --storage-account storagemain \ --storage-container fabric-import \ --data-format json # 2. Create Eventhouse in Fabric and configure ingestion from ADLS # (via Fabric UI: Eventhouse → New → Data connection → ADLS Gen2) # 3. Redirect applications to Fabric RTI endpoint # Update connection strings from ADX cluster to Eventhouse # 4. Validate query results match # Run parallel queries on both ADX and Eventhouse, compare results Migration timeline: 2-4 weeks for a typical ADX cluster with 5-10 tables and moderate data volume.Key TakeawaysFabric Real-Time Intelligence eliminates infrastructure management for streaming analytics. Eventhouse, KQL, and Activator are fully managed — no cluster provisioning, scaling, or patching.Use Fabric RTI when you are already on Fabric. The integration with Lakehouse, Power BI, and the Fabric ecosystem is the primary value. Do not adopt it as a standalone streaming platform.Eventhouse is not a replacement for complex event processing. Use Stream Analytics or Flink for windowed joins, stateful processing, and exactly-once guarantees.Activator replaces custom alerting infrastructure. Monitor KQL query results and trigger actions without polling scripts or custom monitoring code.CDC replication (GA April 2026) bridges real-time and historical data. Operational databases can feed Eventhouse for real-time analysis and Lakehouse for historical analytics.ResourcesFabric Real-Time Intelligence OverviewEventhouse DocumentationKQL Query ReferenceActivator Documentation --- ## Microsoft Fabric OneLake Security: RBAC, Row-Level Security, and Purview Governance at Scale URL: https://wenfeng.my/microsoft-fabric-onelake-security-rbac-row-level-purview-governance/ Author: admin Published: 2026-08-04 06:30:17 Modified: 2026-10-04 15:29:57 Microsoft Fabric OneLake Security: RBAC, Row-Level Security, and Purview Governance at ScaleThe question that killed Fabric adoption in 2025 was simple: "How do I prevent Team A from seeing Team B's data?" Workspace-level permissions were the only option, and they were too coarse for any organization with sensitive data.OneLake security went GA in early 2026 with granular folder-level and table-level RBAC, shortcuts-aware governance policies, and REST APIs for automated role management. This is the missing piece that makes Fabric viable for enterprises.The OneLake Security Model┌──────────────────────────────────────────────────┐ │ OneLake │ │ ┌────────────────────────────────────────────┐ │ │ │ Fabric Workspace │ │ │ │ ┌──────────────┐ ┌──────────────────┐ │ │ │ │ │ Lakehouse A │ │ Lakehouse B │ │ │ │ │ │ │ │ │ │ │ │ │ │ Folder: /raw │ │ Folder: /sensitive│ │ │ │ │ │ (team-shared) │ │ (restricted) │ │ │ │ │ │ │ │ │ │ │ │ │ │ ACL: Team A │ │ ACL: Team B only │ │ │ │ │ └──────────────┘ └──────────────────┘ │ │ │ │ │ │ │ │ Workspace ACL: Analysts (read) │ │ │ │ Folder ACL: Override workspace (restricted)│ │ │ └────────────────────────────────────────────┘ │ │ │ │ ┌────────────────────────────────────────────┐ │ │ │ Shortcut: /lakehouse-a/external-data │ │ │ │ → Points to ADLS Gen2 / S3 bucket │ │ │ │ → Inherits governance from Lakehouse │ │ │ │ → Can have independent ACL │ │ │ └────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────┘ Key Design PrinciplesInheritance with override. Folder ACLs inherit from workspace ACLs but can be overridden at any level. A user with workspace-level read access can be denied access to a specific folder.Shortcuts respect governance. When you create a shortcut to external data (ADLS Gen2, S3), the shortcut inherits the governance policy of the parent Lakehouse. You can also apply independent ACLs to the shortcut itself.REST API for CI/CD. All role assignments are manageable via REST API, enabling automated governance in your deployment pipelines.RBAC ImplementationBuilt-in Roles Role Scope Permissions Admin Workspace Full control, manage permissions Contributor Workspace/Lakehouse Read, write, create artifacts Member Workspace Read, write (no permission management) Viewer Workspace/Folder Read-only None — No access (explicit deny) Custom Role via REST API# Grant folder-level access via OneLake Security REST API curl -X POST "https://api.fabric.microsoft.com/v1/workspaces/{workspace-id}/lakehouses/{lakehouse-id}/security/permissions" \ -H "Authorization: Bearer {token}" \ -H "Content-Type: application/json" \ -d '{ "principal": { "id": "{security-group-id}", "type": "SecurityGroup" }, "role": "Viewer", "scope": { "type": "Folder", "path": "/sensitive/financial-data" } }' # Grant table-level access curl -X POST "https://api.fabric.microsoft.com/v1/workspaces/{workspace-id}/lakehouses/{lakehouse-id}/security/permissions" \ -H "Authorization: Bearer {token}" \ -H "Content-Type: application/json" \ -d '{ "principal": { "id": "{user-id}", "type": "User" }, "role": "Contributor", "scope": { "type": "Table", "name": "customer_pii" } }' Terraform Automation# Automate OneLake security via Fabric REST API + Terraform null_resource resource "null_resource" "onelake_security" { for_each = var.folder_permissions triggers = { folder_path = each.value.folder_path principal = each.value.principal_id role = each.value.role } provisioner "local-exec" { command = <<-EOT curl -X POST "${var.fabric_api_url}/workspaces/${var.workspace_id}/lakehouses/${var.lakehouse_id}/security/permissions" \ -H "Authorization: Bearer ${var.fabric_token}" \ -H "Content-Type: application/json" \ -d '${jsonencode({ principal = { id = each.value.principal_id, type = each.value.principal_type } role = each.value.role scope = { type = "Folder", path = each.value.folder_path } })}' EOT } } variable "folder_permissions" { type = map(object({ folder_path = string principal_id = string principal_type = string role = string })) default = { "finance-team-raw" = { folder_path = "/finance/raw-data" principal_id = "sec-group-finance" principal_type = "SecurityGroup" role = "Contributor" } "finance-team-sensitive" = { folder_path = "/finance/sensitive" principal_id = "sec-group-finance-leads" principal_type = "SecurityGroup" role = "Viewer" } } } Row-Level and Column-Level SecurityOneLake security handles folder and table access. For finer-grained control within tables, use T-SQL security features in Fabric Data Warehouse:-- Row-Level Security: Users see only their region's data CREATE FUNCTION security.fn_region_access(@region NVARCHAR(50)) RETURNS TABLE WITH SCHEMABINDING AS RETURN SELECT 1 AS result WHERE @region = SYSTEM_USER_CONTEXT() OR IS_MEMBER('db_owner'); CREATE SECURITY POLICY region_filter ADD FILTER PREDICATE security.fn_region_access(region) ON dbo.sales_data WITH (STATE = ON); -- Column-Level Security: Hide sensitive columns from non-privileged users GRANT SELECT ON dbo.sales_data(region, product, amount) TO analyst_role; GRANT SELECT ON dbo.sales_data(region, product, amount, customer_ssn, customer_email) TO compliance_role; Purview Integration for Automated ComplianceMicrosoft Purview classifies data automatically and applies governance policies:from azure.purview.scanning import PurviewScanningClient from azure.identity import DefaultAzureCredential # Register a Fabric Lakehouse as a Purview data source purview_client = PurviewScanningClient( endpoint="https://your-purview.purview.azure.com/", credential=DefaultAzureCredential() ) # Create a scan rule for PDPA classification scan_rule = { "name": "PDPA-Classification", "type": "SystemScanRuleset", "dataSourceType": "AzureDataLakeStorage", "customRules": [ { "ruleName": "Malaysian-IC", "columnName": "*", "regexPattern": "\\d{6}-\\d{2}-\\d{4}", "sensitivityLabel": "Highly Confidential" }, { "ruleName": "Email-Address", "columnName": "*", "regexPattern": "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}", "sensitivityLabel": "Confidential" } ] } The compliance pipeline:OneLake Data → Purview Scan → Classification → Sensitivity Labels → └── Auto-apply RBAC based on label ├── "Highly Confidential" → Restricted folder + encrypted ├── "Confidential" → Team-only folder └── "Public" → Workspace-wide access Comparison with Databricks Unity Catalog and Snowflake Feature Fabric OneLake Security Databricks Unity Catalog Snowflake Granularity Folder, Table, Column Catalog, Schema, Table, Column Database, Schema, Table, Column Shortcuts awareness ✅ Native ❌ External locations only ❌ Row-level security T-SQL native Row filters Row access policies Column masking T-SQL GRANT Dynamic views Dynamic data masking CI/CD API REST API (GA) Terraform provider Terraform provider Purview/DLP integration Native Purview External (Microsoft Purview) External Cross-platform Fabric only Databricks only Snowflake only Cost Included in Fabric Included in Databricks Included in Snowflake Common Anti-PatternsAnti-Pattern 1: Over-granting at workspace level# BAD: Giving everyone Contributor at workspace level # Then trying to restrict individual folders az fabric workspace access add --role Contributor --principal everyone # GOOD: Give minimal workspace access, restrict at folder level az fabric workspace access add --role Viewer --principal everyone # Then grant Contributor on specific folders Anti-Pattern 2: Ignoring shortcutsShortcuts to external data inherit the parent Lakehouse's governance, but the external data itself may not be governed. Always apply explicit ACLs to shortcuts pointing to sensitive external data.Anti-Pattern 3: Manual governance at scaleDo not manage permissions through the Fabric UI for more than 10 folders. Use the REST API and automate via Terraform or CI/CD pipelines.Key TakeawaysOneLake security is the critical missing piece that makes Fabric enterprise-viable. Folder-level and table-level RBAC with shortcuts-aware governance means you can finally enforce least-privilege access.Inheritance with override is the design pattern. Start with minimal workspace access, then grant granular permissions at folder and table level.Automate governance via REST API and Terraform. Manual permission management does not scale. Build governance into your deployment pipelines.Purview integration enables automated classification and compliance. Data sensitivity labels drive RBAC decisions — classify first, then govern.Compare with Databricks Unity Catalog before committing. Both platforms offer similar governance features; the choice depends on your broader ecosystem (Microsoft vs Databricks/Spark).ResourcesOneLake Security DocumentationFabric OneLake Security REST APIMicrosoft Purview Data Classification --- ## Azure AI Foundry Agent Service: Production AI with Memory + A2A URL: https://wenfeng.my/azure-ai-foundry-agent-service-production-ai-agents-managed-memory-a2a/ Author: admin Published: 2026-08-04 00:13:55 Modified: 2026-10-04 15:29:57 Azure AI Foundry Agent Service: Building Production-Grade AI Agents with Managed Memory and A2A ProtocolMicrosoft shipped more agent infrastructure in the December 2025 to January 2026 window than the previous six months combined. If you blinked, you missed three rebrandings and a fundamental shift in how Azure positions its agent platform.The short version: Azure AI Foundry became Microsoft Foundry (cross-cloud, not Azure-only), the SDK consolidated into a single azure-ai-projects v2 package, managed long-term memory entered public preview (free during preview), and A2A protocol tooling launched for cross-framework orchestration. Production observability — monitoring, evaluations, cost tracking — is now GA.If you are building AI agents on Azure, this changes your architecture options significantly. Here is what actually matters and what is still marketing.What Changed1. Managed Long-Term Memory (Public Preview)The headline feature. Foundry Agent Service now provides persistent, managed memory for agents — no external vector database required.from azure.ai.projects import AIProjectClient from azure.identity import DefaultAzureCredential client = AIProjectClient( endpoint="https://your-project.cognitiveservices.azure.com/", credential=DefaultAzureCredential() ) # Create an agent with managed memory agent = client.agents.create_agent( name="invoice-processor", model="gpt-4o", instructions="Process invoices and remember vendor payment patterns.", # Managed memory — agent remembers across sessions enable_long_term_memory=True ) # The agent can now recall information from past interactions response = client.agents.create_run( thread_id=thread.id, agent_id=agent.id, messages=[{"role": "user", "content": "What were the last 3 invoices from Contoso?"}] ) What this replaces: Previously, you needed to build your own memory layer — typically Azure AI Search (vector store) + custom embedding pipeline + session management. Managed memory eliminates this for standard use cases.When to use managed memory vs. external vector stores: Factor Managed Memory External Vector Store Setup complexity Zero config Full pipeline setup Data control Microsoft-managed You control the data Custom embedding models No Yes Query flexibility Basic recall Advanced filtering, hybrid search Compliance requirements May not meet data residency Full compliance control Cost at scale Free during preview, TBD after Predictable (AI Search pricing) My recommendation: Start with managed memory for prototyping and non-sensitive data. Move to Azure AI Search or Cosmos DB vector search when you need data residency guarantees, custom embeddings, or advanced query patterns.2. SDK Consolidation (azure-ai-projects v2)Microsoft consolidated four SDKs into one:Before (v1): azure-ai-ml → Model management azure-ai-resources → Project management azure-ai-openai → OpenAI endpoints azure-ai-inference → Inference APIs After (v2): azure-ai-projects → Everything Migration is straightforward but not automatic:pip install azure-ai-projects --upgrade # v1 pattern (deprecated) from azure.ai.ml import MLClient from azure.ai.resources import AIProjectClient as ResourceClient # v2 pattern (current) from azure.ai.projects import AIProjectClient The main breaking change: authentication patterns unified under DefaultAzureCredential. If you were using API keys directly, you need to switch.3. A2A Protocol SupportFoundry agents can now advertise AgentCards and participate in A2A-orchestrated workflows. This is the interoperability play — your Foundry agents can collaborate with LangGraph agents, AutoGen agents, or any A2A-compatible system.# Foundry agent with A2A enabled agent = client.agents.create_agent( name="data-analyst", model="gpt-4o", instructions="Analyze data and return structured insights.", a2a_enabled=True # Automatically exposes AgentCard ) # The agent is now discoverable at: # https://your-project.cognitiveservices.azure.com/.well-known/agent.json 4. Production Observability (GA)The monitoring dashboard now tracks: - Token usage per agent run - Cost attribution by agent and thread - Latency distributions (P50, P95, P99) - Evaluation metrics (groundedness, relevance, coherence) - Tool call traces end-to-endThis is the feature that makes Foundry viable for production. Without observability, you are flying blind. With it, you can justify agent deployment to finance and compliance teams.Managed vs Self-Hosted: The Real DecisionThe question I get most often: "Should I use Foundry's managed agent runtime, or build my own with LangGraph/AutoGen/custom?"Choose Foundry when: - Your team is Microsoft-centric (Azure, Entra ID, Microsoft 365) - You need compliance certifications out of the box (SOC 2, ISO 27001, HIPAA) - Managed memory is sufficient for your recall requirements - You want cost tracking and observability without building infrastructure - Your agents are relatively straightforward (tool calls, conversation, memory)Choose self-hosted when: - You need full control over the model (fine-tuned, self-hosted, non-Azure) - Your agent architecture is complex (multi-agent with custom routing) - You need custom memory implementations (graph RAG, hierarchical memory) - Data residency requires specific infrastructure not available in Foundry regions - You are already invested in a framework (LangGraph, AutoGen, custom)The hybrid approach — which I recommend for most enterprise deployments — uses Foundry for model hosting and observability while keeping custom orchestration:Custom Orchestrator (LangGraph / Hermes / custom) │ ├── Model calls → Azure AI Foundry (hosted models) ├── Observability → Foundry monitoring (cost, latency, evals) ├── Tool access → MCP servers (custom + Foundry-hosted) └── Agent communication → A2A protocol (Foundry agents participate) This gives you the best of both: Azure's model hosting and monitoring without being locked into Foundry's orchestration model.SDK Migration ChecklistIf you are on the v1 SDK, here is the migration path:# 1. Install v2 pip install azure-ai-projects --upgrade # 2. Update imports # Find all: from azure.ai.ml import / from azure.ai.resources import # Replace with: from azure.ai.projects import AIProjectClient # 3. Update authentication # Old: MLClient(subscription_id, resource_group, workspace) # New: AIProjectClient(endpoint, credential=DefaultAzureCredential()) # 4. Update agent creation calls # Check azure-ai-projects SDK reference for parameter changes # 5. Test locally before deploying az login python -m pytest tests/test_agents.py -v Cost ConsiderationsFoundry pricing during the preview period is aggressive:Managed memory: Free during previewAgent runs: Billed per token (model-dependent)Observability: Free during previewA2A protocol: Free (open standard)Post-preview, expect managed memory to be priced per GB-month (similar to Azure AI Search). For a typical agent handling 1,000 conversations/day with 50 tokens of memory per conversation, that is roughly 1.5M tokens/month of memory — likely $5-15/month based on current storage pricing.The real cost driver is model inference, not memory or orchestration. A GPT-4o agent handling complex multi-turn conversations costs roughly $0.15-0.50 per 1K conversations depending on message length.What I Would Do DifferentlyAfter building agents on both Foundry and custom infrastructure:Start with Foundry for the first agent. The managed runtime eliminates infrastructure overhead and lets you validate the business case. Migrate to custom orchestration only when you hit a specific limitation.Use managed memory for prototyping, external vector stores for production. The managed memory preview is excellent for getting started, but data residency and compliance requirements will eventually force you to externalize.Invest in MCP servers early. Whether you use Foundry or custom orchestration, standardizing tool access via MCP pays dividends when you add new agents or tools.Do not skip observability. Foundry's monitoring dashboard is the easiest path to production-grade observability. Even if you run custom orchestration, forward traces to Application Insights and use Foundry's evaluation tools.Key TakeawaysMicrosoft Foundry (formerly Azure AI Foundry) is now the unified agent platform. The SDK consolidation, managed memory, and A2A support make it the most complete managed agent runtime on Azure.Managed memory eliminates the vector store setup tax. Use it for prototyping and non-sensitive workloads. Externalize for compliance and advanced query needs.A2A support means Foundry agents are interoperable. Your Foundry agents can collaborate with LangGraph, AutoGen, or any A2A-compatible agent without custom bridges.The hybrid approach is usually right. Use Foundry for model hosting and observability; keep custom orchestration for complex workflows. Do not force your architecture into Foundry's orchestration model unless it fits.Production observability (GA) is the real game-changer. Cost tracking, latency monitoring, and evaluation metrics make agent deployment justifiable to stakeholders who need numbers, not demos.ResourcesMicrosoft Foundry Agent Serviceazure-ai-projects SDK v2A2A ProtocolManaged Long-Term Memory Preview --- ## Azure Immutable Backups: Building Ransomware-Proof Recovery with WORM Vaults and Blob Immutability URL: https://wenfeng.my/azure-immutable-backups-ransomware-proof-recovery-worm-vaults/ Author: admin Published: 2026-08-03 23:30:04 Modified: 2026-10-04 15:29:57 Azure Immutable Backups: Building Ransomware-Proof Recovery with WORM Vaults and Blob ImmutabilityThe attack pattern is well-documented now. Ransomware operators do not just encrypt production data — they target backup repositories first. By the time you realize you are under attack, your backups are already compromised. The 2025-2026 wave of attacks specifically targets Azure Backup vaults, VMware backup proxies, and Veeam repositories before deploying the encryption payload.Azure Backup Immutable Vaults use WORM (Write Once, Read Many) storage to make recovery points irrevocable. Even the Azure subscription owner cannot delete them until the retention period expires. Combined with soft-delete, MFA-guarded delete protection, and geo-redundant immutable copies, this is now the minimum baseline for enterprise disaster recovery.SOC 2 Type II auditors in Malaysia are increasingly requiring immutable backup evidence. If you are serving global clients, this is not optional anymore.How Immutable Vaults WorkAn Immutable Vault enforces WORM policy at the vault level. Once enabled:No recovery point can be deleted or modified until its retention period expiresNo backup admin, subscription owner, or Azure Support can override the retentionSoft-delete is mandatory with a minimum 14-day retention periodMFA is required for any delete operation (even after retention expires)This is not "soft immutability" — it is enforced by Azure's storage layer. The WORM policy is cryptographically verified and cannot be bypassed through the portal, CLI, API, or PowerShell.┌─────────────────────────────────────────────┐ │ Azure Backup Vault │ │ ┌─────────────────────────────────────────┐│ │ │ WORM Policy (Immutable) ││ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐││ │ │ │ RP Day 1 │ │ RP Day 2 │ │ RP Day 3 │││ │ │ │ (locked) │ │ (locked) │ │ (locked) │││ │ │ └──────────┘ └──────────┘ └──────────┘││ │ │ Cannot delete until retention expires ││ │ └─────────────────────────────────────────┘│ │ ┌─────────────────────────────────────────┐│ │ │ Soft-Delete: 14-day minimum ││ │ │ MFA Guard: Required for any delete ││ │ └─────────────────────────────────────────┘│ └─────────────────────────────────────────────┘ Time-Based vs Legal-Hold PoliciesAzure offers two immutability policy types:Time-Based Retention (TBR): Recovery points are locked for a fixed number of days. After the retention period expires, they become eligible for deletion (subject to soft-delete and MFA).# Create an immutable vault with 30-day time-based retention az backup vault create \ --name vault-immutable-prod \ --resource-group rg-backup \ --location malaysiaeast \ --immutability "Enabled" # Set backup policy with 30-day retention az backup policy set \ --vault-name vault-immutable-prod \ --resource-group rg-backup \ --name policy-immutable-30d \ --policy '{ "backupManagementType": "AzureIaasVM", "schedulePolicy": { "schedulePolicyType": "SimpleSchedulePolicy", "scheduleRunFrequency": "Daily", "scheduleRunTimes": ["2026-06-04T02:00:00+08:00"] }, "retentionPolicy": { "retentionPolicyType": "LongTermRetentionPolicy", "dailySchedule": { "retentionTimes": ["2026-06-04T02:00:00+08:00"], "retentionDuration": {"count": 30, "durationType": "Days"} } } }' Legal-Hold: Recovery points are locked indefinitely until an authorized user removes the hold. Used for compliance scenarios where retention duration is不确定 (regulatory investigations, litigation holds).# Apply a legal hold tag az backup item set-policy \ --vault-name vault-immutable-prod \ --resource-group rg-backup \ --container-name vm-backup \ --item-name vm-webserver-01 \ --policy-name policy-immutable-30d \ --retention-override 365 \ --legal-hold "investigation-2026" My recommendation: Use Time-Based Retention for standard backup policies (30-90 days daily, 12 months monthly, 3-5 years yearly). Use Legal-Hold only for specific compliance scenarios — it complicates lifecycle management.Step-by-Step ImplementationStep 1: Create the Immutable Vault# Create a Recovery Services vault with immutability enabled az backup vault create \ --name vault-immutable-prod \ --resource-group rg-backup \ --location malaysiaeast # Enable immutability (one-way operation — cannot be disabled) az backup vault update \ --name vault-immutable-prod \ --resource-group rg-backup \ --immutability "Enabled" ⚠️ Warning: Enabling immutability is a one-way operation. You cannot disable it. Plan your retention policy before enabling.Step 2: Configure Backup Policy# Create a comprehensive backup policy cat > policy.json << 'EOF' { "backupManagementType": "AzureIaasVM", "schedulePolicy": { "schedulePolicyType": "SimpleSchedulePolicy", "scheduleRunFrequency": "Daily", "scheduleRunTimes": ["2026-06-04T02:00:00+08:00"] }, "retentionPolicy": { "retentionPolicyType": "LongTermRetentionPolicy", "dailySchedule": { "retentionTimes": ["2026-06-04T02:00:00+08:00"], "retentionDuration": {"count": 30, "durationType": "Days"} }, "weeklySchedule": { "daysOfTheWeek": ["Sunday"], "retentionTimes": ["2026-06-04T02:00:00+08:00"], "retentionDuration": {"count": 12, "durationType": "Weeks"} }, "monthlySchedule": { "retentionScheduleFormatType": "Weekly", "retentionScheduleWeekly": { "daysOfTheWeek": ["Sunday"], "weeksOfTheMonth": ["First"] }, "retentionTimes": ["2026-06-04T02:00:00+08:00"], "retentionDuration": {"count": 12, "durationType": "Months"} }, "yearlySchedule": { "retentionScheduleFormatType": "Weekly", "retentionScheduleWeekly": { "daysOfTheWeek": ["Sunday"], "weeksOfTheMonth": ["First"] }, "monthsOfYear": ["January"], "retentionTimes": ["2026-06-04T02:00:00+08:00"], "retentionDuration": {"count": 5, "durationType": "Years"} } } } EOF az backup policy create \ --vault-name vault-immutable-prod \ --resource-group rg-backup \ --name policy-immutable-30d \ --policy @policy.json Step 3: Enable Soft-Delete with MFA Guard# Configure soft-delete (minimum 14 days, recommended 14-30) az backup vault update \ --name vault-immutable-prod \ --resource-group rg-backup \ --soft-delete "Enabled" # Note: Soft-delete is automatically enabled when immutability is enabled # with a minimum of 14 days. You cannot set it lower. Step 4: Enable Cross-Region Restore# For geo-redundant recovery points az backup vault update \ --name vault-immutable-prod \ --resource-group rg-backup \ --cross-region-restore "Enabled" Step 5: Test Recovery Under Simulated RansomwareThis is the critical step most organizations skip. Test your recovery process:# 1. Trigger an on-demand backup az backup backup-now \ --vault-name vault-immutable-prod \ --resource-group rg-backup \ --container-name vm-backup \ --item-name vm-webserver-01 \ --retain-until "2026-07-04" # 2. Verify the recovery point exists and is immutable az backup recovery-point list \ --vault-name vault-immutable-prod \ --resource-group rg-backup \ --container-name vm-backup \ --item-name vm-webserver-01 \ --output table # 3. Attempt to DELETE the recovery point (should fail) az backup recovery-point delete \ --vault-name vault-immutable-prod \ --resource-group rg-backup \ --container-name vm-backup \ --item-name vm-webserver-01 \ --name "<recovery-point-id>" \ --yes # Expected: Error — cannot delete immutable recovery point # 4. Restore to a test VNet az backup restore restore-disks \ --vault-name vault-immutable-prod \ --resource-group rg-backup \ --container-name vm-backup \ --item-name vm-webserver-01 \ --recovery-point-id "<recovery-point-id>" \ --storage-account-type Standard_LRS \ --target-resource-group rg-test-restore \ --target-vnet vnet-test \ --target-subnet subnet-test Cost ImplicationsWORM storage costs are not significantly higher than standard backup storage, but retention duration is the cost driver: Retention Period Storage Cost (per 100GB/month) Notes 30 days daily ~$2-4 Standard LRS 90 days daily ~$6-12 Standard LRS 12 months monthly ~$2-4 Only monthly snapshots 5 years yearly ~$1-2 Only yearly snapshots For a typical 1TB VM with daily 30-day retention, expect approximately $20-40/month for immutable backup storage in Malaysia East.The real cost savings come from not being ransomed. The average ransomware downtime for enterprises is 24 days. If your daily revenue is $50K, that is $1.2M in downtime costs — far exceeding any backup storage premium.Audit Evidence for SOC 2SOC 2 Type II auditors want to see:Immutable vault configuration — Screenshot or CLI output showing immutability is enabledRecovery point deletion attempts — Logs showing deletion was blocked by WORM policyRecovery test results — Documented evidence that recovery was successfulRetention policy documentation — Written policy matching the technical configuration# Generate audit evidence report echo "=== Immutable Vault Status ===" > audit-report.txt az backup vault show \ --name vault-immutable-prod \ --resource-group rg-backup \ --query "{Immutability:properties.immutabilitySettings.state, SoftDelete:properties.softDeleteSettings.state}" \ --output table >> audit-report.txt echo "" >> audit-report.txt echo "=== Recent Recovery Points ===" >> audit-report.txt az backup recovery-point list \ --vault-name vault-immutable-prod \ --resource-group rg-backup \ --container-name vm-backup \ --item-name vm-webserver-01 \ --query "[].{ID:name, Created:properties.instantRpDetails.sourceResourceType, RetainedUntil:properties.policyRetentionPolicy}" \ --output table >> audit-report.txt echo "" >> audit-report.txt echo "=== Deletion Attempts (last 30 days) ===" >> audit-report.txt az monitor activity-log list \ --start-time "$(date -d '30 days ago' +%Y-%m-%dT%H:%M:%S)" \ --query "[?contains(operationName.value, 'delete') && contains(resourceProviderValue, 'RecoveryServices')].{Time:eventTimestamp, Operation:operationName.value, Result:status.value}" \ --output table >> audit-report.txt Key TakeawaysImmutable Vaults are a one-way operation. Once enabled, you cannot disable immutability. Plan your retention policy before enabling.Time-Based Retention for standard backups, Legal-Hold for compliance. Do not over-complicate lifecycle management with legal holds unless you have a specific regulatory requirement.Test recovery under simulated ransomware conditions. Verify that deletion attempts fail, and that restore operations succeed to an isolated network segment.SOC 2 Type II auditors increasingly require immutable backup evidence. Configure audit reporting before the audit, not during.The cost of immutability is trivial compared to ransomware downtime. At $20-40/month for a 1TB VM, the insurance value is overwhelming.ResourcesAzure Backup Immutable VaultsPlan Azure Backup Against RansomwareAzure Backup Pricing --- ## Azure Private Endpoints Are Breaking DNS: Surviving the March 2026 Default Outbound Access Retirement URL: https://wenfeng.my/azure-private-endpoints-breaking-dns-march-2026-outbound-shutdown/ Author: admin Published: 2026-08-02 23:30:23 Modified: 2026-10-04 15:29:57 Azure Private Endpoints Are Breaking DNS: Surviving the March 2026 Default Outbound Access RetirementOn March 31, 2026, Azure completed the retirement of default outbound internet access for new virtual networks. If you deployed a VM after that date without explicit outbound connectivity — NAT Gateway, Public IP, or Firewall — you had no internet access. Period.That was the announcement. The reality, three months later, is messier. Organizations that rushed to deploy Private Endpoints to maintain connectivity to PaaS services are now dealing with a wave of DNS resolution failures that are subtle, hard to diagnose, and devastating when they hit production.I have seen this pattern repeat across multiple client environments in Malaysia and Singapore. The root cause is always the same: recursive DNS forwarding loops between on-premises resolvers and Azure Private DNS Zones. Here is what is happening and how to fix it.The Three Failure ModesFailure Mode 1: Recursive DNS Forwarding LoopsThe most common failure. When you enable Azure Private DNS Zones, Azure's internal DNS resolver (168.63.129.16) handles resolution for privatelink.database.windows.net and similar zones. But if your VM's DNS configuration points to an on-premises resolver that forwards queries to Azure DNS, and that resolver also tries to resolve private endpoints, you get a loop.The query path looks like this:VM → On-prem DNS (forwarder) → Azure DNS (168.63.129.16) → On-prem DNS (forwarder) → Azure DNS → ... TTL expires. Resolution fails. Your application cannot connect to the SQL Server it has been using for years.Symptoms: - Intermittent DNS timeouts - nslookup works from some VMs but not others - Connections to PaaS services fail from new subnets but work from existing onesThe fix: Stop forwarding all queries to Azure DNS. Instead, use conditional DNS forwarding with a suffix-based policy:# On your on-premises DNS server (Windows DNS or BIND) # Forward ONLY privatelink.* zones to Azure DNS # Windows DNS example Add-DnsServerForwarder -IPAddress "168.63.129.16" -PassThru # Then create a zone delegation or conditional forwarder: Add-DnsServerConditionalForwarderZone -Name "privatelink.database.windows.net" -ReplicationScope "Forest" -MasterServers "168.63.129.16" Add-DnsServerConditionalForwarderZone -Name "privatelink.blob.core.windows.net" -ReplicationScope "Forest" -MasterServers "168.63.129.16" # Terraform example for Azure DNS Private Resolver resource "azurerm_private_dns_resolver" "main" { name = "dns-resolver-main" resource_group_name = azurerm_resource_group.main.name location = azurerm_resource_group.main.location virtual_network_id = azurerm_virtual_network.main.id } resource "azurerm_private_dns_resolver_inbound_endpoint" "main" { name = "inbound-endpoint" private_dns_resolver_id = azurerm_private_dns_resolver.main.id location = azurerm_resource_group.main.location ip_configurations { private_ip_allocation_method = "Dynamic" subnet_id = azurerm_subnet.dns_resolver.id } } Failure Mode 2: The WireServer Black HoleThe IP address 168.63.129.16 is Azure's WireServer — the internal metadata and DNS service. It is routable within a VNet but not across VPN tunnels or ExpressRoute circuits. When your on-premises DNS tries to forward queries to this IP, the packets are silently dropped.This is particularly insidious because it does not cause an error — the query just times out after 5 seconds, and the retry behavior varies by DNS client implementation.Why this matters for hybrid environments: If you have configured your on-premises DNS to forward Azure zone queries to 168.63.129.16, those queries will fail for any resolution originating outside Azure. This includes:CI/CD pipelines running on-premises that need to connect to Azure SQLDisaster recovery failover scenarios where on-premises systems need Azure PaaS accessDeveloper machines accessing staging databasesThe fix: Deploy Azure DNS Private Resolver with both inbound and outbound endpoints. The inbound endpoint gives your on-premises DNS a routable IP (within the VNet) to forward queries to. The outbound endpoint lets Azure DNS forward non-Azure queries to your on-premises DNS.# Create the resolver and endpoints az network private-dns resolver create \ --name dns-resolver-main \ --resource-group rg-networking \ --location malaysiaeast \ --virtual-network id="/subscriptions/.../virtualNetworks/vnet-main" # Inbound endpoint (routable from on-prem via VPN/ER) az network private-dns resolver inbound-endpoint create \ --name inbound-ep \ --dns-resolver-name dns-resolver-main \ --resource-group rg-networking \ --ip-configurations '[{"private-ip-allocation-method":"Dynamic","subnet":"/subscriptions/.../subnets/dns-resolver"}]' # Outbound endpoint (for conditional forwarding to on-prem) az network private-dns resolver outbound-endpoint create \ --name outbound-ep \ --dns-resolver-name dns-resolver-main \ --resource-group rg-networking \ --subnet "/subscriptions/.../subnets/dns-resolver-outbound" # Forwarding rule: forward *.internal.contoso.com to on-prem DNS az network private-dns resolver forwarding-rule create \ --name forward-to-onprem \ --dns-forwarding-ruleset name=ruleset-main \ --resource-group rg-networking \ --domain-name "internal.contoso.com." \ --forwarding-rules '[{"ip-address":"10.0.1.10","port":53}]' \ --priority 100 Failure Mode 3: Subnet ExhaustionThis one is not a DNS failure per se, but it kills Private Endpoint deployments just as effectively. Azure reserves 5 IP addresses per subnet (first four plus the broadcast). A /27 subnet gives you 27 usable IPs, minus 5 reserved, leaving 22 addresses. If you also deploy an Azure DNS Private Resolver inbound endpoint in that subnet, that consumes one more, leaving 21.Now deploy Private Endpoints. Each one needs its own IP. A typical enterprise environment with SQL Server, Storage Accounts, Key Vault, Service Bus, Cosmos DB, and a few App Services quickly exhausts a /27.The math:/27 subnet: 32 - 5 reserved = 27 usable - DNS resolver inbound: 1 IP - Available for Private Endpoints: 26 If you have more than 26 PaaS services, you need multiple subnets or a larger CIDR block.The fix: Plan your Private Endpoint subnet sizing before deployment. Here is a capacity planning script:#!/usr/bin/env python3 """Private Endpoint capacity planner""" import ipaddress def plan_endpoints(cidr: str, resolver_count: int = 1, azure_reserved: int = 5): """Calculate available Private Endpoint slots in a subnet.""" network = ipaddress.ip_network(cidr) total = network.num_addresses usable = total - azure_reserved available = usable - resolver_count print(f"Subnet: {cidr}") print(f"Total IPs: {total}") print(f"Azure reserved: {azure_reserved}") print(f"DNS resolver endpoints: {resolver_count}") print(f"Available for Private Endpoints: {available}") if available < 10: print("⚠️ WARNING: Consider a larger subnet (/26 or /25)") return available # Example: planning for a /27 with 1 DNS resolver plan_endpoints("10.0.10.0/27", resolver_count=1) # For a larger deployment: plan_endpoints("10.0.10.0/24", resolver_count=2) Recommended subnet sizing by environment: Environment Expected PEs Subnet Size Available PEs Development 5-10 /27 21 Staging 10-20 /26 53 Production 20-50 /25 117 Enterprise hub 50-100 /24 247 The Correct ArchitectureAfter diagnosing dozens of these failures, here is the DNS architecture that actually works for hybrid Azure environments with Private Endpoints:On-premises VMs/Servers │ ▼ On-premises DNS (Conditional Forwarder) │ ├── *.internal.contoso.com → on-prem AD DNS (10.0.1.10) ├── privatelink.*.azure.com → Azure DNS Private Resolver inbound (10.0.10.4) └── everything else → public DNS (8.8.8.8) Azure DNS Private Resolver │ ├── Inbound endpoint: 10.0.10.4 (routable from on-prem via VPN/ER) └── Outbound endpoint → Azure DNS (168.63.129.16) Azure VMs │ ├── DNS servers: 10.0.10.4 (resolver inbound) + 168.63.129.16 (fallback) └── Private DNS Zones: linked to VNet Key design principles:Never forward all queries to 168.63.129.16 from on-prem. It is not routable over VPN/ExpressRoute.Use conditional forwarding, not global forwarding. Only route privatelink.* zones to Azure.Deploy DNS Private Resolver for hybrid resolution. It provides a routable inbound IP and handles the split-brain DNS scenario cleanly.Size your Private Endpoint subnets generously. A /27 is fine for dev; production needs /25 or /24.Use Terraform or Bicep to manage Private Endpoints as code. Manual deployment at scale is error-prone and unrepeatable.Terraform Auto-Generation for Bulk Private EndpointsWhen migrating a large environment, you often need to create Private Endpoints for dozens of PaaS services. Here is a pattern that scales:variable "private_endpoints" { type = map(object({ service_id = string subresource = string dns_zone = string })) default = { "sql-main" = { service_id = "/subscriptions/.../Microsoft.Sql/servers/sql-main" subresource = "sqlServer" dns_zone = "privatelink.database.windows.net" } "storage-main" = { service_id = "/subscriptions/.../Microsoft.Storage/storageAccounts/storagemain" subresource = "blob" dns_zone = "privatelink.blob.core.windows.net" } "keyvault-main" = { service_id = "/subscriptions/.../Microsoft.KeyVault/vaults/kv-main" subresource = "vault" dns_zone = "privatelink.vaultcore.azure.net" } } } resource "azurerm_private_endpoint" "this" { for_each = var.private_endpoints name = "pe-${each.key}" location = azurerm_resource_group.main.location resource_group_name = azurerm_resource_group.main.name subnet_id = azurerm_subnet.private_endpoints.id private_service_connection { name = "conn-${each.key}" private_connection_resource_id = each.value.service_id subresource_names = [each.value.subresource] is_manual_connection = false } private_dns_zone_group { name = "dns-group-${each.key}" private_dns_zone_ids = [azurerm_private_dns_zone.zones[each.value.dns_zone].id] } } This pattern lets you add new Private Endpoints by simply adding an entry to the private_endpoints map. Terraform handles the DNS zone linking, the service connection, and the subnet placement.Auditing Your Current StateBefore you start fixing, audit. Here is a script to discover all existing Private Endpoints and their subnet utilization across a subscription:#!/bin/bash # Audit Private Endpoints across a subscription az account set --subscription "$SUBSCRIPTION_ID" echo "=== Private Endpoints by Subnet ===" az network private-endpoint list \ --query "[].{name:name, subnet:ipConfigurations[0].subnet.id, group:privateLinkServiceConnections[0].privateLinkServiceId}" \ --output table echo "" echo "=== Subnet Utilization ===" az network vnet subnet list --vnet-name vnet-main --resource-group rg-main \ --query "[?contains(name, 'private')].{name:name, addresses:addressPrefix, used:ipConfigurations | length(@)}" \ --output table Save the output and cross-reference it against your expected PaaS service count. If you are close to the limit, resize the subnet before it becomes a production incident.What Surprised MeThe hardest part of this migration was not the Private Endpoints themselves — Azure makes that fairly straightforward. The hard part was the DNS. Specifically:The failure is delayed. You deploy Private Endpoints, everything looks fine, and then three days later a different application breaks because it happens to use a different DNS path.Debugging tools are limited. nslookup and dig show you the resolution chain, but they do not tell you where the loop is. You need packet captures or DNS query logging on your resolver to see the actual traffic pattern.Documentation assumes a greenfield. Microsoft's docs walk you through setting up Private Endpoints from scratch. They do not adequately address the scenario where you have 200 VMs with existing DNS configurations that all need updating.The 168.63.129.16 address confuses everyone. New team members consistently try to ping or telnet to it from on-prem. It is not a regular IP — it is a link-local service that only works within an Azure VNet.Key TakeawaysAzure's March 2026 default outbound retirement forces Private Endpoint adoption. If you have not migrated yet, you will, and DNS will be the hard part.Recursive DNS forwarding loops are the number one failure mode. Use conditional forwarding with suffix-based policies, not global forwarding to 168.63.129.16.Deploy Azure DNS Private Resolver for hybrid environments. It solves the routable-IP problem and the split-brain DNS scenario cleanly.Plan subnet sizing before deploying Private Endpoints. A /27 gives you ~21 slots; production environments with many PaaS services need /25 or /24.Automate Private Endpoint deployment with Terraform or Bicep. Manual management at scale is error-prone and does not survive team changes.ResourcesAzure Networking Updates — Security, Reliability, and HAAzure Private DNS Zones DocumentationAzure DNS Private ResolverDefault Outbound Access Retirement Announcement --- ## LLM Inference in Production 2026: Quantization, KV Cache, Speculative Decoding, and the vLLM vs SGLang Decision URL: https://wenfeng.my/llm-inference-production-2026-quantization-kv-cache-vllm-sglang/ Author: admin Published: 2026-08-01 23:30:10 Modified: 2026-08-01 23:30:10 LLM Inference in Production 2026: Quantization, KV Cache, Speculative Decoding, and the vLLM vs SGLang DecisionThe gap between a demo and a production LLM deployment is measured in dollars per token and milliseconds of latency. A naive inference setup on an A100 costs 5-10x more per query than an optimized one. A poorly configured KV cache turns a 2GB model into a 16GB memory hog. The wrong serving framework means your 99th-percentile latency is 4 seconds instead of 800ms.I have deployed LLM inference on Azure VMs for three different use cases in the last six months. Here is what actually matters.The Optimization StackProduction LLM inference has four optimization layers, each with real trade-offs:┌──────────────────────────────────────────────┐ │ Layer 4: Serving Framework │ │ vLLM / SGLang / TensorRT-LLM / llama.cpp │ ├──────────────────────────────────────────────┤ │ Layer 3: Speculative Decoding │ │ Draft model acceleration │ ├──────────────────────────────────────────────┤ │ Layer 2: KV Cache Optimization │ │ GQA / MLA / PagedAttention │ ├──────────────────────────────────────────────┤ │ Layer 1: Quantization │ │ GPTQ / AWQ / GGUF / FP8 │ └──────────────────────────────────────────────┘ Most teams focus on Layer 1 (quantization) and ignore the rest. The real gains are in Layers 2-4.Layer 1: Quantization — Which Method, WhenQuantization reduces model precision from FP16/BF16 to INT8, INT4, or mixed precision. The result: lower VRAM usage with minimal quality loss.The three production-grade quantization methods: Method VRAM Reduction Quality Loss Speed Best For GPTQ 50-75% <2% perplexity Medium GPU inference (A100, H100) AWQ 50-75% <1% perplexity Fast GPU inference with quality priority GGUF 50-80% Variable (Q4_K_M best) CPU-fast Edge, CPU, mixed CPU/GPU My decision rule:def choose_quantization(model_size_gb: float, gpu_vram_gb: int, quality_critical: bool) -> str: """Select quantization method based on deployment constraints.""" # If the model fits in FP16, do not quantize if model_size_gb * 2 <= gpu_vram_gb * 0.8: # 80% VRAM threshold return "fp16" if quality_critical: # Quality-sensitive workloads (RAG, code generation, reasoning) if model_size_gb * 0.5 <= gpu_vram_gb * 0.8: return "awq" # Best quality-to-compression ratio return "gptq" # Fallback for tight VRAM if gpu_vram_gb >= 80: # Datacenter GPU — GPTQ is fine return "gptq" if gpu_vram_gb <= 24: # Consumer/prosumer GPU — need maximum compression return "gguf-q4_k_m" # Best 4-bit quant for mixed CPU/GPU return "gptq" The FP8 option: NVIDIA H100/H200 GPUs support FP8 natively. If you are on H-series hardware, FP8 gives you 2x throughput over FP16 with negligible quality loss. It is not quantization in the traditional sense — it is native hardware precision.# Example: Deploy a 70B model on 2xH100 with AWQ vllm serve meta-llama/Llama-3.1-70B-Instruct-AWQ \ --quantization awq \ --tensor-parallel-size 2 \ --max-model-len 8192 \ --gpu-memory-utilization 0.9 \ --dtype auto Layer 2: KV Cache OptimizationThe KV (Key-Value) cache stores the attention state for all previous tokens. For a 70B model with 8192 context length, the KV cache alone can consume 16-24GB of VRAM. This is often the bottleneck, not the model weights.Three KV cache techniques in production:Grouped Query Attention (GQA)Most modern models (Llama 3, Qwen 3, DeepSeek) use GQA by default. Multiple query heads share the same key/value heads, reducing KV cache size by 4-8x without quality loss.# Check if your model uses GQA from transformers import AutoConfig config = AutoConfig.from_pretrained("meta-llama/Llama-3.1-70B-Instruct") print(f"Num KV heads: {config.num_key_value_heads}") # 8 (vs 64 query heads) print(f"GQA ratio: {config.num_attention_heads // config.num_key_value_heads}") # 8x reduction Multi-head Latent Attention (MLA)DeepSeek's MLA compresses KV cache into a low-rank latent representation. DeepSeek V4 and V3 use this, achieving 10x KV cache compression over standard MHA.The trade-off: MLA requires a custom attention kernel. Not all serving frameworks support it. vLLM added MLA support in v1.x; SGLang has experimental support.PagedAttentionvLLM's signature innovation. Instead of allocating contiguous VRAM for each sequence's KV cache (which wastes memory on fragmented sequences), PagedAttention uses block-based allocation — the same concept as virtual memory pages.Traditional KV Cache: [Seq 1: 8192 tokens, 16GB contiguous] [waste: 4GB gap] [Seq 2: 2048 tokens, 4GB] PagedAttention: [Block 1: Seq1] [Block 2: Seq1] [Block 3: Seq2] [Block 4: free] → No fragmentation, 30-50% better memory utilization Practical impact: PagedAttention typically allows 2-4x more concurrent requests per GPU by eliminating KV cache fragmentation. This is the single biggest throughput lever for production deployments.Layer 3: Speculative DecodingThe idea: use a small, fast "draft" model to generate candidate tokens, then verify them in parallel with the large model. If the draft model predicts correctly (which it does 70-90% of the time for common patterns), you get those tokens at draft-model speed.# vLLM with speculative decoding vllm serve meta-llama/Llama-3.1-70B-Instruct \ --speculative-model meta-llama/Llama-3.1-8B-Instruct \ --num-speculative-tokens 5 \ --tensor-parallel-size 4 When speculative decoding helps: - High-throughput scenarios (many concurrent requests) - Draft model is fast relative to target model (8B draft for 70B target) - Tasks with predictable token patterns (code completion, structured output)When it does not help: - Low-latency, single-request scenarios (the draft model adds overhead) - Creative generation (draft predictions are less accurate) - Small models where the draft model is not significantly fasterMy benchmark results on Azure ND96amsr_A100_v4 (8xA100-80GB): Config Throughput (tokens/s) Latency P99 70B FP16, no speculation 1,200 3.2s 70B FP16 + 8B draft (k=5) 1,850 2.8s 70B AWQ + 8B draft (k=5) 2,100 2.5s 70B AWQ, no speculation 1,600 3.0s The 50-75% throughput improvement from speculative decoding is real, but it requires careful tuning of the num_speculative_tokens parameter.Layer 4: The Serving Framework DecisionThis is the most consequential choice. vLLM, SGLang, TensorRT-LLM, and llama.cpp each have distinct strengths.vLLM (v1.x)The default choice for most production deployments. PagedAttention, continuous batching, tensor/pipeline parallelism, and the broadest model support.Strengths: - Widest model support (HuggingFace format out of the box) - PagedAttention for maximum memory efficiency - OpenAI-compatible API - Active community and rapid development - Multi-modal support (vision, audio) in v1.xWeaknesses: - Higher single-request latency than SGLang - MLA support is newer and less battle-tested - Complex configuration for optimal performanceSGLangOptimized for low-latency single requests and structured output. The SGLang runtime compiles attention kernels ahead-of-time for maximum performance.Strengths: - Fastest single-request latency in benchmarks - Excellent structured output (JSON, regex constraints) - RadixAttention for prefix caching (reuses KV cache across requests with shared prefixes) - Smaller memory footprintWeaknesses: - Narrower model support than vLLM - MLA support is experimental - Smaller communityTensorRT-LLMNVIDIA's inference engine. Maximum performance on NVIDIA hardware but requires model conversion.Strengths: - Highest throughput on NVIDIA GPUs - FP8 native support on H100/H200 - In-flight batching for maximum GPU utilizationWeaknesses: - Model conversion step adds complexity - NVIDIA-only (no AMD, Intel support) - Longer startup time (kernel compilation)llama.cppCPU and edge inference. GGUF format, runs on anything from a Raspberry Pi to a Mac Studio.Strengths: - Runs on CPU, Apple Silicon, and any GPU - GGUF format for easy distribution - Minimal resource requirements - Perfect for edge and developmentWeaknesses: - Lower throughput than GPU-optimized frameworks - Limited multi-GPU support - Not suitable for high-concurrency productionDecision Matrix Criteria vLLM SGLang TensorRT-LLM llama.cpp Throughput ★★★★ ★★★ ★★★★★ ★★ Single-request latency ★★★ ★★★★★ ★★★★ ★★★ Model support ★★★★★ ★★★ ★★★ ★★★★ Multi-GPU ★★★★ ★★★ ★★★★★ ★★ Structured output ★★★ ★★★★★ ★★★ ★★★ Ease of deployment ★★★★★ ★★★★ ★★ ★★★★★ NVIDIA optimization ★★★★ ★★★ ★★★★★ N/A My recommendation: Start with vLLM for general-purpose serving. Switch to SGLang if you need low-latency structured output. Use TensorRT-LLM only if you need maximum NVIDIA throughput and can handle the conversion complexity. Use llama.cpp for edge and development.Practical Azure DeploymentHere is a production-ready deployment on Azure for a 70B model:# Azure ND96amsr_A100_v4 (8xA100-80GB) # Deploy with vLLM + AWQ quantization + tensor parallelism pip install vllm vllm serve meta-llama/Llama-3.1-70B-Instruct-AWQ \ --host 0.0.0.0 \ --port 8000 \ --quantization awq \ --tensor-parallel-size 4 \ --max-model-len 16384 \ --gpu-memory-utilization 0.92 \ --enable-chunked-prefill \ --max-num-batched-tokens 32768 \ --max-num-seqs 64 \ --dtype auto \ --trust-remote-code # Health check curl http://localhost:8000/health # Benchmark python -m vllm.entrypoints.openai.api_client \ --model meta-llama/Llama-3.1-70B-Instruct-AWQ \ --base-url http://localhost:8000 Key parameters explained: - --tensor-parallel-size 4: Split model across 4 GPUs (70B AWQ fits in 4x80GB) - --gpu-memory-utilization 0.92: Leave 8% headroom for KV cache growth - --enable-chunked-prefill: Process long prompts in chunks to avoid latency spikes - --max-num-seqs 64: Maximum concurrent requests (tune based on latency requirements)What Surprised MeKV cache is the bottleneck, not model weights. For most workloads, the KV cache consumes more VRAM than the model itself at inference time. Optimizing KV cache (PagedAttention, GQA) has more impact than quantization.Speculative decoding is underused. Most teams do not know it exists. The 50-75% throughput improvement is real and requires minimal configuration.Framework choice matters more than hardware choice. The same model on the same GPU with different frameworks can show 2-3x throughput difference.AWQ beats GPTQ in almost every metric. Better quality, faster inference, easier deployment. GPTQ is legacy at this point.Structured output is a killer feature. SGLang's constrained decoding for JSON/schema output eliminates post-processing overhead and guarantees valid output format.Key TakeawaysQuantize with AWQ for GPU inference, GGUF for CPU/edge. AWQ gives the best quality-to-compression ratio; GGUF is the only option for non-NVIDIA hardware.KV cache optimization (PagedAttention, GQA) gives the biggest throughput gains. Invest here before exploring more exotic optimizations.Start with vLLM. It has the broadest support, the best documentation, and PagedAttention. Switch to SGLang or TensorRT-LLM only when you have a specific performance requirement.Speculative decoding is a free 50-75% throughput boost. Use it for high-throughput scenarios with a small draft model.The right serving framework on the same hardware can double your throughput. Benchmark before committing to a deployment architecture.ResourcesvLLM DocumentationSGLang DocumentationTensorRT-LLMllama.cpp --- ## The 2026 Multi-Agent Protocol Stack: MCP + A2A + ADK Define the Enterprise Interoperability Standard URL: https://wenfeng.my/multi-agent-protocol-stack-mcp-a2a-adk-enterprise-standard-2026/ Author: admin Published: 2026-08-01 02:31:35 Modified: 2026-08-01 02:31:35 The 2026 Multi-Agent Protocol Stack: MCP + A2A + ADK Define the Enterprise Interoperability StandardSix months ago, building a multi-agent system meant choosing a framework and accepting its constraints. LangGraph gave you graphs. AutoGen gave you conversations. CrewAI gave you roles. Each had its own memory model, its own tool interface, and its own way for agents to talk to each other. If you wanted agents from different frameworks to collaborate, you wrote custom bridges and hoped they held.That era ended in May 2026. Three pieces arrived — not from the same company, but converging on the same architecture:MCP (Model Context Protocol) — Anthropic's open standard for tool and data accessA2A (Agent-to-Agent) — Google's protocol for stateful inter-agent communication, now under Linux Foundation governanceADK (Agent Development Kit) — Google's orchestration framework, graduated to 1.0 GA with Python, Go, Java, and TypeScriptTogether, they form the first interoperable enterprise protocol stack for multi-agent systems. If you have been waiting for the "HTTP moment" for AI agents, this is it.The Three-Layer ArchitectureThink of it as three distinct protocol layers, each solving a different problem:┌─────────────────────────────────────────────┐ │ ADK / Orchestration Layer │ │ Agent definition, task routing, workflows │ │ (Google ADK, LangGraph, Semantic Kernel) │ ├─────────────────────────────────────────────┤ │ A2A Protocol Layer │ │ Agent discovery, stateful communication, │ │ task delegation between agents │ │ (Linux Foundation A2A, 150+ organizations) │ ├─────────────────────────────────────────────┤ │ MCP Protocol Layer │ │ Tool invocation, data access, resource │ │ discovery for individual agents │ │ (Anthropic MCP, open-source) │ └─────────────────────────────────────────────┘ MCP answers: "How does an agent call a tool or read data?" A2A answers: "How does one agent talk to another agent?" ADK answers: "How do I define, route, and manage a multi-agent workflow?"These are not competing standards. They are complementary layers that each handle a distinct concern. The confusion in 2025 was that frameworks tried to do all three in proprietary ways. The 2026 stack separates the layers.MCP: The Tool and Data LayerMCP is the simplest to understand. It provides a standard JSON-RPC interface for agents to invoke tools, access resources, and query prompts. Before MCP, every framework had its own function-calling schema — OpenAI's function format, Anthropic's tool_use format, custom JSON schemas.MCP standardizes this:{ "jsonrpc": "2.0", "method": "tools/call", "params": { "name": "query_azure_sql", "arguments": { "query": "SELECT TOP 10 * FROM sales.orders WHERE region = 'APAC'", "connection": "prod-sql-main" } }, "id": 1 } An MCP server wraps any data source or tool — a database, an API, a file system, a cloud service — and exposes it through the standard protocol. Any MCP-compatible agent can consume it without custom integration code.Production reality check: MCP adoption is genuine, but uneven. The protocol works well for synchronous tool calls. Where it struggles:Long-running tasks. MCP has no native concept of async job submission and polling. If your tool takes 30 seconds (common for data queries), the connection stays open.Streaming. MCP 2026 added streaming support, but implementations vary. Check your agent framework's MCP client before assuming streaming works.Authentication. MCP defines transport (stdio, HTTP+SSE) but not authentication. In production, you need to layer Entra ID, API keys, or mTLS on top.When to use MCP directly: Tool access for a single agent. Connecting agents to databases, APIs, and cloud services. Any scenario where you need "agent calls function."A2A: The Agent Communication LayerA2A solves the harder problem: how do two independent agents collaborate? Not in a master-worker pattern (that is orchestration), but as peers with different capabilities, running on different systems, possibly built on different frameworks.The core concept is the AgentCard — a machine-readable descriptor that advertises an agent's capabilities:{ "name": "Invoice Processor", "description": "Processes vendor invoices, validates amounts, and creates purchase orders", "url": "https://invoice-agent.internal.contoso.com", "capabilities": { "streaming": true, "pushNotifications": true }, "skills": [ { "id": "invoice-validation", "name": "Validate Invoice", "description": "Validates invoice amounts against PO and contract terms" } ], "authentication": { "schemes": ["Bearer"] } } Agent discovery works via an AgentCard endpoint (/.well-known/agent.json) — the same pattern as OAuth well-known URLs. An orchestrating agent can discover what other agents are available and what they can do at runtime, without hard-coded integrations.The event compaction feature is the most practical improvement. A2A supports structured event streams with automatic compaction — reducing token usage by up to 38% for long-running agent conversations. Instead of feeding the full conversation history to the LLM, A2A compacts events into summaries while preserving task state.# A2A task creation and event streaming import httpx # Create a task response = httpx.post( "https://invoice-agent.internal.contoso.com/a2a/tasks", json={ "id": "task-20260604-001", "message": { "role": "user", "parts": [ {"type": "text", "text": "Validate invoice INV-2026-0847 against PO-2026-0312"} ] }, "metadata": {"priority": "high"} } ) # Stream events (with automatic compaction) async for event in a2a_stream(response.json()["task_id"]): if event["type"] == "statusUpdate": print(f"Status: {event['status']}") elif event["type"] == "artifact": # Compacted artifact — full context in minimal tokens process_result(event["artifact"]) Production reality check: A2A is new (GA May 2026). The Linux Foundation governance is strong, and 150+ organizations signed on, but production implementations are still early. Key considerations:Framework support is limited. Google ADK has native A2A support. LangGraph and AutoGen require adapter libraries. Custom frameworks need to implement the protocol.Authentication between agents is still maturing. The spec supports Bearer tokens and OAuth, but service-to-service mTLS patterns are not yet standardized.No built-in persistence. A2A defines communication, not state management. You still need an external store for task history and agent memory.ADK: The Orchestration LayerADK 1.0 (GA May 2026) is Google's answer to "how do I define and run multi-agent workflows." It supports four languages (Python, Go, Java, TypeScript) with feature parity, which is a significant claim in a landscape where most frameworks are Python-only.The core abstraction is the Agent definition:from google.adk import Agent, Sequential, Loop # Define specialized agents invoice_validator = Agent( name="invoice-validator", model="gemini-2.5-pro", instruction="Validate invoices against purchase orders and contracts.", tools=[validate_invoice, lookup_po, check_contract_terms] ) amount_checker = Agent( name="amount-checker", model="gemini-2.5-flash", instruction="Verify invoice amounts are within tolerance of PO amounts.", tools=[calculate_variance, check_approval_thresholds] ) # Orchestrate as a sequential pipeline workflow = Sequential( agents=[invoice_validator, amount_checker], description="Validate and approve vendor invoices" ) # Or as a conditional loop review_loop = Loop( agent=invoice_validator, condition=lambda result: result.needs_human_review, max_iterations=3 ) ADK integrates natively with A2A. An ADK agent automatically exposes an AgentCard endpoint and can discover and call other A2A-compatible agents. This is where the stack becomes powerful: ADK handles the workflow, A2A handles the cross-agent communication, and MCP handles the tool access.The Hermes/OpenClaw PerspectiveI run a production multi-agent system — a content pipeline with Research Scout, Content Strategist, Technical Writer, SEO Optimizer, and Social Writer agents, coordinated through a kanban board. It predates the MCP+A2A+ADK stack, so it is worth comparing what the new stack offers versus what a custom orchestration provides.What we built (pre-stack): - Custom agent definitions with role-based prompts - SQLite-backed kanban board for task routing and state - Direct tool invocation (terminal, web search, Ghost API) per agent - Manual memory management across sessionsWhat the 2026 stack adds: - Standardized tool access via MCP — our agents could consume any MCP server without custom integration - Agent discovery via A2A AgentCards — new agents can join the pipeline without modifying the orchestrator - Event compaction — long-running research tasks would use fewer tokens - Multi-language support — non-Python agents (e.g., a Go-based monitoring agent) could participate nativelyWhat the stack does NOT solve: - Business logic routing — "assign topic X to the writer because it matches pillar Y" is application logic, not protocol logic - Content-specific tooling — Ghost API, Buffer API, MySQL operations are domain-specific and need custom MCP servers anyway - Human-in-the-loop workflows — both ADK and Hermes support this, but the integration patterns are differentThe honest assessment: the protocol stack is the right architecture for enterprise multi-agent systems where agents come from different vendors and run on different platforms. For single-vendor pipelines where you control all agents, the overhead of implementing MCP+A2A may not justify the abstraction benefit.Decision Framework: When to Use What Scenario Recommendation Single agent, multiple tools MCP only — standardize tool access Multiple agents, same framework Orchestration framework (ADK, LangGraph) without A2A Multiple agents, different frameworks A2A for communication + orchestration for workflow Enterprise agent marketplace Full stack: MCP + A2A + ADK Custom pipeline with domain-specific logic Custom orchestration + MCP for tool standardization Cross-organization agent collaboration A2A (AgentCard discovery + authentication) The Enterprise Readiness ChecklistBefore deploying multi-agent systems in production, verify these capabilities in your chosen stack:Observability. Can you trace a request across multiple agents? A2A defines structured events, but your monitoring system needs to consume them.Human-in-the-loop. Does the framework support pausing execution for human approval? Both ADK and A2A support this natively; LangGraph supports it via checkpointers.Memory. Does the agent retain context across interactions? MCP and A2A do not define memory — you need an external store (vector DB, relational DB, or application memory).Multi-agent coordination. Can agents delegate subtasks to other agents and aggregate results? A2A's task model supports this; the quality depends on the orchestration layer.Error handling. When an agent fails mid-task, does the system retry, escalate, or fall back? This is where most frameworks are still weak.Key TakeawaysThe three-layer stack is real. MCP for tools, A2A for agent communication, ADK for orchestration. These are complementary, not competing.MCP is production-ready today. If you are standardizing tool access across agents, start here. The protocol is simple, the ecosystem is growing, and the benefit is immediate.A2A is the future of agent interoperability. AgentCard discovery, event compaction, and cross-framework communication are powerful, but the ecosystem is early. Adopt now if you need cross-vendor agent collaboration; wait if your agents are all from the same framework.ADK 1.0 is the most complete orchestration framework. Multi-language support, native A2A integration, and Google's backing make it the strongest candidate for new enterprise projects. But LangGraph and Semantic Kernel remain viable if you are already invested.Custom orchestration still wins for domain-specific pipelines. The protocol stack is designed for general-purpose agent interoperability. If your pipeline has unique routing logic and custom tools, building on top of the protocols — not replacing your orchestration — is the pragmatic path.ResourcesA2A Protocol — GitHubGoogle ADK DocumentationMCP SpecificationJetBrains Agentic Frameworks Comparison 2026 --- ## Azure Databricks + OneLake Unity Catalog Integration — The End of the Data Platform Wars URL: https://wenfeng.my/azure-databricks-onelake-unity-catalog-integration-the-end-of-the-data-platform-wars/ Author: admin Published: 2026-07-29 23:44:22 Modified: 2026-10-04 15:29:57 Azure Databricks + OneLake Unity Catalog Integration — The End of the Data Platform WarsFor the past three years, the enterprise data platform conversation has been dominated by a single question: Fabric or Databricks? Every architecture review, every RFP, every conference hallway debate eventually lands on the same fork in the road. Organizations have been forced to pick a side, maintain duplicate data copies, and build expensive integration layers to bridge the two ecosystems.That era ended on June 16, 2026.Microsoft and Databricks announced that Azure Databricks can now store Unity Catalog managed tables directly in OneLake — the shared data lake at the heart of Microsoft Fabric. This is not a connector. Not a file export. Not a third-party ETL pipeline. It's native, bidirectional interoperability at the catalog level.Your Unity Catalog tables are now natively accessible from Fabric workloads, and your Fabric lakehouse tables are accessible from Databricks notebooks. One copy of the data, governed by Unity Catalog, usable from both platforms.What Actually ChangedBefore this integration, the typical enterprise pattern looked like this:Databricks Workspace Fabric Workspace ↓ ↓ Unity Catalog OneLake ↓ ↓ Delta Lake tables Lakehouse tables ↓ ↓ Separate data copies, separate governance, duplicate storage costs Organizations running both platforms maintained two copies of the same datasets. Data engineers synchronized them nightly (or hourly, or real-time). Governance policies were duplicated. Storage costs doubled. And every time a schema changed, two pipelines broke.After the integration:OneLake (Unified Storage) ↓ Unity Catalog (Governance) ↓ ┌─────────────────┐ ┌─────────────────┐ │ Databricks │ │ Fabric │ │ Workspaces │ │ Workloads │ └─────────────────┘ └─────────────────┘ One copy. One governance layer. Two compute engines.Why This Matters for Enterprise ArchitectureEliminating Data DuplicationThe most immediate impact is cost and complexity reduction. Enterprises typically spend 30-50% of their data platform budget on data movement — ETL pipelines, synchronization jobs, change data capture, and reconciliation processes between platforms. The OneLake-Unity Catalog integration eliminates the synchronization layer entirely.For a mid-size enterprise processing 10TB of shared datasets, this translates to: - ~$500/month in reduced storage costs (one copy instead of two) - ~20-40 hours/month in reduced data engineering maintenance - Elimination of data staleness issues (both platforms see the same data at the same time)Unified GovernanceUnity Catalog becomes the single governance layer for all data, regardless of which compute engine is accessing it. This means: - Row-level security policies defined once in Unity Catalog apply to both Databricks SQL queries and Fabric SQL Analytics Endpoint queries - Column masking works identically whether a data scientist runs a Databricks notebook or a business analyst runs a Fabric KQL query - Lineage tracking captures the full data flow across both platforms - Access auditing shows who accessed what data through which platformThis is the governance story that enterprise architects have been waiting for. One policy definition, enforced everywhere.Compute SpecializationWith shared data, each platform can focus on what it does best:Databricks for: - Complex Spark-based data engineering (ETL, data transformation) - ML model training with MLflow integration - Advanced analytics with Python/R notebooks - Streaming workloads with structured streamingFabric for: - Business intelligence with Power BI integration - Real-time dashboards with Eventhouse/KQL - Data warehousing with SQL Analytics Endpoint - Self-service analytics through business user interfacesThe architecture decision shifts from "which platform owns our data" to "which compute engine is optimal for this workload."Migration PathIf you're currently running Databricks with your own data lake, the migration path is straightforward:Enable OneLake connectivity in your Databricks workspace — this is an Azure portal configuration changePoint Unity Catalog to OneLake storage — existing tables migrate with CREATE TABLE ... USING OneLake or the migration utilityVerify Fabric workloads can read the tables — no additional configuration needed; Fabric automatically discovers Unity Catalog tables in OneLakeDecommission duplicate data pipelines — remove the synchronization ETL jobsThe migration is non-destructive. Existing Databricks notebooks continue to work with zero code changes. The data location changes, but the query interface doesn't.What This Means for Malaysian EnterprisesFor Malaysian organizations evaluating or running both Fabric and Databricks, this changes the architecture conversation fundamentally:If you're Azure-first with Fabric: You can now bring in Databricks for complex Spark workloads without duplicating data. Use Fabric for BI and warehousing, Databricks for data engineering and ML — governed by a single Unity Catalog.If you're Databricks-first: You can add Fabric for Power BI integration and business self-service without migrating away from Databricks. Your data stays in Unity Catalog, your governance stays centralized.If you're starting fresh: You no longer need to choose. Start with OneLake as your unified data lake, Unity Catalog as your governance layer, and deploy both Fabric and Databricks workspaces as compute engines. This is the architecture Microsoft and Databricks have jointly endorsed.For enterprises subject to PDPA, the single-copy model also simplifies data residency compliance — your data lives in one place, governed by one policy, in the region you specify.The Competitive LandscapeThis integration fundamentally changes the competitive dynamics:Snowflake now faces a combined Fabric+Databricks proposition with native interoperability — previously their strongest selling point was simplicity over the two-platform headacheGoogle Cloud BigQuery loses the "we have one platform" argument when Fabric+Databricks operate as oneAWS (Redshift + SageMaker) doesn't have an equivalent unified governance storyThe data platform wars didn't end with a winner. They ended with a merger.Key TakeawaysOne copy of data, two compute engines — Unity Catalog in OneLake eliminates data duplication between Databricks and Fabric, reducing costs and complexity.Unified governance is the real win — row-level security, column masking, lineage, and auditing defined once in Unity Catalog apply across both platforms.Compute specialization becomes possible — use Databricks for Spark/ML workloads and Fabric for BI/warehousing without architectural compromise.Migration is non-destructive — existing Databricks notebooks and Fabric reports continue to work with zero code changes after enabling OneLake connectivity.The platform choice is no longer either/or — Malaysian enterprises can deploy both platforms with confidence that governance, data residency, and operational complexity are managed centrally. --- ## Bicep vs. Terraform on Azure in 2026: When to Use Which, and How to Make Them Coexist URL: https://wenfeng.my/bicep-vs-terraform-on-azure-in-2026-when-to-use-which-and-how-to-make-them-coexist/ Author: admin Published: 2026-07-29 06:30:12 Modified: 2026-10-04 15:29:57 If you're managing Azure infrastructure today, you've almost certainly faced this question: Bicep or Terraform?For years the answer was straightforward. Terraform was the multi-cloud heavyweight with a mature ecosystem. Bicep was Microsoft's young challenger — promising, but lacking real-world polish. Fast-forward to 2026, and the picture has shifted. Bicep has gone from promising upstart to genuinely production-hardened. Terraform has weathered licensing turbulence, the OpenTofu fork, and shifting community dynamics.This article isn't a "which is better" flamewar. It's a practical guide covering where each tool excels, where it falls short, and — most importantly — a proven pattern for running both in the same organisation during migration periods.The State of Play in 2026Bicep (current 1.x-era CLI releases)Bicep is Microsoft's DSL for Azure Resource Manager. It transpiles to ARM JSON but spares you writing that JSON by hand. What's changed recently:Azure Verified Modules (AVM) have reached critical mass for common Azure patterns. Microsoft lists 170+ published Bicep resource modules, with additional proposed modules in progress. AVM should be your default starting point for standard Azure resources, while niche services may still require direct resource declarations.Deployment remains Azure Resource Manager based. The supported deployment path is still Azure CLI or Azure PowerShell, for example az deployment group create and az deployment group what-if. The Bicep CLI focuses on build, decompile, publish, restore, lint, format, and newer local validation features such as snapshots.Language improvements include user-defined types, compile-time loops, full ternary expressions, and module scoping that mirrors Azure resource hierarchies.CI/CD tooling is mature: GitHub Actions, Azure DevOps, and GitLab CI can all run Azure CLI/Bicep CLI workflows. Avoid relying on marketplace popularity claims; the important point is that Bicep deployments fit cleanly into standard pipeline runners.Terraform / OpenTofu (current 1.x releases)Terraform remains the undisputed leader for multi-cloud infrastructure. Despite the 2023 BSL licensing controversy:AzureRM provider v4.x is the current major line for Azure resource management. Treat provider upgrades as engineering work: read the v4 upgrade guide, pin versions, and test plans before production rollout.HCP Terraform is a common enterprise remote execution platform, offering policy-as-code, cost estimation, and managed state workflows.OpenTofu remains a viable alternative for teams that cannot accept BSL terms. The projects are diverging; for example, OpenTofu 1.8 introduced early evaluation and provider mocking, while Terraform continues to expand its enterprise and HCP workflow capabilities.Terraform Stacks have moved from beta to general availability in HashiCorp's documentation and are designed to coordinate infrastructure lifecycle at scale. They are useful for multi-component orchestration, but they should still be evaluated against your current HCP Terraform or Terraform Enterprise operating model.The module ecosystem remains enormous: the Terraform Registry exposes more than 20,000 modules overall and more than 2,000 AzureRM modules, with Azure-specific coverage that can still be stronger than AVM for niche services.Practical Code: Deploy the Same Thing in BothLet's compare concrete infrastructure. I'll deploy a standard pattern: a resource group, a storage account with network rules, and an App Service plan and app with a system-assigned managed identity.Bicep// main.bicep param environment string = 'dev' param location string = resourceGroup().location var namePrefix = 'bicepdemo-${environment}' var tags = { environment: environment managedBy: 'bicep' project: 'demo-2026' } // AVM module for storage account module storage 'br/public:avm/res/storage/storage-account:0.22.1' = { name: '${namePrefix}-sa' params: { name: '${namePrefix}sa' location: location tags: tags kind: 'StorageV2' sku: 'Standard_GRS' minimumTlsVersion: 'TLS1_2' networkAcls: { bypass: 'AzureServices' defaultAction: 'Deny' ipRules: [] virtualNetworkRules: [] } blobProperties: { deleteRetentionPolicy: { days: 7; enabled: true } containerDeleteRetentionPolicy: { days: 7; enabled: true } } containers: [ { name: 'appdata'; publicAccess: 'None' } { name: 'logs'; publicAccess: 'None' } ] } } // AVM module for App Service module appService 'br/public:avm/res/web/site:0.15.0' = { name: '${namePrefix}-app' params: { name: '${namePrefix}-app' location: location tags: tags kind: 'app' serverFarmResourceId: appServicePlan.outputs.resourceId siteConfig: { alwaysOn: true minTlsVersion: '1.2' ftpsState: 'FtpsOnly' appSettings: [ { name: 'STORAGE_CONNECTION_STRING'; value: storage.outputs.connectionStrings[0] } { name: 'WEBSITE_RUN_FROM_PACKAGE'; value: '1' } ] } identity: { type: 'SystemAssigned' } } } // Direct resource declaration when AVM doesn't have a module resource appServicePlan 'Microsoft.Web/serverfarms@2023-12-01' = { name: '${namePrefix}-plan' location: location tags: tags sku: { name: 'P1v3'; tier: 'PremiumV3' } } Key observations about the Bicep approach:AVM modules are consumed via br/public:avm/res/... from the public Bicep module registry, and module versions are pinned explicitly. Public AVM modules do not require private registry authentication.The language is declarative but Azure-native — resource types use exact ARM API version strings (Microsoft.Web/serverfarms@2023-12-01). This gives you perfect fidelity with the Azure control plane but zero portability.No state management. There is no terraform state equivalent. Bicep relies entirely on Azure Resource Manager's resource tracking. This is both a blessing (no state file to secure) and a curse (no terraform destroy modelling outside of deployment scopes).Azure deployment what-if is close to terraform plan, but it is executed through ARM deployment commands such as az deployment group what-if and works per deployment scope rather than per Terraform state.Terraform# main.tf terraform { required_version = ">= 1.15" required_providers { azurerm = { source = "hashicorp/azurerm" version = "~> 4.0" } } } provider "azurerm" { features { storage { data_plane_available = true } } } variable "environment" { type = string default = "dev" } locals { name_prefix = "tfdemo-${var.environment}" tags = { environment = var.environment managed_by = "terraform" project = "demo-2026" } } resource "azurerm_resource_group" "main" { name = "${local.name_prefix}-rg" location = "Southeast Asia" tags = local.tags } resource "azurerm_storage_account" "main" { name = "${local.name_prefix}sa" resource_group_name = azurerm_resource_group.main.name location = azurerm_resource_group.main.location account_tier = "Standard" account_replication_type = "GRS" min_tls_version = "TLS1_2" tags = local.tags network_rules { default_action = "Deny" bypass = ["AzureServices"] } } resource "azurerm_storage_container" "appdata" { name = "appdata" storage_account_name = azurerm_storage_account.main.name container_access_type = "private" } resource "azurerm_storage_container" "logs" { name = "logs" storage_account_name = azurerm_storage_account.main.name container_access_type = "private" } resource "azurerm_service_plan" "main" { name = "${local.name_prefix}-plan" resource_group_name = azurerm_resource_group.main.name location = azurerm_resource_group.main.location os_type = "Linux" sku_name = "P1v3" tags = local.tags } resource "azurerm_linux_web_app" "main" { name = "${local.name_prefix}-app" resource_group_name = azurerm_resource_group.main.name location = azurerm_resource_group.main.location service_plan_id = azurerm_service_plan.main.id tags = local.tags site_config { always_on = true ftps_state = "FtpsOnly" min_tls_version = "1.2" application_stack { node_version = "18-lts" } } app_settings = { "STORAGE_CONNECTION_STRING" = azurerm_storage_account.main.primary_connection_string "WEBSITE_RUN_FROM_PACKAGE" = "1" } identity { type = "SystemAssigned" } } Key observations about the Terraform approach:Provider abstraction means the same HCL skills transfer to AWS, GCP, or on-prem. The Terraform lifecycle (init → plan → apply) is identical across clouds.Explicit state via terraform.tfstate or a remote backend. State is the source of truth — Terraform reconciles your config against state, not directly against Azure. This is powerful (drift detection, partial updates, targeted destroys) but demands discipline around locking, backups, and access control.terraform plan gives you a detailed diff of every resource attribute that will change, with no side effects — machine-readable and reviewable.The module registry offers community modules that abstract entire stacks. AVM modules also exist for Terraform — AVM is provider-agnostic.The Decision MatrixHere is the framework I use with teams. Score honestly — the answer is rarely "always" for either side. Criterion Bicep Terraform / OpenTofu Cloud scope Azure only Multi-cloud (AWS, GCP, OCI, on-prem) Learning curve Shallow for Azure devs Moderate; HCL is its own language State management None (ARM manages state) Required (state file/backend must be managed) Drift detection Azure deployment what-if at deployment scope terraform plan from state diff; richer Module ecosystem AVM (mature, Microsoft-verified) Terraform Registry (vast, community + verified) CI/CD integration Native in GitHub/Azure DevOps Excellent everywhere via CLI Team permissions Uses RBAC directly; no extra infra State backend needs its own RBAC Policy enforcement Azure Policy (built-in, deploy-time) Sentinel/OPA (separate config) Licensing MIT (open source) BSL 1.1 (IBM/HashiCorp); OpenTofu is MPL-2.0 Maturity 4+ years, production-ready 10+ years, battle-tested Complex logic Limited (loops, conditions, ternary) Rich (count, for_each, dynamic blocks, functions) Resource lifecycle ARM-native (no destroy tracking) Full lifecycle (CRUD + destroy) When to Pick BicepAzure-only shop. If your org is all-in on Azure with no near-term multi-cloud plans, Bicep is the simplest path. No state backend to manage, no extra RBAC for Terraform storage, and Microsoft's engineering team fixes bugs in the language when they're ARM bugs too.Azure Policy compliance is critical. Bicep deployments are natively evaluated against Azure Policy at deploy time. Terraform requires Sentinel (HCP Terraform) or a separate OPA layer.You hate managing state files. There is no terraform state rm, no import gymnastics, no state migration drama. The mental overhead is meaningfully lower.Small-to-medium Azure deployments. For a team managing 10–50 resource groups, Bicep is faster to write, faster to deploy, and easier to onboard new engineers onto.When to Pick TerraformMulti-cloud or hybrid cloud. Azure + AWS, or Azure + GCP + on-prem. Terraform (or OpenTofu) is the only tool that gives you a unified configuration language across all of them.Complex dependency graphs. Terraform's for_each, count, dynamic blocks, and depends_on handle conditional infrastructure in ways Bicep can't quite match.You need terraform destroy. In ephemeral environments (PR branches, preview environments, sandboxes), the ability to destroy an entire environment with one command is a killer feature. Bicep has no built-in destroy lifecycle.Existing Terraform investment. 200+ Terraform modules, an established state backend, engineers who think in HCL — there is zero business case to migrate. Bicep gains are marginal; rip-and-replace cost is real.Making Them Coexist: The Hybrid PatternHere's the pattern I've seen work successfully at several Malaysian enterprises. It acknowledges that most orgs don't have a greenfield choice — you have Bicep in some teams and Terraform in others, often from M&A inheritance, organic team preference, or a partial migration.The Layer Cake Architecture┌──────────────────────────────────────────┐ │ Terraform (multi-cloud orchestration) │ │ - Network connectivity / hub-spoke │ │ - DNS zones / global resources │ │ - Cross-cloud VPN / Private Link │ ├──────────────────────────────────────────┤ │ Bicep + AVM (Azure workload owners) │ │ - App Service / AKS / Functions │ │ - Storage / SQL / Redis │ │ - Workload-specific networking │ ├──────────────────────────────────────────┤ │ Terraform (shared services / landing) │ │ - Subscription vending / Mgmt groups │ │ - Policy / RBAC base │ │ - Log Analytics / Sentinel │ └──────────────────────────────────────────┘ How this works in practice:The bottom Terraform layer provisions the Azure landing zone — management groups, subscription orchestration, base policies, and connectivity resources (hub VNet, Azure Firewall, ExpressRoute). This is a slow-moving layer updated quarterly.The middle Bicep layer is owned by individual workload teams. They use AVM modules to deploy their application infrastructure. Because the landing zone is already in place, they just need resourceGroup().location and a param for their VNet ID. No state file coordination needed — Azure RBAC handles access control, and ARM sees each deployment as an independent transaction.The top Terraform layer manages global connectivity. If the organisation also runs workloads in AWS, this layer manages cross-cloud transit gateway attachments, DNS resolution across clouds, and Private Link endpoints bridging Azure and on-prem.What Makes This WorkExplicit boundaries. Each layer owns distinct Azure resources with no overlap. Landing zone Terraform never touches a storage account's retention policy — that's the workload team's job. Bicep layers never create a VNet or management group.Azure Policy prevents overlap. Deny assignments at the management group level prevent accidental resource creation outside agreed boundaries. A workload team's Bicep pipeline literally cannot create a VNet because policy rejects it.State isolation. Terraform manages its own state for layers 1 and 3. Bicep has no state. There is no shared state file, no remote state data source cross-contamination, and no accidental terraform destroy taking out the production app along with the network.Migration Path: Terraform to BicepIf you've decided to move workload modules from Terraform to Bicep (a choice accelerating in 2025–2026 for Azure-only teams):Start with stable resources — storage accounts, service plans. Avoid anything with active daily changes.bicep decompile your existing ARM templates (extract ARM JSON from a recent apply, run bicep decompile template.json). The output won't be pretty — expect hand editing — but it saves typing from scratch.Validate against existing resources with az deployment group what-if or the equivalent subscription/management-group scope command. When clean, deploy with az deployment group create. ARM does not simply skip existing resources; it reconciles them to the declared template, so review what-if output carefully before applying.Retire Terraform state with terraform state rm on the migrated resources. Do NOT terraform destroy — that deletes them.Repeat incrementally. One resource type at a time. A weekend cut-over is a recipe for a Monday outage.Pitfalls to Watch ForBicep-Specific"It's just ARM, so drift is invisible." Bicep has no persistent state, so manual changes in the portal are invisible until you run Azure deployment what-if again. Unlike Terraform, there's no terraform refresh. Solution: Use Azure Policy for guardrails and schedule az deployment group what-if or the equivalent scope command in CI to detect drift.Module version management. AVM modules release frequently. Pinning to an old version misses bug fixes; using latest risks breaking changes. Solution: Use Dependabot or Renovate for automated dependency PRs, just like npm or NuGet packages.Incremental vs. Complete mode. Azure deployments default to incremental mode. Complete mode can delete resources in scope that are not declared in the template, which surprises teams from Terraform's surgical model. Solution: Avoid Complete mode in production unless you have a tightly controlled deployment scope and explicit deletion intent.Terraform-SpecificState file leaks. Terraform state contains resource IDs and can contain sensitive values, depending on the resources and providers used. A committed .tfstate in a public repo is a serious exposure. Solution: Use remote backends with encryption at rest, restrict state access, and add *.tfstate to .gitignore.count vs. for_each pain. Switching a resource from count to for_each destroys and recreates all instances. Solution: Use for_each by default — map keys provide stable state addresses.Provider version drift. terraform init -upgrade can pull a provider version that changes API schema and forces unwanted resource recreations. Solution: Pin provider versions with ~> X.Y and commit .terraform.lock.hcl.Hybrid-SpecificCross-layer dependencies. Terraform manages the VNet; Bicep deploys the app. A new subnet for Private Endpoints needs two PRs and two pipelines. Solution: Use Azure Policy to create an "allowed subnets" convention rather than hardcoded IDs. Or use Terraform data sources to export subnet IDs for Bicep parameters.Two teams, one certificate. An App Gateway in Terraform needs its TLS cert from a Key Vault managed by Bicep — chicken-and-egg. Solution: Bootstrapping: Terraform creates the Key Vault in the landing zone, then Bicep populates the secrets. Track this formally in your architecture decision record.ConclusionThe Bicep-versus-Terraform debate in 2026 is a false binary. Both tools are production-grade, both have thriving ecosystems, and both serve you well for Azure infrastructure. The right choice depends on your team's context — not GitHub stars.For Azure-only teams tired of state file management, Bicep + AVM is now a complete solution. For multi-cloud organisations, Terraform (or OpenTofu) remains the clear winner. And for the messy middle where most of us live, the hybrid layer-cake pattern lets you have both without the operational debt of a forced migration.The real skill isn't picking the "best" tool. It's understanding your boundaries well enough to let each tool do what it does best.Key TakeawaysBicep in 2026 is production-ready for Azure-only workloads. AVM modules, Azure CLI/PowerShell deployment workflows, and no separate state backend make it a low-friction option for teams committed to Azure.Terraform remains essential for multi-cloud. No other tool gives you unified configuration language, lifecycle management, and state tracking across Azure, AWS, GCP, and on-prem.You don't have to choose one. The layer-cake pattern — Terraform for landing zones and cross-cloud connectivity, Bicep for workload teams — isolates state, responsibilities, and failure domains.Plan for state, whatever you pick. Bicep trades state management for drift invisibility; Terraform trades simplicity for state complexity. Neither is free.Start with a decision matrix, not a preference. Run your workloads through the criteria in this article. The answer will be different for a startup building on Azure-only vs. a mature enterprise spanning three cloud providers — and that's okay.Law Wen Feng is a Principal Solution Architect based in Malaysia, specialising in cloud infrastructure, DevOps, and platform engineering. He writes about infrastructure-as-code, cloud architecture, and the engineering practices that make teams productive. Views are his own.Got an IaC decision to make or a migration from Terraform to Bicep (or vice versa) on your roadmap? I help Malaysian enterprises design their infrastructure tooling strategy and review landing zone architectures. Connect on LinkedIn. --- ## Azure Brain: What Microsoft’s Reliability System Means for Your Operations URL: https://wenfeng.my/azure-brain-ai-microsofts-digital-twin-for-cloud-reliability/ Author: admin Published: 2026-07-28 11:08:25 Modified: 2026-10-21 15:45:30 Technical source checked on 21 October 2026. Microsoft describes Brain as a system for understanding Azure’s health and supporting operational responses. For a workload owner, that is useful context for platform notifications—not a reason to promise that an application can no longer fail. What Microsoft describes In its July 2 introduction to Brain, Microsoft explains how the system combines platform signals and service relationships to produce health assessments. Documented uses include resource-health notifications, incident routing and deployment gates that can pause harmful rollouts. The article also discusses a digital representation of Azure’s health and future development. Those descriptions should not be converted into a guarantee that every workload failure can be forecast hours or days ahead, or that automatic migration will always prevent customer impact. The cited source does not substantiate those guarantees. No special configuration recipe is established here This article does not provide an App Service or Bicep setting for enabling Brain. The previous example linking zone redundancy to Brain was not supported by the cited evidence and has been removed. Configuration guidance should come from documentation for the specific service and setting being changed. Correction: This revision corrects the source date from July 5 to July 2 and removes unsupported prediction horizons, zero-impact assurances, regional performance claims, SLA-credit claims and the unvalidated configuration example. --- ## Terraform to OpenTofu Migration in 2026 — The Post-BSL Decision Framework for Azure Teams URL: https://wenfeng.my/terraform-to-opentofu-migration-2026-post-bsl-decision-framework/ Author: admin Published: 2026-07-15 23:33:02 Modified: 2026-10-04 15:30:09 ⚡ TL;DRTwo years ago, I wrote about HashiCorp's switch from the Mozilla Public License (MPL) to the Business Source License (BSL) and the community's response — the creation of OpenTofu under the Linux Foundation.The State of Play in 2026The 3-Command MigrationDecision Matrix: OpenTofu vs Terraform Cloud vs BicepGitHub Actions CI/CD with OpenTofuAzure-Specific ConsiderationsTerraform to OpenTofu Migration in 2026 — The Post-BSL Decision Framework for Azure TeamsBy Law Wen Feng | June 2026Two years ago, I wrote about HashiCorp's switch from the Mozilla Public License (MPL) to the Business Source License (BSL) and the community's response — the creation of OpenTofu under the Linux Foundation. At the time, many of us in the Azure ecosystem took a wait-and-see approach. OpenTofu was promising, but it hadn't yet reached feature parity. In 2026, that has changed. If you're leading an Azure-first team in Southeast Asia and still running Terraform, it's time to make a deliberate decision — not just react to blog posts.This article lays out the decision framework I use with clients: a practical, no-hype assessment of what OpenTofu offers today, when Terraform (and IBM's stewardship) still makes sense, and when Azure-native tooling like Bicep deserves a second look.The State of Play in 2026What Happened with BSLIn August 2023, HashiCorp relicensed Terraform and its entire toolchain from MPL 2.0 to the Business Source License v1.1. The BSL is not open source by the Open Source Definition — it permits use but restricts competitive commercial use. If you're building a product that competes with HashiCorp Cloud Platform (HCP), you can't use the BSL-licensed code.In 2025, IBM completed its acquisition of HashiCorp. The acquisition didn't change the BSL terms, but it raised legitimate questions about long-term stewardship. For most end-users running terraform apply in their CI/CD pipelines, the BSL doesn't directly restrict usage. But for teams building internal platforms, CI tooling, or drift-detection systems that wrap Terraform, the license creates real legal and architectural constraints.My take: if you're an enterprise team simply applying infrastructure, the BSL itself may not be your primary concern. But the governance model matters. IBM's priorities shift. The Linux Foundation's governance of OpenTofu doesn't.OpenTofu's Feature Parity MilestoneAs of OpenTofu 1.10 (released early 2026), the project has reached genuine feature parity with Terraform for the vast majority of Azure workloads. Here's what matters:State Encryption. Since OpenTofu 1.7, state files can be natively encrypted at rest using AES-256-GCM. This was a community priority from day one. In 1.10, external key provider support is fully mature — you can integrate with Azure Key Vault, HashiCorp Vault, or any secrets manager through a simple command-line provider. This is something Terraform still doesn't offer natively.Removed Blocks. The removed block lets you clean up state without destroying infrastructure. This is incredibly useful for decommissioning resources that were managed by OpenTofu but are being retired or moved to another system.removed { from = azurerm_network_security_rule.legacy lifecycle { destroy = false } }Dynamic Provider Configurations. OpenTofu 1.8 introduced early evaluation of variables, letting you use var and local references in backend blocks and module sources. Before this, you couldn't parameterize your state backend. Now you can.terraform { backend "azurerm" { resource_group_name = var.state_resource_group storage_account_name = var.state_storage_account container_name = var.state_container key = "${var.environment}/main.tfstate" } }OCI Registry Support. OpenTofu 1.10 can pull providers and modules from OCI-compliant container registries. For air-gapped Azure environments (common in government and banking sectors across Malaysia and Singapore), this eliminates the dependency on public registries.Native S3 State Locking. No more DynamoDB dependency for S3 backends. OpenTofu uses S3's conditional writes natively. For Azure teams, this matters less (we use Azure Blob), but it signals the project's engineering maturity.The bottom line: OpenTofu is no longer a "promising fork." It's a production-ready tool with features Terraform doesn't have.The 3-Command MigrationI know this sounds too simple. It's not. The real work is in the preparation. But the actual tool migration from Terraform to OpenTofu is genuinely three commands:# 1. Back up your state (do this regardless) az storage blob download \ --account-name mystataccount \ --container-name tfstate \ --name main.tfstate \ --file ./backups/main.tfstate.bak # 2. Install OpenTofu and initialize tofu init upgrade # 3. Validate with a plan — should be identical to your last terraform plan tofu plan -out=tfplanThat's it. OpenTofu reads the same HCL configuration files, the same state format, and the same provider binaries. If tofu plan shows a clean plan identical to what terraform plan showed, you've migrated.The preparation work is where the real effort lives:Audit your provider versions. OpenTofu supports the same Terraform Registry providers. Pin your versions explicitly and confirm they're available.Review any Terraform-specific features you use. If you're using Terraform Cloud state locking, HCP Terraform workspaces, or Sentinel policies, those require separate planning.Update your CI/CD pipelines. Swap terraform commands for tofu (or alias it — OpenTofu respects TF_CLI_ARGS env vars).Test with a non-critical environment first. I recommend starting with a dev or staging subscription.Decision Matrix: OpenTofu vs Terraform Cloud vs BicepThis is the question I get asked most often. There's no universal answer, but here's how I frame it:Choose OpenTofu if:You want permissive, OSI-approved licensing (MPL 2.0) with no commercial restrictions.Your team builds internal platforms or wraps IaC in custom tooling — BSL would be a legal concern.State encryption is a compliance requirement (it's native in OpenTofu, not in Terraform).You prefer Linux Foundation governance over IBM stewardship.You need fast release cadence — OpenTofu ships monthly, Terraform quarterly.You're in an air-gapped or highly regulated Azure environment where OCI registry support matters.Choose Terraform Cloud (HCP Terraform) if:Your team already pays for HCP Terraform and deeply uses its managed state, policy-as-code (Sentinel), and run trigger features.You need vendor-backed SLAs for compliance audits in regulated industries.Your workloads rely on Terraform-specific features like Stacks or native Sentinel integration.Migration cost outweighs licensing concern — if BSL doesn't affect your use case, stay.Choose Bicep if:You are 100% Azure-native with no multi-cloud requirements.You want first-class ARM template integration and Azure CLI tooling.Your team already has strong Bicep/Azure CLI skills and no HCL investment.You prefer Microsoft's governance over community or third-party governance.You're building greenfield projects without existing Terraform state.For most of my clients — Azure-first teams in Southeast Asia with existing Terraform investments — the answer is OpenTofu. Bicep is excellent for greenfield Azure-only work, but switching IaC languages mid-flight is a much bigger lift than swapping the CLI binary.GitHub Actions CI/CD with OpenTofuIf you're running Azure deployments through GitHub Actions, here's a practical CI/CD pattern using OpenTofu. This is a simplified version of what I deploy for clients:name: OpenTofu Azure Deploy on: push: branches: [main] pull_request: branches: [main] env: ARM_CLIENT_ID: ${{ secrets.AZURE_CLIENT_ID }} ARM_CLIENT_SECRET: ${{ secrets.AZURE_CLIENT_SECRET }} ARM_SUBSCRIPTION_ID: ${{ secrets.AZURE_SUBSCRIPTION_ID }} ARM_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }} TF_VERSION: "1.10.3" jobs: plan: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Setup OpenTofu uses: opentofu/setup-opentofu@v1 with: tofu_version: ${{ env.TF_VERSION }} tofu_wrapper: false - name: OpenTofu Init run: tofu init -backend-config="backend.tfvars" - name: OpenTofu Format Check run: tofu fmt -check -recursive - name: OpenTofu Validate run: tofu validate - name: OpenTofu Plan id: plan run: tofu plan -no-color -out=tfplan continue-on-error: true - name: Upload Plan Artifact uses: actions/upload-artifact@v4 with: name: tfplan path: tfplan apply: needs: plan runs-on: ubuntu-latest if: github.ref == 'refs/heads/main' steps: - uses: actions/checkout@v4 - name: Setup OpenTofu uses: opentofu/setup-opentofu@v1 with: tofu_version: ${{ env.TF_VERSION }} tofu_wrapper: false - name: Download Plan uses: actions/download-artifact@v4 with: name: tfplan - name: OpenTofu Apply run: tofu apply -auto-approve tfplanA few things to note about this pipeline:The opentofu/setup-opentofu action is maintained by the OpenTofu project. It's the equivalent of hashicorp/setup-terraform.We store the plan artifact between jobs — this ensures the apply uses the exact same plan that was reviewed.For state encryption with Azure Key Vault, add this to your backend configuration:terraform { encryption { key_provider "external" "azkv" { command = ["az", "keyvault", "secret", "show", "--vault-name", "mykeyvault", "--name", "opentofu-state-key", "--query", "value", "-o", "tsv"] } method "aes_gcm" "state" { keys = key_provider.external.azkv } state { method = method.aes_gcm.state } } }This encrypts your state file end-to-end using a key stored in Azure Key Vault. If your compliance team has asked "is our state encrypted at rest?" — this is the answer.Azure-Specific ConsiderationsWorking in the Azure ecosystem with OpenTofu brings a few Azure-specific nuances worth calling out:Provider Compatibility. The azurerm provider is fully compatible with OpenTofu. It's MPL-licensed and available in the OpenTofu registry. No changes needed to your provider blocks.State Backend. Azure Blob Storage works as a backend for both tools. If you're using the azurerm backend, your configuration carries over unchanged:terraform { backend "azurerm" { resource_group_name = "rg-terraform-state" storage_account_name = "stterraformstate" container_name = "tfstate" key = "prod/aks.tfstate" } }Azure Identity. OpenTofu supports the same Azure authentication mechanisms — ARM_CLIENT_ID/ARM_CLIENT_SECRET for service principals, or Workload Identity for GitHub Actions and AKS workloads. No changes needed.Bicep as a Co-Existence Strategy. I've seen teams successfully use Bicep for subscription-level governance (management groups, policy assignments, RBAC) while using OpenTofu for resource-level infrastructure (AKS clusters, databases, networking). This hybrid approach gives you the best of both worlds — Microsoft-native governance with OpenTofu's flexibility and licensing.Private Endpoints and Air-Gapped Environments. OpenTofu 1.10's OCI registry support is particularly valuable here. You can mirror providers to Azure Container Registry and configure tofu init to pull from your private registry:provider_installation { oci_mirror { repository_template = "myacr.azurecr.io/opentofu-providers/${namespace}/${type}" include = ["registry.opentofu.org/azurerm/*"] } }What I Tell My ClientsAfter walking through the technical details, here's the practical advice I give Azure-first teams in the region:If you're running Terraform today and it works, you don't need to panic. The BSL doesn't force you to do anything tomorrow. But you should plan a migration window.The migration is a weekend project, not a quarter-long initiative. Three commands to swap the binary. A week of testing. Then you're done.State encryption alone justifies the move for many organizations. If your security team has ever asked about state file protection, OpenTofu gives you a native, auditable answer.Don't use this as an excuse to rewrite everything in Bicep. Unless you have zero existing Terraform investment and are 100% Azure-only, the migration cost isn't worth it. OpenTofu is a drop-in replacement, not a rewrite.Governance matters more than features. OpenTofu's Linux Foundation governance gives you a community-driven roadmap that isn't beholden to any single vendor's quarterly earnings. In a region where vendor lock-in is a real concern, this matters.Key TakeawaysOpenTofu 1.10 has reached genuine feature parity with Terraform for Azure workloads, including state encryption, removed blocks, dynamic provider configurations, and OCI registry support — features Terraform doesn't offer natively.The migration is a drop-in replacement. Swap the terraform binary for tofu, run tofu init upgrade, and validate with tofu plan. Your HCL code, state files, and providers carry over unchanged.The BSL license doesn't force you to migrate, but it should inform your decision. If your team builds internal platforms, wraps IaC in custom tooling, or operates in regulated environments, the licensing and governance model matters.For Azure-first teams, OpenTofu with Azure Blob backends and Azure Key Vault encryption is the pragmatic choice. It gives you native state encryption, air-gapped registry support, and Linux Foundation governance — all without changing your existing architecture.Consider a hybrid strategy: Bicep for Azure subscription governance, OpenTofu for resource-level infrastructure. This gives you Microsoft-native policy integration where it matters and OpenTofu's flexibility everywhere else.Law Wen Feng is a Principal Solution Architect based in Malaysia, specializing in cloud-native architecture and infrastructure automation for enterprise teams across Southeast Asia. Find more at wenfeng.my/. --- ## Microsoft Conductor: Deterministic Multi-Agent Orchestration in YAML URL: https://wenfeng.my/microsoft-conductor-deterministic-multi-agent-orchestration-yaml/ Author: admin Published: 2026-07-05 23:33:02 Modified: 2026-10-04 15:31:15 ⚡ TL;DREvery multi-agent system eventually confronts the same question: who decides what runs next?The Case Against LLM-Driven OrchestrationHow Conductor WorksWhen Deterministic Beats LLM-DrivenWhen LLM-Driven Routing WinsHybrid Architecture: The Real AnswerMicrosoft Conductor: Deterministic Multi-Agent Orchestration in YAMLEvery multi-agent system eventually confronts the same question: who decides what runs next?In LLM-driven orchestration frameworks like LangGraph, CrewAI, or Microsoft Agent Framework, the answer is "the model." An LLM reads the current state, decides the next action, and routes to the appropriate agent. This works beautifully when you need dynamic, context-dependent routing. It fails spectacularly when you need predictability, cost control, and reproducibility.Microsoft released Conductor on May 14, 2026 as an open-source CLI (MIT license) that takes the opposite approach: define your multi-agent workflows in YAML with deterministic routing — zero tokens spent on orchestration decisions.For teams building production agent systems, Conductor forces a fundamental architectural question: when should you let the LLM decide, and when should the routing be baked in?The Case Against LLM-Driven OrchestrationLLM-driven routing has three problems that compound at scale:1. Non-DeterminismRun the same workflow twice with the same input, and the LLM might route to different agents. In development, this is annoying. In production, it's unacceptable. Your compliance officer doesn't want to hear that the system "might" route to the right agent.2. CostEvery routing decision is an LLM call. In a 10-step workflow, you're spending 10 tokens (plus context) just on orchestration. For a pipeline that runs thousands of times daily, this adds up. The orchestration cost can exceed the actual agent work.3. LatencyEach LLM routing decision adds 200-500ms of latency. In a sequential workflow with 5 steps, that's 1-2.5 seconds of pure orchestration overhead — before any agent has done useful work.How Conductor WorksConductor defines workflows as YAML files with two core sections: a workflow header and a list of agents.AgentsEach agent is a unit of work — an LLM prompt, a script step, or a human gate:workflow: name: content-pipeline entry_point: researcher agents: - name: researcher prompt: | Research the topic: {{ workflow.input.topic }} output: findings: type: string routes: - to: writer - name: writer prompt: | Write an article based on: {{ researcher.output.findings }} output: draft: type: string routes: - to: $endRoutingDeterministic routing based on conditions, defined on each agent's routes list. Jinja2 expressions evaluate at runtime — no LLM involved:agents: - name: reviewer prompt: "Review this draft: {{ writer.output.draft }}" output: quality_score: type: number routes: - to: writer when: "{{ reviewer.output.quality_score < 7 }}" - to: $end when: "{{ reviewer.output.quality_score >= 7 }}"First matching condition wins. The writer agent re-runs until quality passes the threshold — a built-in loop without any special loop syntax.Script StepsRun shell commands and route on exit code or parsed JSON stdout: - name: lint type: script command: "python lint.py {{ writer.output.draft }}" routes: - to: writer when: "{{ lint.exit_code != 0 }}" - to: $endIntegrationThe YAML file IS the workflow. Run it with:conductor run workflow.yaml --input topic="Azure Kubernetes Service"The CLI handles execution, state passing between agents, and output collection.When Deterministic Beats LLM-DrivenContent Pipeline WorkflowsThis is the exact pattern we use for the wenfeng.my/ content pipeline: Research → Strategy → Writing → SEO Review → Social Content → Publishing. The routing is fixed. The agents change based on the step. LLM-driven routing adds cost and latency without value.Conductor handles this cleanly: each agent is an LLM call with a specific prompt and model, the routing between agents is deterministic, and the only "intelligence" is in the agent content, not the orchestration.Data Processing PipelinesETL workflows where the sequence is known: Extract → Validate → Transform → Load → Verify. The data might vary, but the routing doesn't. Using an LLM to decide "should I transform next or load next?" wastes tokens on a decision that's always the same.CI/CD Agent WorkflowsBuild → Test → Security Scan → Deploy → Verify. Again, fixed routing. The LLM adds value in the security scan (analyzing code) but not in deciding whether to test after building.Compliance-Governed WorkflowsWhen regulators need to see exactly how decisions are made, deterministic routing provides an auditable, reproducible flow. LLM routing decisions are probabilistic and difficult to explain in compliance documentation.When LLM-Driven Routing WinsCustomer Support TriageA customer sends a message. Should it go to billing support, technical support, or escalation? The routing depends on the message content, customer history, and context. This is genuinely dynamic and benefits from LLM judgment.Incident ResponseAn alert fires. Is this a known issue (route to runbook), a new issue (route to investigation), or a security incident (route to SOC)? The routing depends on correlating multiple signals — a good use of LLM reasoning.Adaptive Research WorkflowsA researcher asks a question. The next step depends on what's found: if the answer is clear, summarize; if not, expand the search; if conflicting, compare sources. This branching is inherently dynamic.Hybrid Architecture: The Real AnswerThe most effective production systems use both patterns:YAML Workflow (Conductor) ├── Agent 1: Deterministic routing ├── Agent 2: Deterministic routing ├── Agent 3: Dynamic routing (LLM decides next action) │ ├── Branch A: Deterministic │ └── Branch B: Deterministic └── Agent 4: Deterministic routingUse Conductor for the fixed structure. Use LLM routing within specific agents where genuine decision-making is needed. This gives you the cost and latency benefits of deterministic orchestration with the intelligence of LLM routing where it matters.Getting Started with ConductorInstallation is straightforward — Conductor is a Python CLI tool (Python 3.12+) installed via a one-line script:curl -sSfL https://aka.ms/conductor/install.sh | sh conductor run --helpAlternatively, install directly with uv:uv tool install git+https://github.com/microsoft/conductor.gitThe YAML syntax is deliberately simple. If you've written GitHub Actions workflows, Docker Compose files, or Ansible playbooks, Conductor YAML will feel familiar. The key difference: Conductor YAML is executable. It's not documentation that someone needs to translate into code — it IS the code.For teams already running multi-agent pipelines (like the content pipeline architecture many organizations use), Conductor provides a migration path from custom orchestration scripts to a standardized, version-controlled workflow definition. The YAML file lives in your repository, gets reviewed in PRs, and runs identically in dev and production.Key TakeawaysNot every routing decision needs an LLM — deterministic YAML workflows eliminate orchestration cost, latency, and non-determinism for pipelines with fixed structure.Conductor is a CLI tool, not a framework — define workflows in YAML, run them with conductor run, and the tool handles execution. Minimal dependencies, no heavyweight runtime.Hybrid orchestration is the production pattern — use deterministic routing for fixed workflow structure, LLM routing only where genuine dynamic decision-making adds value.Cost reduction is significant — eliminating LLM routing calls per workflow execution saves substantial orchestration tokens in fixed-structure pipelines.Compliance and reproducibility improve — deterministic routing produces identical execution paths for identical inputs, which is what auditors want to see. --- ## Azure Resiliency Gets Real — Availability Zone Drills, Zonal Recovery Orchestration URL: https://wenfeng.my/azure-resiliency-in-2026-availability-zone-drills-and-zonal-recovery-that-actually-work/ Author: admin Published: 2026-06-26 23:33:19 Modified: 2026-10-04 15:30:09 ⚡ TL;DRAzure Resiliency, launched in June 2026, is the first platform-native toolset for validating Availability Zone resilience.What Azure Resiliency Actually ProvidesThe Malaysian Enterprise ContextPractical AZ Drill ChecklistIntegration with Existing DR StrategyKey TakeawaysAzure Resiliency Gets Real — Availability Zone Drills, Zonal Recovery OrchestrationAzure Resiliency, launched in June 2026, is the first platform-native toolset for validating Availability Zone resilience. Before this, testing AZ failover meant either hoping nothing broke in production or building expensive custom simulation environments. Now Azure provides built-in AZ Down Drills, Zonal Resiliency Goals with AI-powered recommendations, and automated Recovery Orchestration Plans.For Malaysian enterprises that deployed across Availability Zones in Malaysia West but never actually tested failover — which is most of them — this changes the game.What Azure Resiliency Actually ProvidesAvailability Zone Down DrillsThe most significant feature: you can simulate an AZ outage for specific service groups without impacting production traffic.How it works:1. Define a "service group" — a logical collection of Azure resources that must survive an AZ failure together2. Select the AZ to simulate down3. Azure Resiliency blocks traffic to the selected resources in the target AZ4. Monitor whether your remaining resources in other AZs absorb the load5. Azure measures recovery time and reports actual RTOKey difference from Chaos Studio: Azure Chaos Studio disrupts individual resources (kill a VM, block a network port). Azure Resiliency simulates AZ-level failure — an entire availability zone going dark. This tests the architectural resilience of your multi-AZ deployment, not just individual component fault tolerance.Example service group definition:# Create a service group for your web application tier az rest --method put \ --url "https://management.azure.com/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Resiliency/serviceGroups/web-app-tier?api-version=2026-01-01" \ --body '{ "properties": { "members": [ "/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Compute/virtualMachines/web-vm-1", "/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Network/loadBalancers/web-lb", "/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Network/applicationGateways/web-agw" ] } Zonal Resiliency GoalsAzure Resiliency assesses your resource group or subscription against defined resiliency goals:Built-in goals:All production VMs are deployed across 2+ AZsLoad balancers distribute traffic across AZsManaged disks use zone-redundant storage (ZRS)Databases have cross-AZ replicasStorage accounts use GRS or ZRSAI Resiliency Agent: An AI agent that analyzes your deployment, identifies gaps, and provides cost-aware recommendations. For example: "Your web VMs are single-AZ. Deploying a second VM in AZ2 would cost $0.096/hour and eliminate this single point of failure."Cost-aware recommendations are critical: Many enterprises skip AZ deployment because they assume the cost is prohibitive. The AI agent shows the actual cost difference — often 10-20% more than single-AZ, not 2x as expensive.Recovery Orchestration PlansAutomated recovery plans that execute across multiple resource types:Supported resources:Virtual Machines (with custom script extensions)Azure SQL Database (failover groups)Cosmos DB (multi-region failover)Azure Kubernetes Service (node pool redistribution)Azure Storage (failover to secondary region)Plan structure:Recovery Plan: WebApp-Failover ├── Step 1: Verify source AZ is degraded ├── Step 2: Start VMs in target AZ (parallel) ├── Step 3: Wait for VM health checks ├── Step 4: Update load balancer backend pool ├── Step 5: Trigger SQL failover group ├── Step 6: Verify application health Custom scripts: Each step can run Azure CLI commands, PowerShell scripts, or Azure Automation runbooks. This handles application-specific recovery logic that platform-native failover does not cover.The Malaysian Enterprise ContextMalaysia West AZ AvailabilityMalaysia West launched with Availability Zones, but not all services are zone-redundant. Before using Azure Resiliency, verify:# Check zone-redundant services in Malaysia West az provider show \ --namespace Microsoft.Compute \ --query "resourceTypes[].zoneMappings" \ Common single-AZ gaps in Malaysia West:Some VM series may not be available in all 3 AZsCertain managed disk SKUs may not support ZRSSome Azure PaaS services may not be zone-redundant at launchVerification approach: Before running AZ drills, confirm that your critical resources actually span multiple AZs. An AZ drill against single-AZ resources will cause an actual outage, not a simulation.PDPA and DR TestingMalaysia's PDPA requires organizations to implement measures to prevent unauthorized access, disclosure, or loss of personal data. DR testing is an implicit requirement — you cannot claim compliance if you have never validated your recovery procedures.Azure Resiliency provides an audit trail of AZ drills, including:Which resources were affectedActual recovery time achievedResources that failed to recoverRecommendations for improvementThis documentation supports PDPA compliance audits by demonstrating that DR procedures are tested, not just documented.Cost of ResiliencyAzure Resiliency itself has no additional cost — it is included in your Azure subscription. The cost is in the underlying infrastructure:| Component | Single-AZ Cost | Multi-AZ Cost | Difference ||-----------|---------------|---------------|------------|| D2s_v5 VM (×2) | $0.192/hr | $0.192/hr (2 AZs) | 0% || ZRS Managed Disk | $0.076/GB/mo | $0.095/GB/mo | +25% || Zone-redundant Load Balancer | $0.025/hr | $0.025/hr | 0% || SQL failover group | Single: $0.34/hr | Paired: $0.68/hr | +100% |Key insight: The cost of AZ resilience is primarily in database replication, not compute. VMs and load balancers have minimal cost difference; the real investment is in database zone redundancy.Practical AZ Drill ChecklistPre-Drill1. Map your service groups. Identify which resources form a logical unit and must survive AZ failure together.2. Verify multi-AZ deployment. Confirm all critical resources exist in 2+ AZs.3. Set up monitoring. Ensure Azure Monitor alerts are configured for the target resources.4. Notify stakeholders. Even though the drill is simulated, notify teams that monitoring may show anomalies.5. Define success criteria. What RTO are you testing against? 5 minutes? 30 minutes? 1 hour?During Drill1. Start the drill. Use Azure Portal or CLI to initiate the AZ Down Drill.2. Monitor in real-time. Watch Azure Monitor dashboards for resource health, traffic distribution, and error rates.3. Document observations. Note any unexpected behavior — failover delays, error spikes, capacity issues.4. Measure actual RTO. How long did it take for the application to fully recover in the target AZ?Post-Drill1. Compare actual vs. target RTO. Did recovery meet your SLA?2. Identify gaps. Which resources failed to recover? Which took longer than expected?3. Update recovery plans. Fix any issues discovered during the drill.4. Schedule recurring drills. Azure Resiliency supports scheduled drills — automate monthly or quarterly AZ failover tests.5. Document for compliance. Archive drill results for PDPA and internal audit purposes.Integration with Existing DR StrategyAzure Resiliency complements (does not replace) Azure Site Recovery:| Capability | Azure Resiliency | Azure Site Recovery ||-----------|-----------------|-------------------|| Scope | AZ failover (same region) | Cross-region failover || Simulation | Non-disruptive drills | Test failover (isolated) || Automation | Recovery orchestration plans | Recovery plans || Best for | AZ resilience validation | Regional DR |Recommended approach:1. Use Azure Resiliency for monthly AZ drills — validate that your multi-AZ deployment actually works2. Use Azure Site Recovery for quarterly cross-region DR tests — validate that you can recover in another region3. Document both for compliance auditsKey Takeaways1. Azure Resiliency enables non-disruptive AZ failover simulation — the first platform-native tool for testing multi-AZ resilience without impacting production.2. Zonal Resiliency Goals with AI recommendations make it practical to identify and fix resilience gaps, with cost-aware guidance that shows the real price of redundancy.3. Recovery Orchestration Plans automate multi-resource failover across VMs, SQL, Cosmos DB, AKS, and Storage — handling application-specific recovery that platform-native failover cannot.4. The cost of AZ resilience is primarily in database replication, not compute — VMs and load balancers have minimal cost difference across zones.5. Schedule recurring drills — Azure Resiliency supports automated monthly AZ drills that produce compliance-ready documentation for PDPA and internal audits. --- ## Bicep vs Terraform in 2026 — The Enterprise Verdict for Azure-First Teams URL: https://wenfeng.my/bicep-vs-terraform-in-2026-the-enterprise-verdict-for-azure-first-teams/ Author: admin Published: 2026-06-24 23:32:50 Modified: 2026-10-04 15:30:10 The Bicep-vs-Terraform debate has been settled — not by a winner-take-all outcome, but by a clear division of territory. In 2026, the honest answer is no longer "which tool is better" but "which tool matches your team's cloud strategy, governance requirements, and skill base." I have deployed both tools across multiple Azure enterprise environments, including Azure-first teams that run them side by side. This article is the practical verdict: what changed in 2026, how the two tools compare where it actually matters, and real Bicep and Terraform code you can drop into your own pipelines. If you are an Azure-first team — which describes most enterprises in Malaysia and Singapore that I work with — the decision framework at the end should save you a painful migration in either direction. The Problem: A Tooling Decision That Becomes an Operating Model Infrastructure-as-code choices look technical, but they are really operating-model decisions. Pick the wrong tool and you inherit: A state management burden that your platform team must own forever, or one they never have to think about. A hiring profile. Terraform skills are abundant and cloud-agnostic; Bicep skills are Azure-specific but easier for existing Azure admins to pick up. A governance toolchain. Policy-as-code, drift detection, and audit trails come from different places depending on the tool. A license posture. Since HashiCorp's move to the Business Source License — and the subsequent IBM acquisition — Terraform's licensing is a genuine governance question for enterprises with strict open-source policies. Teams that treat this as a cosmetic choice usually regret it eighteen months later, when they are either fighting state corruption on a tool they never needed, or rebuilding multi-cloud modules in a tool that only speaks Azure. The 2026 State of Play Bicep: Azure-Native and Mature Bicep has matured significantly since its 2021 GA, and in 2026 it is a genuinely complete platform tool: Day-zero ARM support. When Azure releases a new resource provider or API version, Bicep supports it immediately — the compiler reads the resource types directly from Azure. Terraform's azurerm provider can lag days to weeks behind on new resources. No state management. Bicep uses Azure Resource Manager as the state store. There is no .tfstate file, no state locking, no state backend to configure, and no state corruption to recover from. You deploy, and Azure tracks what exists. Native RBAC and identity integration. Managed identities, role assignments, and Azure Policy all work first-class. You can assign RBAC in the same template that creates the resource. What-if operations. az deployment group what-if previews every change against the live ARM control plane before anything is touched. Azure Verified Modules (AVM). A curated, Microsoft-backed library of production-ready modules for landing zones, networking, AKS, and data platforms, with standards that span both Bicep and Terraform registries. Terraform: Multi-Cloud King, With Licensing Asterisks Terraform remains the dominant multi-cloud IaC tool, but the landscape shifted under it: BSL license, IBM ownership. Terraform is source-available under the Business Source License, not open source in the OSI sense. For most enterprises this is fine in practice, but for organizations with strict open-source procurement policies it requires a formal review. OpenTofu as the open fork. The Linux Foundation's OpenTofu tracks Terraform compatibility and has shipped exclusive features Terraform lacks — end-to-end state encryption and early variable/locals evaluation. (Both tools now support removed blocks, which shipped in Terraform 1.7 and the corresponding OpenTofu releases.) Migration from Terraform is typically a matter of renaming .tf files' expectations and swapping binaries; for standard configurations it is effectively a drop-in. Unmatched provider ecosystem. Over 4,000 providers cover AWS, GCP, Azure, Kubernetes, Cloudflare, GitHub, Datadog, and hundreds of SaaS platforms. No other tool comes close. Mature state tooling. Terraform Cloud, Spacelift, env0, and Atlantis give you enterprise state management, drift detection, run approvals, and policy-as-code (Sentinel/OPA) out of the box. The Decision Matrix Choose Bicep when: Your estate is Azure-only. If everything you run is on Azure — the reality for most Microsoft-centric enterprises — Bicep eliminates the entire state-management category of problems. Your team is Azure-admin-first. Azure administrators can become productive in Bicep in days, because the resource model maps 1:1 to what they already know from the portal and ARM. You need day-zero Azure features. If you adopt new Azure services early (AI Foundry, Fabric capacities, new networking features), Bicep supports them the day they ship. Compliance favors first-party tooling. Some regulated industries prefer Microsoft-supported tooling with a single vendor escalation path. Choose Terraform when: You are genuinely multi-cloud. Azure + AWS or Azure + GCP estates are where Terraform earns its complexity. One language, one workflow, every cloud. You have existing Terraform investment. Established modules, pipelines, and expertise are worth more than marginal Bicep benefits. Do not migrate for ideology. You manage non-Azure resources. DNS records, SaaS configuration, Kubernetes providers — anything outside Azure's resource model. You need advanced state surgery. import blocks, moved blocks, and mature state manipulation are stronger in Terraform today. You run Terraform Cloud or equivalent. Workspace models, Sentinel policies, and run triggers provide governance Bicep only matches via separate Azure services. Choose hybrid when you are Azure-first but not Azure-only — which is most large enterprises. The pattern that works: Bicep owns the Azure platform layer (management groups, subscriptions, policies, hub networking), and Terraform owns workload-level and cross-boundary resources. More on this below. Practical Example 1: The Same Storage Account in Both Tools Comparing tooling is easier when you see identical infrastructure in both languages. Here is a storage account with HTTPS-only and a private container. Bicep (storage.bicep): param location string = resourceGroup().location param storageName string = 'stwenfengprod01' resource storage 'Microsoft.Storage/storageAccounts@2023-01-01' = { name: storageName location: location sku: { name: 'Standard_LRS' } kind: 'StorageV2' properties: { supportsHttpsTrafficOnly: true minimumTlsVersion: 'TLS1_2' allowBlobPublicAccess: false } } output storageId string = storage.id Deploy it: az deployment group create \ --resource-group rg-production \ --template-file storage.bicep \ --parameters location=westus2 Terraform (storage.tf): resource "azurerm_storage_account" "prod" { name = "stwenfengprod01" resource_group_name = "rg-production" location = "westus2" account_tier = "Standard" account_replication_type = "LRS" https_traffic_only_enabled = true min_tls_version = "TLS1_2" allow_nested_items_to_be_public = false } output "storage_id" { value = azurerm_storage_account.prod.id } Deploy it: terraform init terraform plan -out=tfplan terraform apply tfplan The Bicep version has no init step, no provider block, and no state file. The Terraform version looks nearly identical — until you need a second cloud, at which point Bicep stops being an option entirely. That asymmetry is the whole argument in miniature. Practical Example 2: CI/CD Pipelines GitHub Actions with Bicep name: Deploy Azure Infrastructure (Bicep) on: push: branches: [main] paths: ['infra/bicep/**'] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Azure Login uses: azure/login@v2 with: client-id: ${{ secrets.AZURE_CLIENT_ID }} tenant-id: ${{ secrets.AZURE_TENANT_ID }} subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }} - name: What-If run: | az deployment group what-if \ --resource-group rg-production \ --template-file infra/bicep/main.bicep \ --parameters @infra/bicep/params.prod.json - name: Deploy run: | az deployment group create \ --resource-group rg-production \ --template-file infra/bicep/main.bicep \ --parameters @infra/bicep/params.prod.json Note what is absent: no terraform init, no state backend configuration, no lock handling. Authentication plus deployment is the entire pipeline. GitHub Actions with Terraform name: Deploy Azure Infrastructure (Terraform) on: push: branches: [main] paths: ['infra/terraform/**'] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Setup Terraform uses: hashicorp/setup-terraform@v3 with: terraform_version: "1.10" - name: Azure Login uses: azure/login@v2 with: client-id: ${{ secrets.AZURE_CLIENT_ID }} tenant-id: ${{ secrets.AZURE_TENANT_ID }} subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }} - name: Terraform Init run: terraform init working-directory: infra/terraform - name: Terraform Plan run: terraform plan -out=tfplan working-directory: infra/terraform - name: Terraform Apply run: terraform apply tfplan working-directory: infra/terraform Every extra step here exists because of state: init wires the backend, plan reads state, apply writes it. That is not a flaw — explicit state gives you portability — but it is operational surface area your team owns. Practical Example 3: The Hybrid Landing Zone The pattern I deploy most often for Azure-first enterprises with a few cross-boundary needs: name: Deploy Infrastructure (Hybrid) on: push: branches: [main] jobs: bicep-landing-zone: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: azure/login@v2 with: client-id: ${{ secrets.AZURE_CLIENT_ID }} tenant-id: ${{ secrets.AZURE_TENANT_ID }} subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }} - name: Deploy platform layer run: | az deployment sub create \ --location westus2 \ --template-file infra/bicep/landing-zone.bicep \ --parameters @infra/bicep/lz-params.json terraform-workloads: needs: bicep-landing-zone runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: hashicorp/setup-terraform@v3 - uses: azure/login@v2 with: client-id: ${{ secrets.AZURE_CLIENT_ID }} tenant-id: ${{ secrets.AZURE_TENANT_ID }} subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }} - name: Deploy workloads run: terraform init && terraform apply -auto-approve working-directory: infra/terraform/app The ownership rule that makes hybrid work: one resource, one owner. Bicep owns the subscription, policies, and hub VNet; Terraform owns the AKS cluster, databases, and third-party integrations inside that landing zone. The moment both tools can touch the same resource, you get drift that neither tool can detect — each sees only the state it manages. Cost and Operational Comparison Factor Bicep Terraform State management None — Azure ARM is the state store State file + backend required Learning curve Low for Azure admins Moderate — HCL + state concepts New Azure resource support Day-zero Days-to-weeks lag Module ecosystem Azure Verified Modules 4,000+ providers Multi-cloud No Yes — the core strength Governance tooling Azure Policy + what-if Terraform Cloud + Sentinel/OPA CI/CD pipeline complexity Lower Higher Failure debugging ARM error messages State drift analysis License MIT BSL (OpenTofu: MPL-2.0) Neither column is free. Bicep trades portability for simplicity; Terraform trades simplicity for reach. Pitfalls I Have Seen in Real Deployments Mixing tools within one team's scope. Hybrid works at the platform boundary. It fails when the same engineer edits the same workload in both tools on alternating weeks. Draw the ownership line at a resource boundary and enforce it in the repo layout. Assuming Bicep can import anything. Bicep has no native state-import equivalent to Terraform's import blocks. It offers the existing keyword for read-only references, plus decompile and VS Code Insert Resource to generate Bicep code from existing resources — but adopting those resources under Bicep management still requires manual verification. If you have hundreds of unmanaged resources, prototype the import path before committing to a tool. Treating OpenTofu as untested. OpenTofu is production-ready and compatible with standard Terraform configurations, but enterprise support options differ. If you need a vendor contract, evaluate Terraform Cloud, Spacelift, or env0 alongside the fork. Ignoring state security in Terraform. The state file contains secrets in plaintext. If you choose Terraform, enable state encryption (native in OpenTofu; via backend configuration in Terraform) and lock the storage account down with private endpoints. Chasing day-zero without needing it. If your team does not adopt new Azure services within weeks of release, Bicep's day-zero advantage is theoretical — and not worth a migration by itself. Conclusion and Key Takeaways For Azure-first enterprises — the dominant profile in the Malaysian and Singapore markets I work in — the 2026 verdict is straightforward: default to Bicep unless you have a concrete multi-cloud or ecosystem reason not to, and use hybrid only with a strict ownership boundary. Bicep wins for Azure-only teams — no state management, day-zero Azure support, and native ARM/RBAC integration make it the lower-operational-cost choice. Terraform wins for multi-cloud — its provider ecosystem is unmatched, and existing Terraform investment should be kept, not migrated. OpenTofu is the credible open-source alternative — drop-in compatible, with exclusive features like native state encryption; evaluate it if BSL licensing conflicts with your procurement policy. Hybrid is the pragmatic reality for large estates — Bicep for the Azure platform layer, Terraform for workloads and cross-boundary resources, with exactly one tool owning each resource. The decision is about your operating model, not the syntax — state management, hiring, governance, and license posture outlive any individual template. Pick the tool your cloud strategy actually requires, standardize on it per team, and document the boundary. The tooling debate is settled; the discipline of sticking to your decision is the part that actually determines whether your IaC practice succeeds. --- ## Microsoft Agent Framework 1.0 GA — The AutoGen Successor Goes Production-Grade URL: https://wenfeng.my/microsoft-agent-framework-1-0-ga-the-autogen-successor-goes-production-grade-2/ Author: admin Published: 2026-06-23 23:34:26 Modified: 2026-06-23 23:34:26 ⚡ TL;DRMicrosoft Agent Framework (MAF) hit 1.0 general availability at Build 2026 on April 2, 2026. This is not another preview announcement — MAF unifies AutoGen and Semantic Kernel into a single cross-platform SDK for .NET and Python, with production-grad...What MAF 1.0 Actually UnifiesThe Four Key Features That MatterMAF vs. The CompetitionMigration from AutoGenGetting StartedMicrosoft Agent Framework 1.0 GA — The AutoGen Successor Goes Production-GradeMicrosoft Agent Framework (MAF) hit 1.0 general availability at Build 2026 on April 2, 2026. This is not another preview announcement — MAF unifies AutoGen and Semantic Kernel into a single cross-platform SDK for .NET and Python, with production-grade features that address the gaps most agent frameworks ignore.For Malaysian enterprises already invested in the Microsoft ecosystem, MAF 1.0 changes the build-vs-buy calculus for AI agent systems. Here is what matters architecturally, what the key features actually deliver, and how MAF compares to alternatives in 2026.What MAF 1.0 Actually UnifiesAutoGen was Microsoft's research-oriented multi-agent framework — powerful but experimental. Semantic Kernel was the enterprise SDK for integrating LLMs into .NET applications — production-ready but single-agent focused. MAF merges both into a coherent framework: Capability AutoGen (before) Semantic Kernel (before) MAF 1.0 Multi-agent orchestration ✅ ❌ ✅ .NET support ❌ (Python only) ✅ ✅ Python support ✅ ✅ ✅ Production tooling ❌ ✅ ✅ Enterprise governance ❌ Partial ✅ Hosted agent deployment ❌ ❌ ✅ (Foundry) The Four Key Features That Matter1. Agent HarnessThe Agent Harness is MAF's runtime environment for agents. It provides built-in capabilities that every production agent needs but most frameworks leave to the developer:Context compaction. Automatically summarizes long conversations to stay within context windows. Instead of truncating or failing when context grows, the harness compresses older exchanges into summaries while preserving critical information.File memory. Agents can read and write files as persistent memory — no need for external vector databases for simple memory patterns.TodoProvider. Built-in task tracking for agents that need to manage multi-step workflows. The agent can create, update, and complete tasks within the harness.BackgroundAgentsProvider. Enables agents to spawn background subtasks that run independently — useful for long-running operations that should not block the main agent loop.Why this matters: Most agent frameworks give you the LLM call and leave everything else to you. The Harness provides the operational infrastructure that makes agents production-ready.2. CodeActCodeAct is a pattern where a single model call generates executable code that performs the desired action — instead of the traditional tool-call pattern where the model emits a tool name and parameters, the runtime executes the tool, and the model processes the result.Traditional tool calling:Model → {"tool": "search", "query": "Azure pricing"} Runtime → executes search Model ← search results Model → processes results Total: 3 LLM calls CodeAct:Model → code that calls search API, processes results, returns answer Runtime → executes code Total: 1 LLM call Performance impact: Microsoft reports 52% latency reduction and 64% token reduction for multi-step tasks. Note: CodeAct is currently available in the agent-framework-hyperlight package (alpha), which runs generated code in Hyperlight micro-VMs for sandboxed execution. The model generates a single code block that chains multiple operations, eliminating the round-trip overhead of sequential tool calls.When CodeAct works best: - Tasks with deterministic tool sequences (search → filter → summarize) - High-volume execution where latency matters - Tasks where the model can reliably generate correct code for tool interactionsWhen CodeAct is risky: - Tasks requiring dynamic decision-making at each step - Environments where generated code execution needs sandboxing - Scenarios where the model might generate incorrect API calls3. Foundry Hosted AgentsMAF agents can be deployed as Foundry Hosted Agents — Microsoft's managed agent runtime:Scale-to-zero. Agents only consume resources when processing requests. No idle VM costs.VM-isolated sessions. Each agent session runs in its own VM, providing strong isolation for agents that execute code or access sensitive resources.Managed lifecycle. Foundry handles scaling, patching, and session state. You define the agent; Foundry runs it.Integration with Azure AI Foundry. Hosted agents can access the full Foundry model catalog, evaluation tools, and governance features.Why this matters for Malaysian enterprises: If you are already on Azure, hosted agents eliminate the infrastructure management overhead that makes self-managed agent deployments operationally expensive.4. Multi-Agent Orchestration PatternsMAF provides built-in orchestration patterns for multi-agent workflows:Handoff pattern. Route work between specialist agents — a triage agent delegates to billing or technical support agents based on the request. The HandoffBuilder in agent_framework.orchestrations makes this straightforward.Sequential and concurrent workflows. Chain agents in sequence for ordered processing, or run them in parallel for fan-out tasks with checkpointing and streaming support.Group collaboration. Multiple agents collaborate on a shared task, with built-in human-in-the-loop approval flows for sensitive operations.Why this matters for Malaysian enterprises: These patterns cover the common multi-agent architectures — from simple router-to-specialist flows to complex orchestrated pipelines — without requiring custom orchestration code.MAF vs. The CompetitionMAF vs. LangGraph Factor MAF 1.0 LangGraph Language support .NET + Python Python only Production tooling Built-in (Harness) DIY or LangSmith Hosted deployment Azure Foundry Self-managed or LangGraph Cloud State management Agent Harness Checkpoint-based Enterprise governance Azure-native Third-party Community Microsoft-backed Open-source community Learning curve Moderate (Microsoft patterns) Low (Python-native) When to choose MAF: .NET teams, Azure-native shops, enterprises needing managed hosting and governance.When to choose LangGraph: Python-first teams, organizations not on Azure, teams that want maximum open-source flexibility.MAF vs. CrewAI Factor MAF 1.0 CrewAI Agent definition Code-first (.NET/Python) Python decorator-based Multi-agent patterns Supervisor, swarm, pipeline Role-based crew Enterprise readiness Production-grade Improving but early Model flexibility Azure OpenAI + BYOM OpenAI, Anthropic, others Deployment Azure Foundry Self-managed or CrewAI Enterprise When to choose MAF: Enterprise deployments requiring governance, .NET integration, or Azure-native operations.When to choose CrewAI: Rapid prototyping, Python-centric teams, scenarios where the role-based crew pattern maps naturally to the problem.MAF vs. Custom OrchestrationMany teams (including mine) have built custom multi-agent orchestration using direct API calls. The tradeoff:MAF advantages over custom: - Built-in context management, memory, and error handling - Hosted deployment eliminates infrastructure management - Azure governance integration (Entra ID, RBAC, monitoring) - Microsoft support and SLACustom advantages over MAF: - Full control over every aspect of the agent loop - No framework dependency or version upgrade risk - Can use any LLM provider without MAF's model abstractions - Simpler debugging when you own the entire stackPractical recommendation: If you are starting a new multi-agent project on Azure, evaluate MAF first. If you have a working custom system, the migration cost may not justify the benefit unless you need Foundry hosted deployment or Azure governance integration.Migration from AutoGenFor teams already using AutoGen, MAF 1.0 provides migration guidance:Identify agent patterns. Map your AutoGen agents to MAF equivalents (single agent, multi-agent, supervisor).Adopt the Harness. Replace custom context management, memory, and error handling with Agent Harness built-ins.Evaluate CodeAct. Identify agent chains where CodeAct could reduce latency and cost.Update deployment. Move from self-managed AutoGen to MAF with optional Foundry hosting.Migration effort estimate: For a typical AutoGen deployment with 3-5 agents, expect 2-4 weeks of migration work including testing.Getting Started# Install MAF SDK pip install agent-framework # Or for .NET dotnet add package Microsoft.Agents.AI First agent example (Python):from agent_framework import Agent, tool # Define a tool @tool def search_web(query: str) -> dict: """Search the web for information.""" return {"results": f"Results for: {query}"} # Create a harness agent with built-in context, memory, and task management agent = create_harness_agent( client=client, max_context_window_tokens=128000, name="ResearchAgent", instructions="You research topics and provide structured summaries.", tools=[search_web], ) result = await agent.run( "Research Azure landing zone best practices for Malaysian enterprises" ) Key TakeawaysMAF 1.0 is production-ready — it is not a research preview. The Agent Harness provides the operational infrastructure (context management, memory, error handling) that most frameworks leave to developers.CodeAct delivers real performance gains — 52% latency reduction and 64% token reduction for multi-step tasks by eliminating sequential tool-call round trips.Foundry Hosted Agents eliminate infrastructure overhead — scale-to-zero, VM isolation, and managed lifecycle make agent deployment operationally simple on Azure.MAF is the strongest choice for .NET and Azure-native teams — Python-first teams not on Azure should evaluate LangGraph or CrewAI as alternatives.Migration from AutoGen is straightforward — map agent patterns to MAF equivalents, adopt the Harness, and evaluate CodeAct for performance-critical chains.The agent framework landscape in 2026 is consolidating around a few production-grade options. MAF 1.0 is Microsoft's entry, and for enterprises already in the Azure ecosystem, it is the most complete choice for building, deploying, and governing multi-agent systems at scale. --- ## Southeast Asia 3: Planning Around an Announced Azure Region URL: https://wenfeng.my/southeast-asia-3-cloud-region-what-microsofts-johor-bahru-datacenter-means-for-azure-architects/ Author: admin Published: 2026-06-22 23:32:38 Modified: 2026-10-21 15:44:17 Technical sources checked on 21 October 2026. Microsoft’s announcement of Southeast Asia 3 in Johor Bahru gives architects another development to follow. It does not, by itself, provide a region that can be placed in a production recovery diagram. In its 4 November 2025 announcement, Microsoft described Southeast Asia 3 as upcoming. That source does not establish a 2026 preview deadline. A launch assumption should therefore stay out of delivery commitments unless a later, applicable Microsoft source confirms it. Separate the announcement from deployment evidence The Azure region list checked for this article lists Malaysia West in Kuala Lumpur, with availability zones and no paired region. It does not establish Southeast Asia 3 as an available destination for the proposed architecture. The same documentation cautions that regional availability-zone support does not mean every service supports zones there. A second location is not automatically a region pair Microsoft’s region-pair guidance distinguishes paired and nonpaired regions. Some services use a designated pair for replication; other services support multiregion designs without a pair. Being in a paired region does not automatically provide application failover or disaster recovery. Consequently, this article does not present Malaysia West and Southeast Asia 3 as an available domestic recovery pair. Nor should “nonpaired” be read as a claim that every cross-region recovery option is impossible. The answer depends on the services and capabilities actually available. Correction: This revision removes unsupported launch timing, pricing, latency and compliance claims, and the earlier presentation of the two Malaysian locations as an available disaster-recovery pair. --- ## FinOps for AI Agents: The Cost Optimization Playbook That Separates Viable From Vanity Projects URL: https://wenfeng.my/finops-for-ai-agents-cost-optimization-playbook-2026/ Author: admin Published: 2026-06-19 23:33:27 Modified: 2026-10-04 15:31:15 ⚡ TL;DRAgentic AI is projected to grow from $6.76B to $46B by 2030. But here's the uncomfortable truth most vendors won't tell you: the majority of agent pilots fail on cost, not capability. The agent works beautifully in demo. Then the token bill arrives.The Hidden Cost Architecture of AI AgentsThe Five Cost Optimization LeversCost Modeling for Common Agent PatternsThe Malaysian Enterprise ContextImplementation RoadmapFinOps for AI Agents: The Cost Optimization Playbook That Separates Viable from BankruptAgentic AI is projected to grow from $6.76B to $46B by 2030. But here's the uncomfortable truth most vendors won't tell you: the majority of agent pilots fail on cost, not capability. The agent works beautifully in demo. Then the token bill arrives.If you're building or deploying AI agents in production — whether it's a customer service bot, an internal research assistant, or a multi-agent content pipeline — this article is about the cost discipline that separates a sustainable agent from an expensive experiment.The Hidden Cost Architecture of AI AgentsMost teams budget for AI agent costs the wrong way. They look at the per-token price of the underlying LLM and multiply by expected queries. This misses 70% of the actual cost.The real cost stack of an AI agent:Visible Costs (what you budgeted): ├── LLM API calls (tokens in/out) └── Infrastructure (VMs, containers, storage) Hidden Costs (what actually burns your budget): ├── Retry loops (failed tool calls, malformed output) ├── Context bloat (growing conversation history) ├── Tool orchestration (multi-step agent workflows) ├── Memory and state management (vector DB, session store) ├── Monitoring and logging (traces, evaluations) └── Human review (human-in-the-loop oversight) Real Cost Breakdown from ProductionFrom running a multi-agent content pipeline (Hermes + OpenClaw) daily for 6 months: Component Monthly Cost (USD) % of Total LLM API calls (primary model) $85 34% LLM API calls (vision/reasoning auxiliary) $45 18% Web search/extraction API $40 16% Ghost CMS API calls $5 2% Buffer social media API $0 0% Azure VM compute $35 14% Storage and backups $8 3% Monitoring and logging $15 6% Total ~$250 100% Key insight: LLM tokens are only 52% of the total cost. Tool APIs, compute, and monitoring make up the rest. Optimizing only the LLM costs leaves half the budget untouched.The Five Cost Optimization LeversLever 1: Model Tiering (3-5x Savings)Not every agent task needs the best model. The biggest cost lever is routing different tasks to appropriately-priced models:# Model tier routing for an agent pipeline MODEL_TIERS = { "research": "gemini-3.5-flash", # $1.50/$9 — high volume, simple reasoning "writing": "mimo-v2.5", # $0.50/$2 — medium quality, high volume "review": "claude-sonnet-4", # $3/$15 — quality-critical review "reasoning": "claude-opus-4.8", # $5/$25 — complex multi-step reasoning "vision": "gpt-4o", # $2.50/$10 — image analysis } def select_model(task_type: str, complexity: str = "standard") -> str: """Route to the cheapest model that meets quality bar.""" if complexity == "high": return MODEL_TIERS.get("reasoning", "gpt-4.1") return MODEL_TIERS.get(task_type, "gemini-3.5-flash") Impact: Switching from "use Claude Opus for everything" to tiered routing typically saves 60-75% on token costs with less than 5% quality degradation on non-critical tasks.Lever 2: Prompt Caching (60-85% on Repeated Prompts)Enterprise agents repeat the same system prompt thousands of times. Prompt caching eliminates the cost of re-processing these:Actual savings from production: - System prompt (2K tokens, 10K calls/day): $15/day → $3/day - Knowledge base context (10K tokens, 80% cache hit): $50/day → $10/day - Monthly savings: ~$1,300/month on a mid-volume workloadImplementation:# Anthropic prompt caching response = client.messages.create( model="claude-sonnet-4-20250514", max_tokens=1024, system=[ { "type": "text", "text": LONG_SYSTEM_PROMPT, # This gets cached automatically "cache_control": {"type": "ephemeral"} } ], messages=[{"role": "user", "content": user_query}] ) # cache_creation_input_tokens: billed at 1.25x on first call # cache_read_input_tokens: billed at 0.1x on subsequent calls Lever 3: Retry Budget and Circuit BreakersAgent tool calls fail. APIs timeout. Models produce malformed output. Without a retry budget, a single failing tool can burn through your entire daily token allocation.import time from functools import wraps class RetryBudget: def __init__(self, max_retries=3, max_cost_per_task=0.05): self.max_retries = max_retries self.max_cost_per_task = max_cost_per_task self.total_cost = 0 self.retry_count = 0 def can_retry(self, estimated_cost: float) -> bool: if self.retry_count >= self.max_retries: return False if self.total_cost + estimated_cost > self.max_cost_per_task: return False return True def record(self, cost: float): self.total_cost += cost self.retry_count += 1 def with_retry_budget(budget: RetryBudget): def decorator(func): @wraps(func) def wrapper(*args, **kwargs): last_error = None for attempt in range(budget.max_retries): try: return func(*args, **kwargs) except Exception as e: last_error = e if not budget.can_retry(0.01): # estimate retry cost break time.sleep(2 ** attempt) # exponential backoff budget.record(0.01) raise last_error return wrapper return decorator Impact: Prevents runaway costs from retry loops. A typical agent without a retry budget can spend 3-5x more during API instability.Lever 4: Context Window ManagementAgent conversations grow. A 50-turn conversation that started with 2K tokens can balloon to 50K tokens. Most of that is stale context that the model re-processes every turn.Strategies: 1. Sliding window — keep only the last N turns in context 2. Summarization — periodically summarize old context into a compressed form 3. Selective inclusion — only include relevant past context, not all of itdef manage_context(messages: list, max_tokens: int = 8000) -> list: """Keep context under budget by summarizing old messages.""" total_tokens = estimate_tokens(messages) if total_tokens <= max_tokens: return messages # Split: recent messages + older messages to summarize split_point = len(messages) // 2 old_messages = messages[:split_point] recent_messages = messages[split_point:] # Summarize old messages into a single context block summary = summarize_messages(old_messages) # call a cheap model return [ {"role": "system", "content": f"Previous context: {summary}"}, *recent_messages ] Impact: Reduces token consumption by 40-60% for long-running agent conversations without meaningful quality loss.Lever 5: Tool Call OptimizationEach tool call in an agent workflow is an LLM turn. Reducing unnecessary tool calls is a direct cost reduction:Batch tool calls — combine multiple lookups into one turnCache tool results — don't re-fetch data that hasn't changedPre-validate inputs — check tool parameters before calling (avoid retry from bad input)Use structured output — enforce JSON schema at the model level to avoid re-promptingCost Modeling for Common Agent PatternsPattern 1: Simple Q&A BotVolume: 10K queries/day Model: Gemini 3.5 Flash ($1.50/$9 per 1M tokens) Avg tokens: 500 in, 300 out Monthly cost: ~$75 With caching: ~$25 Pattern 2: Research Agent (Web Search + Summarize)Volume: 500 queries/day Model: Claude Sonnet 4 ($3/$15 per 1M tokens) Avg tokens: 2K in (with search results), 1K out + Web search API: $0.01/query Monthly cost: ~$450 With model tiering (Flash for search, Sonnet for synthesis): ~$200 Pattern 3: Multi-Agent PipelineVolume: 10 complete pipeline runs/day 6 agents per run, avg 3 turns each Models: Mixed (Flash + Sonnet + Opus) Monthly cost: ~$250-400 With all optimizations: ~$150-200 Pattern 4: Enterprise Agent PlatformVolume: 50K queries/day across 10 agent types Models: Auto-routed across 4 tiers Monthly cost: ~$8,000-15,000 With all optimizations: ~$3,000-5,000 The Malaysian Enterprise ContextFor Malaysian enterprises deploying AI agents, there are additional cost factors:1. Cross-region data transfer costs: If your agent processes data that resides in Malaysia but the model API is in Singapore or US East, cross-region data transfer adds $0.08-0.12/GB. For agents processing large documents (50-100MB PDFs), this adds up.Mitigation: Use Azure OpenAI in Malaysia West where available, or cache processed data locally to avoid re-transferring.2. Currency exposure: All major LLM APIs bill in USD. At current exchange rates (1 USD ≈ 4.5 MYR), a 10% USD fluctuation changes your AI budget by 10%.Mitigation: Negotiate reserved capacity in MYR where available (Azure EA), or build a 15% currency buffer into your budget model.3. Talent cost for optimization: Building the cost optimization infrastructure (model routing, retry budgets, monitoring) requires senior engineering time. In Malaysia, a senior platform engineer costs RM15,000-25,000/month.ROI calculation: If your agent platform costs RM15,000/month before optimization and a senior engineer can reduce that by 50% through model tiering and caching, the payback period is 1 month.Implementation RoadmapWeek 1: MeasureAdd token counting to every LLM callLog model, tokens, latency, and cost per requestBuild a cost dashboard (even a spreadsheet works)Week 2: TierIdentify which tasks use premium models unnecessarilyImplement model routing for non-critical tasksExpected savings: 30-50%Week 3: CacheEnable prompt caching for system prompts and repeated contextImplement context window managementExpected savings: additional 20-30%Week 4: GuardAdd retry budgets and circuit breakersSet per-task cost limitsImplement alerting for cost anomaliesKey TakeawaysLLM tokens are only 50-60% of total agent cost — tool APIs, compute, monitoring, and retries make up the rest. Optimize the full stack, not just the model calls.Model tiering is the single biggest lever — routing 70% of queries to cheaper models saves 60-75% on token costs with minimal quality impact.Prompt caching delivers immediate savings — 60-85% reduction on repeated system prompts, available on all major providers today.Retry budgets prevent runaway costs — a single failing tool call can burn your daily budget without circuit breakers.The payback period for optimization is measured in weeks — a senior engineer can implement the full optimization stack in 4 weeks, with ROI realized in the first month. --- ## Azure API Management as AI Gateway: Managing Access, Throttling, and Observability for LLM Endpoints URL: https://wenfeng.my/azure-api-management-as-ai-gateway-managing-access-throttling-and-observability-for-llm-endpoints/ Author: admin Published: 2026-06-07 23:34:12 Modified: 2026-10-04 15:30:10 ⚡ TL;DRThe rapid adoption of Large Language Models (LLMs) across Malaysian enterprises has introduced a new category of infrastructure challenge: how do you secure, govern, and observe API traffic to OpenAI, Azure OpenAI Service, and other LLM providers at ...Introduction1. Rate Limiting and Token-Based Throttling2. Intelligent Backend Routing and Model Selection3. Observability: Logging, Token Auditing, and Cost Attribution4. Usage Tracking and ChargebackIntroductionThe rapid adoption of Large Language Models (LLMs) across Malaysian enterprises has introduced a new category of infrastructure challenge: how do you secure, govern, and observe API traffic to OpenAI, Azure OpenAI Service, and other LLM providers at scale? Traditional API gateways were built for RESTful microservices — request-response patterns with predictable payloads, stable latency profiles, and well-understood rate limits. LLM endpoints break all three assumptions. Token consumption varies wildly between requests, latency can stretch from milliseconds to tens of seconds, and cost is no longer a function of request count but of prompt and completion length.Azure API Management (APIM) has emerged as the natural control plane for this challenge. Already deployed in many Malaysian government-linked companies (GLCs) and financial institutions for legacy API governance, APIM can be extended — using its policy engine, named values, logging integrations, and built-in throttling mechanisms — to serve as a full-fledged AI gateway. This article walks through four practical policy patterns that platform engineers in Malaysia can deploy today to bring order to their LLM traffic.This is not a conceptual overview. Every snippet below is production-tested against Azure OpenAI endpoints with GPT-4o and o3-mini models, deployed in APIM Premium and Standard tiers across Malaysian regions (Southeast Asia). The code is available as reference implementations on the Cloud Catalyst GitHub repository.1. Rate Limiting and Token-Based ThrottlingThe most immediate pain point for any organisation exposing LLM endpoints internally is cost control. Azure OpenAI charges by token — so throttling by request count alone is insufficient. A single RAG query with a 32K context window can cost fifty times more than a simple chat completion. Malaysian enterprises, particularly those in banking and healthcare where budgets are scrutinised by Bank Negara and MOH compliance teams, need token-aware throttling that maps directly to cost.Policy: Token-Based Rate Limit by SubscriptionThe following inbound policy reads the model and max_tokens parameters from the request body, estimates total token consumption, and enforces a per-subscription token quota using APIM's built-in rate-limit-by-key counter.<policies> <inbound> <base /> <!-- Parse request body for token estimation --> <set-variable name="requestBody" value="@(context.Request.Body.As<JObject>(preserveContent: true))" /> <set-variable name="model" value="@((string)((JObject)context.Variables["requestBody"])["model"] ?? "gpt-4o")" /> <set-variable name="maxTokens" value="@((int?)((JObject)context.Variables["requestBody"])["max_tokens"] ?? 4096)" /> <set-variable name="promptTokens" value="@(EstimatePromptTokens((JObject)context.Variables["requestBody"]))" /> <set-variable name="estimatedTotalTokens" value="@((int)context.Variables["maxTokens"] + (int)context.Variables["promptTokens"])" /> <!-- Apply token-based rate limit: max 100,000 tokens per subscription per hour --> <rate-limit-by-key calls="@((int)context.Variables["estimatedTotalTokens"])" renewal-period="3600" counter-key="@(context.Subscription.Id ?? "anonymous")" increment-condition="@(context.Response.StatusCode == 200)" remaining-calls-header-name="x-tokens-remaining" /> <set-header name="x-estimated-tokens" exists-action="override"> <value>@((int)context.Variables["estimatedTotalTokens"])</value> </set-header> </inbound> <outbound> <base /> <!-- Capture actual token usage from response for accurate billing --> <set-variable name="responseBody" value="@(context.Response.Body.As<JObject>(preserveContent: true))" /> <set-variable name="usage" value="@((JObject)context.Variables["responseBody"])["usage"]" /> <set-variable name="totalTokens" value="@((int)((JObject)context.Variables["usage"])["total_tokens"])" /> <set-header name="x-total-tokens-actual" exists-action="override"> <value>@((int)context.Variables["totalTokens"])</value> </set-header> </outbound> </policies> How it works. The policy estimates prompt tokens from the request body using a helper function (available via C# policy expression) that counts message content and applies a model-specific token-per-character ratio. The rate-limit-by-key policy then consumes that many tokens from the subscription's hourly quota. On the outbound side, the actual total_tokens from the Azure OpenAI response is captured and surfaced via a response header, enabling downstream billing reconciliation.Malaysia context. For a bank processing customer-service LLM queries, this policy allows the API team to set a monthly token budget per business unit. When a unit exhausts its quota, the policy returns HTTP 429 with a Retry-After header — giving the business owner a clear signal before costs spiral. This is critical in the Malaysian regulatory landscape where unexpected cloud costs require board-level justifications.2. Intelligent Backend Routing and Model SelectionNot all LLM requests are equal. A simple summarisation task does not need GPT-4o; a complex legal document analysis probably should not use a distilled model. Malaysian enterprises operating across multiple Azure OpenAI deployments (GPT-4o, GPT-4o-mini, o3-mini) need routing logic that maps request characteristics to the most cost-effective backend endpoint without requiring application-level changes.Policy: Content-Aware Model RoutingThis inbound policy inspects the incoming request, classifies the complexity based on prompt length and content keywords, and rewrites the backend URL to target the appropriate Azure OpenAI deployment.<policies> <inbound> <base /> <set-variable name="requestBody" value="@(context.Request.Body.As<JObject>(preserveContent: true))" /> <set-variable name="messages" value="@((JArray)((JObject)context.Variables["requestBody"])["messages"])" /> <set-variable name="promptLength" value="@(string.Join("", ((JArray)context.Variables["messages"]).Select(m => (string)m["content"] ?? "")).Length)" /> <!-- Classification logic: route to appropriate model --> <choose> <!-- Simple queries (chat, FAQ) -> GPT-4o-mini (lowest cost) --> <when condition="@((int)context.Variables["promptLength"] < 500 && !((string)((JObject)context.Variables["requestBody"])["messages"]?.Last?["content"] ?? "") .Contains("analyse", StringComparison.OrdinalIgnoreCase) && !((string)((JObject)context.Variables["requestBody"])["messages"]?.Last?["content"] ?? "") .Contains("contract", StringComparison.OrdinalIgnoreCase))"> <set-backend-service base-url="https://my-openai-gpt4o-mini.openai.azure.com" /> <set-variable name="deployedModel" value="gpt-4o-mini" /> </when> <!-- Complex analysis -> GPT-4o --> <when condition="@((int)context.Variables["promptLength"] >= 500 || ((string)((JObject)context.Variables["requestBody"])["messages"]?.Last?["content"] ?? "") .Contains("analyse", StringComparison.OrdinalIgnoreCase) || ((string)((JObject)context.Variables["requestBody"])["messages"]?.Last?["content"] ?? "") .Contains("contract", StringComparison.OrdinalIgnoreCase))"> <set-backend-service base-url="https://my-openai-gpt4o.openai.azure.com" /> <set-variable name="deployedModel" value="gpt-4o" /> </when> <!-- Default fallback --> <otherwise> <set-backend-service base-url="https://my-openai-gpt4o-mini.openai.azure.com" /> <set-variable name="deployedModel" value="gpt-4o-mini" /> </otherwise> </choose> <!-- Override model in request body to match backend --> <set-variable name="body" value="@(((JObject)context.Variables["requestBody"]).ToString())" /> <return-response> <set-status code="200" reason="OK" /> <set-header name="Content-Type" exists-action="override"> <value>application/json</value> </set-header> <set-body>@(((JObject)context.Variables["requestBody"]).ToString())</set-body> </return-response> </inbound> <backend> <base /> </backend> <outbound> <base /> <set-header name="x-routed-model" exists-action="override"> <value>@((string)context.Variables["deployedModel"])</value> </set-header> </outbound> </policies> Production considerations. In a real deployment, avoid hardcoding credentials or URLs in the policy. Store the backend base URLs in APIM Named Values and reference them via {{NamedValue}} syntax. This allows the API operations team to rotate endpoints without republishing policies — essential for Malaysian enterprises that must comply with PCI-DSS or ISO 27001 change-management processes.Regional nuance. Malaysia's MyDigital initiative encourages public sector agencies to adopt cloud-first strategies, but data sovereignty requirements often mandate that certain workloads remain within Southeast Asia Azure regions. The routing policy can be extended to check a request header (e.g., x-data-classification: SOVEREIGN) and direct traffic to a backend in Malaysia Southeast Asia rather than East US, ensuring compliance with the Personal Data Protection Act (PDPA) 2010.3. Observability: Logging, Token Auditing, and Cost AttributionWithout observability, an AI gateway is blind. Malaysian enterprises deploying LLM endpoints need to answer three questions for every request: Who called it? How many tokens did it consume? How much did it cost? APIM's integration with Azure Monitor, Application Insights, and Event Hubs provides the pipeline, but the policy layer must enrich the telemetry with LLM-specific dimensions.Policy: Structured Logging with Token Cost AttributionThis outbound policy captures token usage and cost metadata into Application Insights custom events, enabling per-subscription and per-model cost dashboards in Azure Monitor Workbooks.<policies> <outbound> <base /> <!-- Parse response for actual token usage --> <set-variable name="responseBody" value="@(context.Response.Body.As<JObject>(preserveContent: true))" /> <set-variable name="usage" value="@((JObject)context.Variables["responseBody"])["usage"]" /> <set-variable name="promptTokens" value="@((int)((JObject)context.Variables["usage"])["prompt_tokens"])" /> <set-variable name="completionTokens" value="@((int)((JObject)context.Variables["usage"])["completion_tokens"])" /> <set-variable name="totalTokens" value="@((int)((JObject)context.Variables["usage"])["total_tokens"])" /> <!-- Calculate cost per model pricing tier (example: GPT-4o rates) --> <set-variable name="promptCost" value="@(((int)context.Variables["promptTokens"]) * 0.00000275)" /> <set-variable name="completionCost" value="@(((int)context.Variables["completionTokens"]) * 0.000011)" /> <set-variable name="totalCost" value="@((double)context.Variables["promptCost"] + (double)context.Variables["completionCost"])" /> <!-- Emit custom telemetry to Application Insights --> <set-variable name="telemetry" value="@{ return new JObject( new JProperty("SubscriptionId", context.Subscription.Id), new JProperty("SubscriptionName", context.Subscription.Name ?? ""), new JProperty("ProductName", context.Product.Name ?? ""), new JProperty("UserId", context.User?.Email ?? "anonymous"), new JProperty("Model", context.Variables.ContainsKey("deployedModel") ? (string)context.Variables["deployedModel"] : "unknown"), new JProperty("PromptTokens", (int)context.Variables["promptTokens"]), new JProperty("CompletionTokens", (int)context.Variables["completionTokens"]), new JProperty("TotalTokens", (int)context.Variables["totalTokens"]), new JProperty("PromptCost", (double)context.Variables["promptCost"]), new JProperty("CompletionCost", (double)context.Variables["completionCost"]), new JProperty("TotalCost", (double)context.Variables["totalCost"]), new JProperty("RequestId", context.RequestId.ToString()), new JProperty("Timestamp", DateTime.UtcNow.ToString("o")), new JProperty("BackendUrl", context.Request.Url.ToString()), new JProperty("StatusCode", context.Response.StatusCode), new JProperty("ResponseLatencyMs", context.Elapsed.TotalMilliseconds) ); }" /> <emit-metric name="LLM Token Usage" value="@((double)context.Variables["totalTokens"])" namespace="AzureAPIM-AIGateway" dimensions="@(new JObject( new JProperty("Subscription", context.Subscription.Id ?? "none"), new JProperty("Model", context.Variables.ContainsKey("deployedModel") ? (string)context.Variables["deployedModel"] : "unknown"), new JProperty("StatusCode", context.Response.StatusCode.ToString()) ))" /> <trace source="ai-gateway" severity="information"> <message>@{ return $"LLM Call | Sub: {context.Subscription.Id} | " + $"Model: {(context.Variables.ContainsKey("deployedModel") ? (string)context.Variables["deployedModel"] : "unknown")} | " + $"Tokens: {(int)context.Variables["totalTokens"]} | " + $"Cost: ${(double)context.Variables["totalCost"]:F6} | " + $"Latency: {context.Elapsed.TotalMilliseconds:F0}ms"; }</message> </trace> <!-- Add cost headers for consumer visibility --> <set-header name="x-token-cost-usd" exists-action="override"> <value>@(((double)context.Variables["totalCost"]).ToString("F8"))</value> </set-header> </outbound> </policies> Dashboarding in practice. With this telemetry flowing into Application Insights, a Malaysian enterprise can build an Azure Monitor Workbook that shows:Total token consumption trended by hour, per subscriptionCost breakdown by model (GPT-4o vs. GPT-4o-mini vs. o3-mini)Top-10 consumers by departmentLatency P50/P95/P99 per model429 (rate-limit) hit rates per subscriptionThis level of granularity is indispensable for the monthly cloud cost review meetings that are standard practice in Malaysian GLCs. Without it, LLM costs appear as a single opaque line item in the Azure bill.4. Usage Tracking and ChargebackFor enterprises operating an internal API marketplace — a pattern increasingly common among Malaysia's larger conglomerates and government agencies — the AI gateway must support chargeback. Each business unit consumes LLM tokens through the gateway, and the central IT team needs to allocate costs back to those units.Policy: Subscription-Based Consumption Tracking with Event Hubs ExportThis policy extends the logging pattern by forwarding enriched consumption records to Azure Event Hubs, from where they can be ingested into an internal billing system or SAP.<policies> <outbound> <base /> <set-variable name="responseBody" value="@(context.Response.Body.As<JObject>(preserveContent: true))" /> <set-variable name="usage" value="@((JObject)context.Variables["responseBody"])["usage"]" /> <set-variable name="totalTokens" value="@((int)((JObject)context.Variables["usage"])["total_tokens"])" /> <!-- Build consumption record for chargeback --> <set-variable name="chargebackRecord" value="@{ return new JObject( new JProperty("schemaVersion", "1.0"), new JProperty("eventType", "LLMConsumption"), new JProperty("subscriptionId", context.Subscription.Id ?? "none"), new JProperty("subscriptionName", context.Subscription.Name ?? "none"), new JProperty("productName", context.Product.Name ?? "none"), new JProperty("userId", context.User?.Email ?? "system"), new JProperty("modelDeployed", context.Variables.ContainsKey("deployedModel") ? (string)context.Variables["deployedModel"] : "unknown"), new JProperty("totalTokens", (int)context.Variables["totalTokens"]), new JProperty("promptTokens", (int)((JObject)context.Variables["usage"])["prompt_tokens"]), new JProperty("completionTokens", (int)((JObject)context.Variables["usage"])["completion_tokens"]), new JProperty("costUsd", (double)context.Variables.ContainsKey("totalCost") ? (double)context.Variables["totalCost"] : 0.0), new JProperty("region", "southeast-asia"), new JProperty("requestId", context.RequestId.ToString()), new JProperty("callerIp", context.Request.IpAddress), new JProperty("timestamp", DateTime.UtcNow.ToString("o")), new JProperty("costCenter", context.Request.Headers.GetValueOrDefault("x-cost-center", "unallocated")), new JProperty("department", context.Request.Headers.GetValueOrDefault("x-department", "unallocated")) ); }" /> <!-- Forward to Event Hubs for downstream billing ingestion --> <send-one-way-request mode="new"> <set-url>@{ return context.Variables.ContainsKey("eventHubUrl") ? (string)context.Variables["eventHubUrl"] : "https://ai-chargeback-ns.servicebus.windows.net/llm-consumption/messages"; }</set-url> <set-method>POST</set-method> <set-header name="Content-Type" exists-action="override"> <value>application/json</value> </set-header> <set-header name="Authorization" exists-action="override"> <value>@("SharedAccessSignature " + (string)context.Variables["eventHubSasToken"])</value> </set-header> <set-body>@(((JObject)context.Variables["chargebackRecord"]).ToString())</set-body> </send-one-way-request> </outbound> </policies> Chargeback model. The Event Hubs consumer can be an Azure Function that aggregates records hourly, computes per-subscription totals, and writes to a SQL database or exports to the enterprise ERP. For Malaysian GLCs operating on a cost-recovery model, this enables accurate showback (visibility) and chargeback (actual cross-charging) reporting.Important consideration. The send-one-way-request policy is fire-and-forget. If the Event Hubs endpoint is unavailable, the policy fails silently (configurable via continue-on-error). For critical billing data, consider using Azure Logic Apps with retry logic as an intermediary.Deployment Architecture for Malaysian EnterprisesCombining the four patterns above yields a robust AI gateway architecture: Client Apps (internal / B2B) | v Azure API Management (Premium) | | | v v v Rate Route Observability Limiting + Model + Audit (Token) Select + Chargeback | | | +-------+---------+ | v Azure OpenAI Service (GPT-4o / GPT-4o-mini / o3-mini) -- Malaysia Southeast Asia region Tier recommendation. The Premium tier is recommended for production AI gateways in Malaysian enterprises because it offers: - Multi-region deployment (active-active across Malaysia Southeast Asia and East Asia for DR) - VNet integration for private connectivity to Azure OpenAI without traversing the public internet - Rate limiting by key with counter-key scoped to subscriptions - Custom metric emission via emit-metric policy - Dedicated event hubs for log forwardingFor proof-of-concept or non-critical workloads, the Standard tier with Application Insights integration provides sufficient observability, but lacks VNet injection and multi-region failover.Operational ConsiderationsSecurity. Never store API keys or connection strings in policy files. Use APIM Named Values with Key Vault back-referencing. Azure OpenAI keys should be rotated regularly, and APIM's managed identity should be used to authenticate to Key Vault.Compliance. For financial institutions regulated by Bank Negara Malaysia, ensure that APIM diagnostic logs are retained for at least seven years. Configure Azure Monitor Log Analytics workspace with appropriate retention policies and enable audit logging on all policy changes.Cost optimisation. The routing policy described in Section 2 alone can reduce LLM costs by 40-60% for workloads with a mix of simple and complex queries. Monitor the x-routed-model header to verify that classification rules are working as intended, and adjust the prompt-length threshold based on observed latency and quality metrics.Testing. Before deploying token-based throttling to production, validate the estimation function against actual usage. A mismatch between estimated and actual tokens can lead to premature 429s or quota overruns. The outbound header x-total-tokens-actual enables this validation during a shadow-testing phase.ConclusionAzure API Management is not just a reverse proxy for REST APIs — it is a fully capable AI gateway that addresses the specific governance, cost-control, and observability challenges posed by LLM endpoints. For Malaysian enterprises navigating the intersection of cloud adoption (MyDigital), data sovereignty (PDPA), and AI enablement, APIM provides a familiar control surface backed by enterprise-grade SLAs.The four policy patterns presented here — token-based throttling, content-aware routing, structured observability, and chargeback tracking — form a foundation that can be incrementally adopted. Start with throttling to prevent cost overruns, add routing to optimise model utilisation, layer observability for visibility, and finally implement chargeback for financial accountability.Cloud Catalyst has deployed this architecture for multiple clients in Malaysia's financial services and public sectors. The consistent feedback is that APIM's policy engine, while originally designed for southbound REST traffic, maps surprisingly well to the northbound challenges of LLM governance. With the patterns above, your enterprise can move from uncontrolled LLM experimentation to a governed, observable, and cost-managed AI platform — without introducing yet another gateway into your stack.Law Wen Feng is Principal Solution Architect at Cloud Catalyst, where he leads Azure platform engineering for enterprise clients across Malaysia and Southeast Asia. He specialises in API management, cloud security architecture, and AI infrastructure design.For a workshop or proof-of-concept deployment of Azure APIM as an AI gateway for your organisation, contact Cloud Catalyst at https://cloudcatalyst.com.my. --- ## Azure AI Foundry: Control Plane for AI Models at Scale (2026) URL: https://wenfeng.my/azure-ai-foundry-as-the-unified-control-plane-deploying-evaluating-and-governing-ai-models-at-scale/ Author: admin Published: 2026-06-02 23:34:33 Modified: 2026-10-04 15:30:20 ⚡ TL;DRIf you have been hands-on with Azure AI over the past two years, you have felt the whiplash. One quarter it is Azure Cognitive Services, the next it is Azure OpenAI Service, then Azure AI Studio, and now — Microsoft Foundry (formerly Azure AI Foundry...IntroductionThe Problem: AI Sprawl Is RealThe Solution: Microsoft Foundry (formerly Azure AI Foundry) as the Unified Co...Practical Examples: Azure CLI and BicepPitfalls: What the Documentation Doesn't Tell YouIntroductionIf you have been hands-on with Azure AI over the past two years, you have felt the whiplash. One quarter it is Azure Cognitive Services, the next it is Azure OpenAI Service, then Azure AI Studio, and now — Microsoft Foundry (formerly Azure AI Foundry). It is tempting to roll your eyes at another rebrand. But this one is different.Microsoft Foundry (formerly Azure AI Foundry) is not just a renamed portal. It is Microsoft's bet on something enterprise AI has desperately needed — a single, unified control plane for the entire AI lifecycle. Not just training and inference, but deployment, evaluation, monitoring, governance, and responsible AI guardrails all in one place. For architects and engineering leads in Malaysian enterprises navigating the AI adoption curve, understanding what Microsoft Foundry (formerly Azure AI Foundry) actually is (and is not) can save months of architectural dead ends.I have spent the last several months helping clients across financial services, government-linked companies, and retail build their AI platforms on Azure. What follows comes straight from those trenches — the good, the messy, and the parts the documentation glosses over.The Problem: AI Sprawl Is RealBefore we talk about the solution, let us be honest about the pain point. Most organisations I work with in Malaysia are not starting from scratch. They already have:A handful of OpenAI GPT-4 deployments in Azure OpenAI Service for chatbot use cases.Custom models on Azure ML compute clusters for niche predictions.Cognitive Services endpoints (OCR, speech-to-text, translation) integrated into line-of-business apps.Maybe a GenAI-powered document search built on Azure AI Search with embeddings in Cosmos DB.Every one of these lives in its own silo. Each has its own deployment pipeline, monitoring dashboard, access control model, and cost centre. Your security team wants a single view of which models are internet-facing. Your compliance team wants evidence that every output passes content safety. Your finance team wants to know why the AI bill jumped 40% last month. And your data science lead wants a staging environment to A/B test a new model before production.Without a unified control plane, answering any of these requires stitching together data from Azure Monitor, Azure Policy, Azure Cost Management, and custom scripts. That works for one or two models. It crumbles at twenty — or fifty.This is the exact problem Microsoft Foundry (formerly Azure AI Foundry) was built to solve.The Solution: Microsoft Foundry (formerly Azure AI Foundry) as the Unified Control PlaneMicrosoft Foundry (formerly Azure AI Foundry) — at its core — is a platform that wraps Azure's existing AI capabilities into a cohesive management layer. Think of it as the orchestration hub that sits above your AI infrastructure, not a replacement for any individual service.What It Actually IncludesLet me break down the key capabilities because the marketing material tends to bundle things vaguely:1. The AI Hub (formerly AI Studio workspace). Your logical container. Every project, model endpoint, and evaluation run lives inside a hub. Multiple hubs per subscription (dev, staging, prod) with independent policies, network isolation, and quotas.2. Model Catalog and Deployment. From a single interface, deploy Azure OpenAI models (GPT-4o, o3-mini, embeddings), open-source models via Model-as-a-Service (Llama, Mistral, Phi-3, Cohere), custom fine-tuned models, or self-managed compute. The workflow is consistent regardless of source — one Bicep template, one CLI command, one pricing model.3. Evaluation and Safety Metrics. Built-in evaluators for groundedness, relevance, coherence, fluency, and similarity. They use GPT-4 as a judge with transparent prompt templates you can inspect and customise. Combined with Content Safety integration, you get automated guardrails for hate speech, self-harm, sexual content, and violence before any output reaches a user.4. Prompt Flow. A visual designer for prompt engineering and LLM orchestration. Chain models, add retrieval steps, integrate AI Search, and bake in safety evaluation — all from a canvas that generates deployable code. For teams with fewer dedicated ML engineers, this lowers the barrier significantly.5. Tracing and Monitoring. OpenTelemetry-based instrumentation with built-in dashboards for token usage, latency, error rates, and safety results — zero custom instrumentation code needed.The Architecture at a GlanceAt the resource level, the hierarchy looks like this:Subscription └── Azure AI Hub (Microsoft.CognitiveServices/accounts) ├── Project (ai_project) │ ├── Model Deployments (GPT-4o, Llama 3, custom models) │ ├── Prompt Flow Runs │ ├── Evaluation Resources │ └── Connections (AI Search, Storage, Content Safety) ├── Shared Compute (optional) └── Policies & Network Rules The AI Hub resource is technically a Cognitive Services account with a new kind of AIServices. This matters for anyone who has dealt with Azure's resource provider limitations — it means you get the same familiar Azure RBAC, diagnostic settings, private endpoints, and managed identity support you already know.Practical Examples: Azure CLI and BicepLet's get into the concrete. Here is how you would set up a production-grade AI Foundry environment using infrastructure-as-code.1. Deploy an AI Hub with Bicepparam hubName string = 'ai-hub-prod-myapp' param location string = 'eastus2' param sku string = 'S0' param tags object = { environment: 'production' costCenter: 'ai-platform' } resource aiHub 'Microsoft.CognitiveServices/accounts@2024-10-01' = { name: hubName location: location kind: 'AIServices' sku: { name: sku } tags: tags properties: { customSubDomainName: hubName networkAcls: { defaultAction: 'Deny' virtualNetworkRules: [] ipRules: [] } publicNetworkAccess: 'Disabled' apiProperties: { statisticsEnabled: false } } } output hubId string = aiHub.id output hubEndpoint string = aiHub.properties.endpoint Note a few deliberate choices here: I set publicNetworkAccess: 'Disabled' and defaultAction: 'Deny' from day one. You can always open access later for development hubs, but locking down production by default saves you from the "oops, I forgot to configure networking" panic.2. Deploy a GPT-4o Model Deployment via Azure CLIOnce your hub is up, deploying a model is straightforward:# List available models in the catalog az cognitiveservices account list-models \ --name ai-hub-prod-myapp \ --resource-group rg-ai-prod # Deploy GPT-4o with a specific SKU az cognitiveservices account deployment create \ --name ai-hub-prod-myapp \ --resource-group rg-ai-prod \ --deployment-name gpt-4o-global \ --model-name gpt-4o \ --model-version "2024-11-20" \ --model-format OpenAI \ --sku-name "GlobalStandard" \ --sku-capacity 10 The GlobalStandard SKU is worth explaining. It routes traffic through Microsoft's global infrastructure, giving higher throughput limits and better resilience than region-specific Standard. For Malaysian customers, this also means lower latency than you might expect. The trade-off? GlobalStandard is not available for all models yet, and it cannot use private network isolation. For sensitive workloads, use Standard with private endpoints.3. Create a Project and Deploy a Prompt Flow# Create a project within the hub az ml workspace create \ --name proj-doc-summariser \ --resource-group rg-ai-prod \ --hub-id /subscriptions/.../resourceGroups/.../providers/Microsoft.CognitiveServices/accounts/ai-hub-prod-myapp # Deploy a prompt flow as a managed endpoint az ml online-endpoint create \ --name doc-summariser-endpoint \ --resource-group rg-ai-prod \ --workspace-name proj-doc-summariser \ --auth-mode key az ml online-deployment create \ --name blue \ --endpoint-name doc-summariser-endpoint \ --resource-group rg-ai-prod \ --workspace-name proj-doc-summariser \ --file deploy/prompt-flow-deployment.yaml The az ml workspace create with the --hub-id flag is how you attach a project to an AI Hub. This is the wiring that enables the unified governance layer — policies applied at the hub level cascade down to all projects.4. Configure Content Safety with Bicepresource contentSafety 'Microsoft.CognitiveServices/accounts@2024-10-01' = { name: 'csa-${hubName}' location: location kind: 'ContentSafety' sku: { name: 'F0' // Free tier for evaluation } } resource aiHubProject 'Microsoft.MachineLearningServices/workspaces@2024-10-01' = { name: 'proj-customer-support' location: location kind: 'Default' properties: { hubResourceId: aiHub.id } // Associate Content Safety connection } The Content Safety integration is configure-once, enforce-everywhere. Every model deployment in the hub automatically gets content filtering unless you explicitly opt out — and you should have a very good reason before you do.Pitfalls: What the Documentation Doesn't Tell YouAfter deploying Microsoft Foundry (formerly Azure AI Foundry) across multiple production environments, here are the sharp edges I have collected.Pitfall 1: Model-as-a-Service Quota TrapOpen-source models via MaaS use a different quota pool than Azure OpenAI models. You cannot manage them through az cognitiveservices account deployment — they require az ml model deployment instead. Your IaC templates need two deployment patterns, and your monitoring setup must account for both.Pitfall 2: Private Endpoint Limitations with GlobalStandardIf your organisation mandates private network connectivity (common in Malaysian banking and GLCs), GlobalStandard is a non-starter — it does not support private endpoints. Use Standard SKU instead, which means lower throughput and regional failover. Plan your capacity modelling accordingly.Pitfall 3: Evaluation Is Not FreeThe built-in evaluators use GPT-4 as a judge model. Each evaluation run costs tokens. If you evaluate every prompt-response pair in production (which you should for compliance), the cost adds up. Budget roughly 10-15% of your inference cost for evaluation.Pitfall 4: RBAC ComplexityThe AI Hub, projects, and underlying Azure ML workspaces each have their own RBAC model. Hub-level contributor access does not automatically grant project access. Your identity team will need careful role planning. The built-in roles (Azure AI Developer, Azure AI Engineer, Azure AI Administrator) help, but coverage across scopes is still evolving.Pitfall 5: Region Availability Is Still Rolling OutNot all features are available in every region. Southeast Asia (Singapore) supports most capabilities, but newer features like the full Prompt Flow designer with custom evaluators remain in eastus2/francecentral. Check the regional availability matrix before committing. For Malaysian customers, Singapore is generally best for latency, but you may need a secondary region for certain evaluation workloads.Conclusion: Five Takeaways for Malaysian Enterprise ArchitectsMicrosoft Foundry (formerly Azure AI Foundry) is not a silver bullet, but it is a genuine step forward in taming AI sprawl. Here is what I want you to take away:1. Start with governance, not models. The AI Hub is powerful because of the policies, network rules, and safety configurations you bake into it. Deploy an empty hub first, configure guardrails, then add models incrementally. Retrofitting governance never goes well.2. Invest in IaC from day one. The az cognitiveservices account deployment and Bicep templates above should be your foundation. If you have more than three models in production, you need repeatable, auditable infrastructure-as-code.3. Budget for evaluation and safety. The cost of evaluating every output is real, but the cost of a compliance incident is far higher. Bake content safety and evaluation into your architecture from the start — the Foundry evaluators are production-grade.4. Plan for hybrid deployment patterns. You will likely end up with some models on GlobalStandard (cost/throughput) and some on Standard with private endpoints (compliance). Design for both. The Foundry control plane handles this complexity, but only if you model it explicitly.5. Stay close to the feature rollout map. Microsoft Foundry (formerly Azure AI Foundry) evolves rapidly. What is unavailable in Southeast Asia today may land next quarter. Test new capabilities in a sandbox subscription before committing in production.Microsoft Foundry (formerly Azure AI Foundry) gives us the control plane we have been asking for. It is not perfect — the documentation still lags behind the product, and some sharp edges remain. But for the first time, I can point to a single pane of glass that covers deployment, evaluation, safety, and governance for every model my teams deploy. That is worth taking seriously.Law Wen Feng is a Principal Solution Architect at Cloud Catalyst, where he helps Malaysian enterprises design and scale their AI platforms on Azure. The views expressed here are his own and do not necessarily reflect those of his employer or Microsoft. --- ## Cost and Sizing Considerations for Malaysia West Azure Workloads URL: https://wenfeng.my/mw-cost-sizing/ Author: admin Published: 2026-05-26 22:50:50 Modified: 2026-10-04 15:30:21 ⚡ TL;DRAuthor: Law Wen FengCampaign: Malaysia West Azure ArchitectureStatus: Draft for review — validate current Azure regional availability before implementation or publication.Practical Executive SummaryArchitecture Context: Cost as a Landing Zone PillarMalaysia West-Specific Cost ConsiderationsAzure Pricing Models and Commitment OptionsCost Estimation Methodology for Malaysia WestCost and Sizing Considerations for Malaysia West Azure WorkloadsAuthor: Law Wen FengCampaign: Malaysia West Azure ArchitectureStatus: Draft for review — validate current Azure regional availability before implementation or publication.Practical Executive SummaryCost and sizing is an architecture decision, not a finance spreadsheet problem. For Malaysian enterprises deploying or migrating workloads into the Azure Malaysia West region, the temptation is to treat cost estimation as a procurement step that happens after architecture design is finalised. This is backwards. Cost structure, pricing model selection, commitment strategy, licensing optimisation, and right-sizing discipline must be embedded into the architecture process from the start — because decisions made during design determine 70-80% of long-term cloud spend.The core recommendation is direct: use Azure Pricing Calculator for pre-deployment cost estimation, Azure Migrate for migration workloads, commitment-based pricing (Reserved Instances and Savings Plans) for stable workload cost reduction, and Azure Hybrid Benefit for licensing optimisation. Build cost governance into the landing zone through Azure Cost Management, resource tagging, budget alerts, and a FinOps review cadence. Most critically for Malaysia West, validate every pricing model, discount offer, service-level cost, and commitment availability against current regional pricing data before architecture sign-off. Malaysia West is a newer Azure region; pricing tiers, commitment model availability, and SKU support may differ from mature regions like Southeast Asia or East US.For IT leaders, this article provides a structured approach to Azure cost planning for Malaysia West deployments. It covers the full cost lifecycle: estimation methodology, pricing model comparison, right-sizing strategy, licensing optimisation, cost governance, and FinOps readiness. The goal is to help architects and finance stakeholders build a realistic cost baseline that survives contact with actual cloud billing.Architecture Context: Cost as a Landing Zone PillarIn the Malaysia West Azure Region: Architecture Planning Guide for Enterprises, cost is positioned as a core planning dimension alongside security, compliance, performance, and availability. This is not cosmetic. Every architectural decision — VM series selection, storage tier, database service, networking topology, DR design — has a direct cost implication. If the architecture team does not consider cost during design, the finance team inherits an unpredictable bill after deployment, and the optimisation work becomes remediation rather than planning.The Azure Landing Zone Design for Malaysia West article places resource tagging, cost allocation, and budget governance as landing-zone design responsibilities. The Hub-and-Spoke vs Azure Virtual WAN for Malaysia-Based Enterprises article discusses networking cost implications — VPN Gateway, ExpressRoute, and Virtual WAN pricing are significant cost components that must be modelled early. The Designing DR Between Malaysia West and Southeast Asia on Azure article highlights cross-region transfer costs and DR commitment pricing. The Monitoring and Backup Architecture for Malaysia West Azure Deployments article covers Log Analytics ingestion costs, backup vault pricing, and monitoring service costs that are often underestimated.This article extends these positions into a practical cost and sizing framework for Malaysia West.Malaysia West-Specific Cost ConsiderationsBefore discussing pricing models and optimisation strategies, the Malaysia West regional cost context must be understood clearly.Malaysia West is a newer Azure region. Newer regions have different cost characteristics than mature regions like Southeast Asia (Singapore) or East US. These differences may include:Pricing tiers that are not identical to other APAC regions. Pay-as-you-go rates, committed-use discounts, and service-level pricing may vary.Commitment model availability — Reserved Instances and Savings Plans may not be available for all VM series or services in Malaysia West. This must be verified before the architecture relies on commitment discounts.SKU availability — not all VM series, storage tiers, database SKUs, or service configurations may be available. This directly affects sizing options and cost modelling.Spot VM availability — Spot VMs provide deep discounts for interruptible workloads, but their availability per VM series per region varies and must be checked.Data egress pricing — cross-region data transfer costs depend on destination. Malaysia West egress to Southeast Asia, East Asia, or other regions must be costed separately from in-region traffic.Indirect cost drivers — data residency requirements that limit multi-region architectures, compliance requirements that mandate specific services, and support plan costs that are globally priced but may influence architecture decisions.Malaysia West cost and pricing availability warning: Do not assume Reserved Instances, Savings Plans, Spot VMs, specific VM series, database services, storage tiers, or pricing rates are identical to other Azure regions until verified against the current Azure Pricing Calculator and Microsoft regional availability pages. Malaysia West is a newer Azure region; pricing, commitment model availability, and SKU support may differ from Southeast Asia, East US, or other mature regions. Validate every pricing assumption, commitment discount, and service-level cost before budget sign-off.Azure Pricing Models and Commitment OptionsUnderstanding the available pricing models is the first step in building a cost baseline. Each model offers a different trade-off between discount depth and flexibility.Pay-as-you-GoThe default pricing model. No commitment. Highest per-unit cost. Appropriate for dev/test environments, proof-of-concept workloads, unpredictable or burst workloads, and short-term projects where commitment is not justified. For Malaysia West, pay-as-you-go rates must be checked on the Azure Pricing Calculator — rates may differ from Southeast Asia and should not be assumed equivalent.Reserved Instances (RIs)One-year or three-year commitment applied to specific VM series, size, region, and runtime (Windows or Linux). Provides up to 72% savings versus pay-as-you-go for eligible services. Scope options include single subscription, shared across subscriptions, or management group.RIs are the most rigid commitment model — the discount applies only to the specific SKU and region purchased. If the workload is resized or migrated, the RI may become underutilised. For Malaysia West, RI availability must be verified before architecture decisions depend on them. Not all VM series may support Reserved Instances in a newer region.When to use RIs: Stable, predictable production workloads where the VM series, size, and region will remain consistent for the commitment period.Savings Plans for ComputeA more flexible commitment model. One-year or three-year hourly spend commitment with cross-region and cross-series flexibility within the same billing scope. Provides up to 65% savings versus pay-as-you-go.Savings Plans are generally more forgiving than RIs for workload changes. If a workload moves between VM series or between regions within the billing scope, the Savings Plan commitment continues to apply. For Malaysia West, Savings Plan availability and discount application must be verified — newer regions may have different eligibility.When to use Savings Plans: Stable production workloads with expected growth, resizing, or regional movement. A good default commitment model for most production workloads.Azure Hybrid BenefitBrings existing Windows Server or SQL Server licences with active Software Assurance to Azure, eliminating OS licensing charges. Can provide up to 80% savings on Windows VMs (when combined with Reserved Instances) and up to 85% on SQL Server versus pay-as-you-go, depending on the licensing model and commitment term.Azure Hybrid Benefit is licence-based, not region-based — it applies regardless of which Azure region the workload runs in. The practical consideration for Malaysia West is whether the target VM series and OS are available in-region, because AHB is only useful if the workload can be deployed.Licence counting model: Windows Server 2016/2019/2022 Standard edition licences can be used for up to two virtual machines (OSEs) on the licensed host. Datacentre edition licences provide unlimited virtualisation rights on the licensed host. SQL Server core-based licences follow similar per-core allocation rules.When to use AHB: Any Windows or SQL Server workload where the organisation holds eligible licences with Software Assurance. This is not optional — it is a baseline optimisation that should be applied by default.Spot VMsDeep discounts (up to 90% in some cases) for interruptible, stateless, or batch workloads. Spot VMs use excess Azure capacity and can be evicted without notice when capacity is needed.For Malaysia West, Spot VM availability must be verified per VM series. Not all series support Spot pricing in all regions. Workloads using Spot VMs must be designed for graceful eviction handling — checkpointing, state externalisation, restart tolerance, and retry logic.When to use Spot VMs: Dev/test environments, CI/CD build agents, batch processing, rendering, simulation, and any workload that can tolerate interruption without data loss or SLA impact.Cost Estimation Methodology for Malaysia WestA disciplined estimation process prevents cost surprises after deployment.Azure Pricing CalculatorThe primary tool for pre-deployment cost estimation. Select Malaysia West as the target region, input compute, storage, database, networking, and other service requirements, and model multiple scenarios (baseline, growth, peak). Export estimates for stakeholder review and architecture sign-off.Best practices for estimation: - Model all cost components: compute, storage, networking, database, backup, monitoring, security, support. - Include data egress and cross-region transfer costs — these are often the largest surprise cost driver. - Factor in Azure support plan costs (Basic, Developer, Standard, Professional Direct, Unified) as a monthly recurring cost. - Build contingency (10-20%) for pricing changes and workload growth. - Save and version estimates so changes can be tracked over time.Azure MigrateFor migration workloads, Azure Migrate assesses on-premises infrastructure and recommends VM sizing, storage configurations, and Azure cost projections. The output includes sizing recommendations that can be imported into the Pricing Calculator. Use Azure Migrate as the first step for any migration business case.Total Cost of Ownership (TCO) CalculatorCompares on-premises infrastructure costs (hardware, facilities, labour, software) to projected Azure cloud costs. Useful for executive-level business case development and board-level approvals. The TCO Calculator is a communication tool, not a precise cost model — it provides directional comparison, not operational budgets.Right-Sizing StrategyRight-sizing is the process of matching Azure resource allocation to actual workload demand. Over-provisioned resources are the most common source of unnecessary cloud spend.Azure Advisor Cost RecommendationsAzure Advisor identifies underutilised VMs and recommends right-sizing or shutdown. Recommendations include potential monthly savings and a confidence level based on monitoring data. For Malaysia West, Advisor recommendations are based on regional SKU availability — not all recommended sizes may be available in-region.VM Right-Sizing ProcessMonitor: Collect CPU, memory, disk, and network utilisation data over a representative period — minimum 14 days, ideally 30+ days to capture weekly patterns and month-end processing peaks.Analyse: Use Azure Monitor metrics, VM insights, and Azure Advisor data to identify underutilised resources (CPU below 10-20%, memory consistently below 50%, disk IOPS far below provisioned capacity).Select: Match workload profile to the appropriate VM series — D-series for general purpose, E-series for memory-optimised, F-series for compute-optimised, Ls-series for storage-optimised. Consider burstable B-series for dev/test or variable workloads.Validate: Deploy the recommended size in a staging environment, validate performance under production-like load, and confirm the right-sized VM meets SLA requirements before decommissioning the larger instance.Storage Right-SizingMatch storage tier to access pattern. Hot, Cool, Cold, and Archive tiers in blob storage each have different per-GB pricing and access costs. Inappropriate tier selection — storing infrequently accessed data on Hot tier, or frequently accessed data on Archive tier — creates both cost waste and operational friction. Use storage lifecycle management to automate tiering based on age policies.Database Right-SizingAzure SQL Database: Choose between DTU and vCore models based on workload predictability. vCore provides more transparency and eligible for Azure Hybrid Benefit; DTU bundles compute and storage.Azure SQL Managed Instance: For workloads requiring SQL Server instance-level features (cross-database queries, SQL Agent, SSIS).Cosmos DB: Provisioned throughput for predictable workloads; serverless for sporadic or unpredictable access patterns; autoscale for variable load within a provisioned model.MySQL/PostgreSQL Flexible Server: Select tier and vCore count based on connection count, query complexity, and memory requirements.For each database service, verify availability in Malaysia West before including in the cost model.Cost Optimisation Checklist for Malaysia WestThe following checklist provides actionable optimisation steps. Each item should be validated against current Malaysia West regional availability before implementation.Azure Hybrid Benefit: Apply for all eligible Windows and SQL Server workloads where the organisation holds Software Assurance licences. This is a default optimisation, not an optional one.Reserved Instances or Savings Plans: Commit for stable, predictable production workloads. Savings Plans are the preferred default due to cross-series and cross-region flexibility. Verify availability in Malaysia West before committing.Right-size VMs and databases: Use Azure Advisor recommendations, validated against actual workload metrics and regional SKU availability.Spot VMs for non-critical workloads: Use for dev/test, CI/CD, batch processing, and any interruptible workload. Verify Spot availability in Malaysia West per VM series.Auto-scale where possible: Match resource allocation to demand rather than provisioning for peak. Use VM scale sets, App Service autoscale, and database autoscale.Delete or deallocate idle resources: Remove unused VMs, unattached disks, orphaned public IPs, and decommissioned endpoints. Azure Advisor identifies idle resources.Storage lifecycle management: Automate tiering from Hot → Cool → Cold → Archive based on data age and access patterns.Azure DevTest subscriptions: Use reduced pricing for non-production environments. Verify DevTest pricing applicability in Malaysia West.Tag all resources: Mandatory tags for environment, owner, cost-centre, project, and application enable cost allocation and accountability.Set Azure Cost Management budgets and alerts: Per-subscription or per-resource-group budgets with threshold alerts at 50%, 75%, 90%, and 100%.Planning Azure cost and sizing for Malaysia West workloads? I can review your workload sizing, pricing model selection, commitment strategy, licensing optimisation, and cost governance framework — then produce a practical Azure cost baseline and sizing recommendation before you deploy.Cost Governance and FinOps ReadinessCost optimisation without governance is a one-time exercise that degrades over time. FinOps is the practice of continuous cost accountability, and it should be embedded into the landing zone from day one.Azure Cost ManagementAzure Cost Management provides spend visibility, anomaly detection, budget management, and billing data export. Use Cost Analysis for real-time spend tracking by subscription, resource group, tag, or service. Set budgets with Action Group notifications at defined thresholds. Use the Exports API to integrate billing data with external financial systems.Resource Tagging StrategyMandatory tags should include at minimum: environment, owner, cost-centre, project, application, and criticality. Use Azure Policy to enforce tagging at deployment — resources without required tags should be flagged or blocked. Tag-based cost allocation in Cost Analysis provides per-team, per-project, and per-environment spend visibility.Budget and Accountability ModelEstablish per-subscription or per-resource-group budgets aligned to approved architecture costs. Configure threshold alerts at 50% (informational), 75% (warning), 90% (critical), and 100% (escalation). Use Action Groups to automate responses — notify the cost owner, trigger a review, or deallocate non-critical resources.FinOps CultureFinOps is not a tool — it is a cultural practice. Align engineering, finance, and operations around cost accountability. Establish a regular cost review cadence (weekly during initial deployment, monthly during steady state). Include cost implications in every architecture review. Make cost performance a team-level metric, not just a finance team concern.Common Cost Mistakes in New Azure RegionsArchitects deploying in Malaysia West should avoid these patterns:Assuming pricing parity with mature regions like Southeast Asia or East US. Every pricing assumption must be verified against current Malaysia West data.Failing to validate RI or Savings Plan availability before relying on commitment discounts. If commitment models are not available, the cost baseline changes significantly.Over-provisioning VMs because sizing estimates are based on on-premises capacity rather than cloud-native right-sizing analysis.Ignoring data egress and cross-region transfer costs. These are often the largest surprise cost component in multi-region or hybrid architectures.Not tagging resources, leading to opaque cost allocation and no accountability.Using pay-as-you-go for stable production workloads instead of commitment pricing. The cost difference is substantial over 12-36 months.Forgetting Azure Hybrid Benefit for Windows and SQL workloads. This is the single highest-impact optimisation for Microsoft-licensed workloads.Ignoring Azure Advisor recommendations. Advisor identifies specific, actionable cost optimisations with estimated savings — these should be reviewed monthly.Risks and Constraints Risk Impact Mitigation Malaysia West pricing differs from projected estimates Budget overrun, stakeholder confidence loss Verify all pricing against Azure Pricing Calculator before architecture sign-off Reserved Instances not available for target VM series Cannot achieve commitment discount; higher ongoing cost Confirm RI/Savings Plan availability before selecting VM series Spot VM eviction disrupts workloads Service interruption for interruptible workloads Design Spot VM workloads for graceful eviction; maintain on-demand fallback Over-provisioning wastes budget Unnecessary spend on unused capacity Implement right-sizing from day one; review Azure Advisor monthly Missing resource tags No cost allocation; no accountability Enforce tagging via Azure Policy; block untagged resources Cross-region data egress costs Unexpected high bandwidth charges Model egress costs explicitly; use private endpoints and avoid unnecessary cross-region traffic Azure Hybrid Benefit not applied Paying full OS licensing cost unnecessarily Audit all Windows/SQL workloads; apply AHB at deployment Related ReadingMalaysia West Azure Region: Architecture Planning Guide for Enterprises — the foundational planning guide for all Malaysia West architecture decisions.Azure Landing Zone Design for Malaysia West — landing zone design including cost governance, resource tagging, and budget management.Hub-and-Spoke vs Azure Virtual WAN for Malaysia-Based Enterprises — networking cost implications for VPN Gateway, ExpressRoute, and Virtual WAN.Designing DR Between Malaysia West and Southeast Asia on Azure — DR cost implications and cross-region transfer pricing.Monitoring and Backup Architecture for Malaysia West Azure Deployments — monitoring and backup service costs including Log Analytics ingestion and backup vault pricing.Planning Azure cost and sizing for Malaysia West workloads? I can review your workload sizing, pricing model selection, commitment strategy, licensing optimisation, and cost governance framework — then produce a practical Azure cost baseline and sizing recommendation before you deploy. Contact Law Wen Feng for an Azure architecture review. --- ## Monitoring and Backup Architecture for Malaysia West Azure Deployments URL: https://wenfeng.my/mw-monitoring-backup/ Author: admin Published: 2026-05-24 22:51:42 Modified: 2026-10-04 15:30:21 ⚡ TL;DRMonitoring and backup are not optional extras for Azure workloads in Malaysia West. They are foundational landing-zone components.Practical Executive SummaryArchitecture Context: Why Monitoring and Backup Belong in the Landing ZoneMalaysia West-Specific ConsiderationsDesign RecommendationsRisks and ConstraintsMonitoring and Backup Architecture for Malaysia West Azure DeploymentsPractical Executive SummaryMonitoring and backup are not optional extras for Azure workloads in Malaysia West. They are foundational landing-zone components. If you deploy compute, networking, identity, and applications without designing observability and data protection from the start, you create operational blind spots and data-loss exposure that are far more expensive to fix after go-live than during initial architecture.For Malaysian enterprises deploying or migrating workloads into the Malaysia West Azure region, the core recommendation is direct: design monitoring and backup architecture as first-class landing-zone pillars. Use Azure Monitor, Log Analytics, Azure Monitor Agent, and diagnostic settings as the observability foundation. Use Azure Backup with Recovery Services vaults and Backup vaults for data protection. Validate every monitoring service, backup feature, retention option, vault placement, zone-redundancy capability, and alert or notification mechanism against current Malaysia West regional availability before deployment. Treat backup and monitoring design as part of the initial landing zone, not a post-migration task.This matters more in Malaysia West than in a long-established Azure region. Newer regions can have service availability gaps, limited SKU support, missing availability-zone coverage for specific services, and unverified cross-region replication behaviour. An architect who copies a Southeast Asia monitoring and backup design into Malaysia West without a structured validation gate is making an assumption that may fail at the worst possible time — during incident response, during data recovery, or during a compliance audit.The architecture design should answer these questions before any production workload is deployed:Which monitoring services and features are confirmed available in Malaysia West?Where should Log Analytics workspaces, Application Insights, and alerting components be deployed?Which backup vault types are supported in Malaysia West, and what redundancy options exist?How do backup job events integrate with operational alerting and incident response?What is the fallback if a required monitoring or backup feature is not yet available in Malaysia West?A practical monitoring and backup architecture gives operational teams confidence that they can detect, alert on, investigate, recover, and audit every workload deployed in Malaysia West.Architecture Context: Why Monitoring and Backup Belong in the Landing ZoneIn a well-structured Azure landing zone, monitoring and backup are not tasks assigned after workloads are deployed. They are architectural decisions made during the platform design phase. The Azure Landing Zone Design for Malaysia West article positions logging, monitoring, and backup as core landing-zone responsibilities alongside identity, networking, security, and governance. This article extends that position into specific monitoring and backup design guidance.Monitoring Architecture for Malaysia WestThe monitoring architecture for Malaysia West should be built on Azure-native services as the default. The reason is straightforward: Azure-native tools are the only monitoring stack that has native integration with Azure resource providers, platform logs, activity logs, metrics, and diagnostic data across every Azure service. Third-party monitoring tools have a role in multi-cloud, on-premises, or application-specific scenarios, but they should supplement, not replace, the Azure-native monitoring foundation.Azure Monitor is the central observability service. It collects metrics from every Azure resource by default, aggregates platform logs through diagnostic settings, and provides alerting, workbooks, dashboards, and integration with operational tools. For Malaysia West, the first design question is whether Azure Monitor and its dependent services are available in-region. This is not a theoretical question. Regional availability determines whether metrics, logs, alerts, and dashboards operate with expected latency and whether data residency requirements are met.Log Analytics workspace design is the most important monitoring architecture decision after Azure Monitor availability. A Log Analytics workspace stores logs and provides the query engine, alert engine, and dashboard foundation. For Malaysia West deployments, architects must decide whether to place the Log Analytics workspace in Malaysia West or in a centralised monitoring region. Placing the workspace in Malaysia West keeps log data in-country, which may matter for data residency. Centralising in a region like Southeast Asia may simplify multi-region operations but introduces cross-region data transfer. The trade-offs are latency, cost, data residency compliance, and operational simplicity. There is no single right answer; the design should document the decision and the rationale.Azure Monitor Agent (AMA) replaces the legacy Log Analytics agent and provides a unified data-collection path. Data Collection Rules define which logs and metrics are collected from which resources and where they are sent. For Malaysia West, verify AMA availability and Data Collection Rules support before designing agent-based collection.Alerting and action groups turn monitoring data into operational signals. Metric alerts, log alerts, and activity log alerts provide different detection mechanisms. Action groups route notifications to email, SMS, webhooks, ITSM tools, Logic Apps, Azure Functions, or automation runbooks. Alert processing rules manage suppression, throttling, and routing for high-volume environments. The design should include severity tiers, escalation paths, on-call integration, and alert fatigue reduction from the start. Do not wait for production incidents to design alerting.Dashboards and workbooks provide operational visibility. Azure workbooks offer interactive monitoring views for VMs, containers, networking, databases, and custom scenarios. Azure dashboards provide team-level operational visibility. For Malaysia West, verify which pre-built monitoring solutions and data sources are available in-region.Application monitoring uses Application Insights for application performance monitoring, including live metrics, distributed tracing, dependency mapping, and availability tests. For Malaysia West, verify Application Insights availability and whether managed-identity-based ingestion is supported.Network and security monitoring involves multiple complementary services. Network Watcher provides connection monitoring, flow logs, packet capture, and topology visibility. Microsoft Defender for Cloud provides security posture management and threat protection. Microsoft Sentinel provides SIEM and SOAR capabilities for security monitoring and incident response. For Malaysia West, verify Network Watcher, Defender for Cloud, and Sentinel availability and feature support independently — each has its own regional availability profile.Container and Kubernetes monitoring uses Azure Monitor for containers, Container Insights, Managed Prometheus, and Managed Grafana. For organisations running AKS workloads in Malaysia West, verify these services are available before committing to a container monitoring architecture.Hybrid monitoring uses Azure Arc to extend monitoring to non-Azure servers and resources. Azure Monitor Agent on hybrid machines collects logs and metrics into the same Log Analytics workspace. For Malaysia West deployments with hybrid requirements, verify Azure Arc agent support and hybrid connectivity requirements.Backup Architecture for Malaysia WestBackup architecture is the data-protection counterpart to monitoring. While monitoring detects problems, backup enables recovery. Both are non-negotiable for production workloads.Azure Backup provides centralised backup management for virtual machines, Azure file shares, SQL Server, SAP HANA, Azure databases, managed disks, and blob storage. The service uses two vault types:Recovery Services vaults are the established vault type, storing backup data, recovery points, and policy configurations for VMs, files, SQL, and other workloads.Backup vaults are a newer vault type designed for blob, disk, and database backup use cases.For Malaysia West, the first question is whether both vault types are available and what redundancy options they support.Recovery Services vault design should follow standard best practice with Malaysia West validation. Vaults should be placed in the same region as the protected workloads. Vault redundancy options include locally-redundant storage (LRS), geo-redundant storage (GRS), zone-redundant storage (ZRS), and read-access geo-redundant storage (RA-GRS). For Malaysia West, verify which redundancy options are available. Do not assume zone-redundant vaults are supported in Malaysia West without confirmation.VM backup architecture uses snapshots and Recovery Services vaults. Backup policies define daily, weekly, monthly, and yearly retention, instant restore snapshots, and application-consistent versus crash-consistent backup options. Cross-region restore capability must be verified independently — do not assume cross-region restore from Malaysia West is supported or that the target region is defined by default.Azure file share backup supports snapshot-based backup and vaulted backup for SMB and NFS file shares. SQL Server and Azure SQL backup uses automated backups, long-term retention, and point-in-time restore. Managed disk backup uses Azure Disk Backup for snapshot-based disk protection. Blob storage backup uses vaulted backup with point-in-time restore and soft delete. For each of these workload types, verify availability in Malaysia West before including them in the architecture.Azure Site Recovery is not a backup service — it is a disaster recovery and replication service. The distinction matters architecturally. Azure Backup provides backup and recovery within a region and optionally across regions through cross-region restore. Azure Site Recovery replicates workloads to a secondary region for disaster recovery failover. For Malaysia West, verify ASR availability, supported source and target regions, and vault support independently from backup availability. Do not assume ASR is available for Malaysia West without verification.Backup security and compliance should be designed from the start, not bolted on later. Encryption at rest and in transit is standard. Soft delete protects against accidental or malicious deletion. Multi-user authorisation (MUA) adds approval requirements for critical backup operations. Immutable vault policies provide WORM compliance. Azure Policy can enforce backup governance by ensuring all protected resources have backup policies applied. For Malaysia West, verify that soft delete, MUA, and immutable vault features are available with the vault types deployed.Monitoring and Backup IntegrationThe two domains should not operate in silos. Monitor backup job success, failure, and restore operations through Azure Monitor. Use BackupCentre and Recovery Services vault insights for operational visibility. Alert on backup failures, missed schedules, and restore issues through the same action groups used for workload monitoring. Monitor activity log events for backup policy changes and vault configuration changes. Integrate monitoring-triggered and backup-triggered procedures with ITSM tools, incident management platforms, and operational runbooks.This integration ensures that the operations team has a unified view of workload health, backup status, and recovery readiness — not two separate dashboards with no correlation.Malaysia West-Specific ConsiderationsMalaysia West introduces specific considerations that do not apply identically in more mature Azure regions.Service Availability Validation Is MandatoryThe Malaysia West Azure Region: Architecture Planning Guide for Enterprises article establishes the principle that every service named in the architecture must be validated against current Microsoft regional availability. This principle applies with particular force to monitoring and backup, because these services span multiple Azure resource providers, each with their own availability profile.A monitoring and backup architecture that includes Azure Monitor, Log Analytics, Application Insights, Network Watcher, Defender for Cloud, Sentinel, Recovery Services vaults, Backup vaults, Azure Backup for VMs, Azure Backup for SQL, Azure Backup for files, Azure Backup for disks, Azure Backup for blobs, Managed Prometheus, Managed Grafana, and Azure Arc is referencing approximately fifteen distinct services or service components. Each must be validated independently. The availability of one does not imply the availability of another.Availability Zone CoverageMalaysia West may support availability zones at the region level, but service-level AZ support varies. A monitoring and backup architecture should not assume that Log Analytics workspaces, Recovery Services vaults, Backup vaults, or Application Insights support zone-redundant deployment in Malaysia West until confirmed. Design the architecture to tolerate scenarios where specific services are not zone-redundant in the target region. This may mean accepting local redundancy for certain monitoring and backup components while designing workload compute and storage for zone redundancy.Cross-Region Restore and Disaster RecoveryFor backup, cross-region restore is a capability that extends data protection beyond a single region. For Malaysia West, the availability of cross-region restore, the supported target region, and the restore performance characteristics must be verified. The Designing DR Between Malaysia West and Southeast Asia on Azure article discusses disaster recovery planning for Malaysia West — backup cross-region restore is one component of the broader DR strategy, but it is distinct from Azure Site Recovery replication.Data Residency and Log RetentionMalaysian organisations subject to data residency requirements must consider where monitoring logs, backup data, and recovery points are stored. Placing Log Analytics workspaces and Recovery Services vaults in Malaysia West keeps data in-country. Centralising in another region may simplify operations but introduces data residency considerations. The architecture should document the data residency decision and its implications for compliance and operational recovery.Third-Party Backup and Monitoring AlternativesAzure-native monitoring and backup should be the default unless workload, compliance, or operational requirements justify alternatives. Some organisations may use third-party backup tools such as Veeam on Azure for specific workloads. Third-party tools may offer features that Azure Backup does not yet support in Malaysia West, or may be preferred for operational or licensing reasons. The architecture should document which workloads use Azure-native services and which use third-party alternatives, with clear rationale.Design RecommendationsDesign monitoring and backup during landing-zone deployment, not after workload migration. These are architectural components, not post-deployment tasks.Use Azure-native monitoring and backup as the default. Supplement with third-party tools only where workload, compliance, or operational requirements justify the exception.Place Log Analytics workspaces and Recovery Services vaults in Malaysia West unless there is a documented reason for centralising in another region.Validate every monitoring and backup service, SKU, feature, and redundancy option against current Microsoft regional availability before deployment. Build a service-by-service evidence table.Design alerting with severity tiers, escalation paths, and alert fatigue reduction from the start. Do not wait for incidents to tune alerts.Integrate backup monitoring with workload monitoring. Use Azure Monitor to alert on backup failures, missed schedules, and restore issues.Implement backup security controls early: encryption, soft delete, MUA, immutable vault policies, and Azure Policy governance.Test backup restores regularly. Backup without tested restore is an assumption, not a guarantee.Document the cross-region restore and DR strategy. Verify ASR availability and target regions for Malaysia West independently from backup availability.Monitor cost drivers: Log Analytics ingestion volume, Application Insights data volume, backup storage retention, and snapshot storage.Risks and ConstraintsService availability risk: Monitoring and backup services may not all be available in Malaysia West at the required feature level. Mitigation: build a service availability matrix and validate before design approval.Zone-redundancy risk: Not all monitoring or backup services may support availability zones in Malaysia West. Mitigation: design for the actual availability-zone support, not the assumed support.Cross-region restore risk: Cross-region restore from Malaysia West may not be available or may have undefined target regions. Mitigation: verify before including in the architecture; design independent DR if cross-region restore is not confirmed.Cost risk: Log Analytics ingestion cost can escalate quickly with high-volume log collection. Mitigation: design Data Collection Rules with intentional filtering, set retention policies based on actual requirements, and model costs using the Azure Pricing Calculator.Operational complexity risk: Monitoring and backup across multiple services in a newer region may surface operational gaps. Mitigation: establish runbooks, test restore procedures, and tune alerting before production workload go-live.Third-party dependency risk: If third-party backup or monitoring tools are used, verify their Malaysia West support, agent compatibility, and vault or data storage location.Malaysia West Monitoring and Backup Validation ChecklistBefore deploying monitoring and backup architecture in Malaysia West, validate the following. Do not assume availability; confirm against current Microsoft documentation and your Azure tenant.Azure Monitor: Confirm metrics, logs, alerts, and activity log availability in Malaysia West.Log Analytics workspace: Confirm workspace creation, data ingestion, query engine, and retention support in Malaysia West.Log Analytics zone-redundant workspace: Verify zone-redundant workspace availability in Malaysia West.Azure Monitor Agent (AMA): Confirm AMA deployment and Data Collection Rules support in Malaysia West.Application Insights: Confirm application performance monitoring availability in Malaysia West.Network Watcher: Confirm connection monitor, flow logs, packet capture, and topology availability in Malaysia West.Microsoft Defender for Cloud: Confirm security posture and threat protection availability in Malaysia West.Microsoft Sentinel: Confirm SIEM/SOAR, data ingestion, analytics rules, and automation availability in Malaysia West.Azure Monitor for containers: Confirm Container Insights, Managed Prometheus, and Managed Grafana availability in Malaysia West.Azure Arc: Confirm hybrid agent support and connectivity requirements for Malaysia West.Recovery Services vault: Confirm vault creation, SKU options, and redundancy support in Malaysia West.Recovery Services vault zone-redundant storage: Verify ZRS vault availability in Malaysia West.Backup vault: Confirm Backup vault creation and feature support in Malaysia West.Azure Backup for VMs: Confirm VM backup, backup policies, and restore capabilities in Malaysia West.Azure Backup for Azure file shares: Confirm file share backup availability in Malaysia West.Azure Backup for SQL Server on Azure VMs: Confirm SQL backup, log backup, and restore capabilities in Malaysia West.Azure Backup for managed disks: Confirm Azure Disk Backup availability in Malaysia West.Azure Backup for blob storage: Confirm blob backup availability in Malaysia West.Cross-region restore: Verify cross-region restore availability and target region for Malaysia West.Azure Site Recovery: Verify ASR source and target region support for Malaysia West.Backup security controls: Confirm soft delete, MUA, immutable vault, and Azure Policy governance support.Diagnostic settings: Confirm platform and resource log routing to Log Analytics, Storage, or Event Hub from Malaysia West.Alerting and action groups: Confirm alert processing and action group delivery from Malaysia West.Planning monitoring and backup for Malaysia West Azure workloads? I can review your monitoring architecture, backup strategy, vault placement, retention policies, zone-redundancy requirements, alerting design, and operational readiness — then produce a practical Azure monitoring and backup architecture recommendation before you deploy. Contact me for an Azure architecture review.Related ReadingMalaysia West Azure Region: Architecture Planning Guide for Enterprises — the foundational guide for Malaysia West Azure architecture planning.Azure Landing Zone Design for Malaysia West — landing-zone architecture that positions monitoring and backup as core pillars.Hub-and-Spoke vs Azure Virtual WAN for Malaysia-Based Enterprises — network architecture that affects Network Watcher and monitoring design.Designing DR Between Malaysia West and Southeast Asia on Azure — disaster recovery strategy that complements backup architecture.Azure Backup vs Veeam on Azure: Which DR Strategy for Malaysian SMEs? — backup strategy alternatives for Malaysian organisations.This article is part of the Malaysia West Azure Architecture campaign on wenfeng.my/. All Malaysia West service availability claims must be verified against current Microsoft regional availability documentation before implementation or publication. --- ## Microsoft Fabric Lakehouse Migration: Moving from Traditional Data Warehouses to a Unified Analytics Platform URL: https://wenfeng.my/microsoft-fabric-lakehouse-migration-moving-from-traditional-data-warehouses-to-a-unified-analytics-platform-2/ Author: admin Published: 2026-05-21 00:48:42 Modified: 2026-10-04 15:30:21 ⚡ TL;DRMicrosoft Fabric is a Microsoft SaaS analytics platform that combines data engineering, data integration, data warehousing, real-time intelligence, data science, and Power BI over OneLake.What Is a Lakehouse?Why Migrate to Fabric Lakehouse in 2026?Migration ArchitectureStep-by-Step MigrationPerformance Optimization for Fabric LakehouseMicrosoft Fabric is a Microsoft SaaS analytics platform that combines data engineering, data integration, data warehousing, real-time intelligence, data science, and Power BI over OneLake. Recent Fabric documentation confirms lakehouse support for Delta Lake, Spark and SQL access, OneLake shortcuts, and Direct Lake semantic models; however, shortcut source support, workload availability, and regional availability must still be verified against current Microsoft Learn documentation before designing around a specific external storage target. For Malaysian enterprises running on-premises SQL Server data warehouses or Azure Synapse Dedicated Pools, the question is no longer "should we migrate?" but "how do we migrate without breaking existing BI reports and ETL pipelines?"This guide provides a practical, incrementally-applicable migration plan — from inventory and assessment through parallel run and decommissioning — with real code examples and Malaysian enterprise context.What Is a Lakehouse?A lakehouse combines the flexibility of a data lake (cheap object storage, open formats like Parquet and Delta Lake) with the reliability of a data warehouse (ACID transactions, schema enforcement, BI integration). Microsoft Fabric implements this with OneLake as the centralized logical data lake for Fabric — designed to support shared analytics data access across Fabric experiences such as lakehouses, warehouses, Spark, SQL, Power BI semantic models, and notebooks.Key architectural principles:One copy, many engines. Data is stored once in Delta format in OneLake. Compute engines (Spark, SQL, Notebooks) read the same files. No costly data movement between systems.Open formats. Delta Lake and Parquet are open-source, Apache-licensed formats. Your data is never locked into a proprietary binary format.Shortcuts instead of copies. Instead of copying data between workspaces or clouds, Fabric shortcuts let you reference external data (ADLS Gen2, AWS S3, GCS) without duplicating it.Why Migrate to Fabric Lakehouse in 2026?1. Cost Reduction vs. Traditional SQL Data WarehousesTraditional Azure Synapse Dedicated Pools are provisioned by data warehouse units (DWUs), while Fabric uses capacity-based billing through Fabric capacity SKUs. Exact monthly cost depends on region, capacity size, reservation/commitment model, workload concurrency, data volume, and whether the workload can pause or scale. Use the official Azure Synapse and Microsoft Fabric pricing pages or Azure Pricing Calculator before committing to a budget.How to build a credible comparison for a Malaysian mid-size enterprise (for example, a 5 TB warehouse): Cost Component Traditional Synapse / SQL Warehouse Fabric Lakehouse Storage Database/data-lake storage, backup retention, redundancy, and export copies OneLake storage plus any shortcut cache/egress implications Compute Provisioned or reserved warehouse capacity, query concurrency, and ETL runtime Fabric capacity SKU, workload concurrency, Spark/SQL usage, and scale/pause pattern Integration ADF/SSIS runtime, self-hosted integration runtimes, and network transfer Fabric Data Factory/Dataflow/Notebook runtime and gateway/network requirements BI Power BI licensing, refresh mode, and semantic model design Power BI licensing plus Direct Lake/Import/DirectQuery suitability Operations Monitoring, performance tuning, backup/restore, security administration Capacity monitoring, workspace governance, lineage, security, and optimization jobs Do not assume a fixed percentage saving. Fabric can reduce duplication and simplify operations, but the commercial outcome depends on workload shape, capacity sizing, reservation choices, data movement, and governance overhead.2. Unified AnalyticsA shared OneLake data foundation can serve multiple analytics workloads when the chosen Fabric item and access mode support the pattern:SQL analytics via the T-SQL endpoint (Power BI, SSMS, Azure Data Studio)Data engineering via Spark notebooks (Python, Scala, SQL)Real-time analytics via Event Streams (Kafka-compatible ingestion)AI/ML via Fabric notebooks with direct MLflow integrationData science via integrated Synapse Data ScienceNo more copying data from the warehouse to a data lake for ML training, then to Power BI for dashboards.3. AI and Copilot IntegrationFabric Copilot capabilities can assist with data engineering, data science, real-time intelligence, data warehousing, and Power BI experiences when they are available and enabled for the tenant. Treat Copilot as an accelerator, not a substitute for data engineering review, governance, and testing.Migration ArchitectureThe target architecture for a Fabric lakehouse migration:Source Systems Fabric Lakehouse Consumption ┌──────────────┐ ┌─────────────────────┐ ┌──────────────┐ │ On-prem SQL │──Data Factory──▶│ OneLake (Delta) │──SQL─────▶│ Power BI │ │ Server │ │ Tables/ │ endpoint │ │ ├──────────────┤ │ ├── sales │ ├──────────────┤ │ Azure SQL DB │──Notebook──────▶│ ├── customers │──Spark───▶│ Notebooks │ │ │ │ ├── inventory │ │ (ML, ETL) │ ├──────────────┤ │ └── orders │ ├──────────────┤ │ CSV/Parquet │──OneLake───────▶│ │──RealTime─▶│ Power BI │ │ files (ADLS) │ Shortcut │ Shortcuts: │ │ (Direct Lake)│ └──────────────┘ │ ├── archive (S3) │ └──────────────┘ │ └── raw-logs (ADLS) │ └─────────────────────┘ Step-by-Step MigrationStep 1: Assess Your Current WarehouseBefore any migration, run a thorough inventory:-- SQL Server: Inventory all tables, sizes, and row counts SELECT s.name AS SchemaName, t.name AS TableName, SUM(p.rows) AS TotalRows, SUM(a.total_pages) * 8 / 1024 AS TotalSizeMB, MAX(i.type_desc) AS IndexType FROM sys.schemas s INNER JOIN sys.tables t ON s.schema_id = t.schema_id INNER JOIN sys.indexes i ON t.object_id = i.object_id INNER JOIN sys.partitions p ON i.object_id = p.object_id AND i.index_id = p.index_id INNER JOIN sys.allocation_units a ON p.partition_id = a.container_id WHERE i.index_id IN (0, 1) -- Heap or clustered index GROUP BY s.name, t.name ORDER BY TotalSizeMB DESC; Also document: - Query patterns: Most frequently run queries, slowest queries, peak concurrency - ETL dependencies: SSIS packages, ADF pipelines, stored procedure chains - BI connections: Which Power BI datasets, Excel files, and reporting tools connect to the warehouse - Data freshness requirements: Real-time? Daily? Weekly batch?Malaysian enterprise tip: For BNM-regulated or PDPA-sensitive workloads, document lineage, data classification, retention, access approvals, and audit evidence before migration. Confirm sector-specific requirements with the organisation's compliance owner rather than assuming one generic regulatory control.Step 2: Create Fabric Lakehouse and OneLake Storage# PowerShell: Create Fabric workspace via API $body = @{ displayName = "analytics-lakehouse-prod" description = "Production lakehouse for migrated data warehouse" } | ConvertTo-Json Invoke-RestMethod -Uri "https://api.fabric.microsoft.com/v1/workspaces" \ -Method POST \ -Headers @{ Authorization = "Bearer $accessToken" "Content-Type" = "application/json" } \ -Body $body In the Fabric portal: 1. Navigate to your workspace -> New item -> Lakehouse 2. Name it analytics-lakehouse 3. Choose a Fabric capacity/workspace region that satisfies the organisation's data residency requirements, and confirm current Fabric workload availability in that region before deployment 4. Create default folders: Tables/, Files/, and subfolders like Tables/sales, Tables/refStep 3: Migrate DataMethod A: Fabric Data Factory (Pipelines) — Best for scheduled, managed migrationsCreate a pipeline with Copy Activity that reads from your source warehouse and writes to OneLake as Delta:Source: SQL Server / Azure SQL DB via Linked ServiceSink: OneLake Lakehouse table (Delta format)Staging: Enabled for large tables (uses temporary ADLS Gen2)Method B: Fabric Notebook (Spark) — Best for complex transformations during migration# Fabric notebook: Schema migration with type mapping from pyspark.sql import SparkSession from pyspark.sql.types import * # Define schema mapping for SQL Server → Delta Lake types type_mapping = { "int": IntegerType(), "bigint": LongType(), "varchar": StringType(), "nvarchar": StringType(), "datetime": TimestampType(), "decimal": lambda p, s: DecimalType(p, s), "float": DoubleType(), "bit": BooleanType(), } # Read from source SQL Server source_table = "sales.orders" df = spark.read \ .format("sqlserver") \ .option("url", "jdbc:sqlserver://source-server.database.windows.net:1433") \ .option("databaseName", "WarehouseDB") \ .option("table", source_table) \ .option("user", dbutils.secrets.get("sql-user")) \ .option("password", dbutils.secrets.get("sql-password")) \ .load() # Write to OneLake as Delta table target_path = "Tables/sales/orders" df.write \ .format("delta") \ .mode("overwrite") \ .option("delta.autoOptimize.optimizeWrite", "true") \ .save(target_path) # Register table for SQL endpoint access spark.sql(f"CREATE TABLE IF NOT EXISTS sales.orders USING DELTA LOCATION '{target_path}'") Method C: OneLake Shortcut — Best for data that must stay in its source locationFor data that cannot move (compliance/regulatory reasons), create a shortcut directly to the source:In your lakehouse, click "New Shortcut"Select source type: Azure Data Lake Storage Gen2, Amazon S3, or Google Cloud StoragePoint to the folder pathFabric creates the shortcut — no data moves, queries read in-placeStep 4: Migrate ETL Pipelines Source ETL Technology Fabric Migration Target Migration Effort SSIS packages Fabric Data Factory / Data Factory in Microsoft Fabric Medium to High — reassess control flow, connectors, custom code, and scheduling ADF pipelines Fabric Data Factory / Data Factory in Microsoft Fabric Low to Medium — concepts transfer, but validate connector parity, expressions, triggers, and integration runtime dependencies SQL Stored Procedures (ETL) Fabric SQL scripts or Spark notebooks Medium to High — T-SQL mostly works but Spark SQL has differences SSAS Multidimensional cubes Power BI datasets (Direct Lake over OneLake) High — different modeling paradigm Azure Data Factory (v2) Fabric Data Factory / Data Factory in Microsoft Fabric Low to Medium — migrate patterns carefully and test feature compatibility Step 5: Configure SecurityFabric lakehouse security uses a layered model:OneLake data access roles — Define which users/groups can read/write which tablesSQL analytics endpoint permissions — Use supported SQL permissions for query access where applicable, and validate the exact permission model for lakehouse and warehouse items.Row-level security (RLS) — Commonly enforced in Power BI semantic models with DAX roles; validate whether any SQL-layer security feature is supported for the specific Fabric item type before using it.For Malaysian PDPA-sensitive data, avoid relying on illustrative SQL masking syntax unless the target engine explicitly supports it. Classify personal data, restrict access through workspace/item/security roles, apply sensitivity labels, minimise copied PII, and use masking, tokenisation, or encryption in the supported layer for the chosen architecture.Step 6: Migrate BI ReportsPoint Power BI datasets to the Fabric SQL analytics endpoint:Old connection string: Server=myserver.database.windows.net;Database=WarehouseDB New connection string: Server=<copy the SQL analytics endpoint server from the Fabric portal>; Database=<copy the endpoint/database name from the Fabric portal> Prefer Direct Lake mode in Power BI (not Import, not DirectQuery). Direct Lake reads Delta files directly from OneLake without going through the SQL engine — it combines the speed of Import mode with the data freshness of DirectQuery.Step 7: Validate with Parallel RunRun both systems in parallel for at least two full business cycles:-- Validation query: compare row counts between old and new SELECT 'Old Warehouse' AS Source, COUNT(*) AS RowCount FROM linked.old_warehouse.sales.orders UNION ALL SELECT 'Fabric Lakehouse', COUNT(*) FROM sales.orders; -- Compare aggregated values SELECT o.Source, o.OrderYear, SUM(o.TotalAmount) AS TotalRevenue FROM ( SELECT 'Old' AS Source, YEAR(OrderDate) AS OrderYear, TotalAmount FROM linked.old_warehouse.sales.orders UNION ALL SELECT 'New' AS Source, YEAR(OrderDate) AS OrderYear, TotalAmount FROM sales.orders ) o GROUP BY o.Source, o.OrderYear HAVING COUNT(DISTINCT o.Source) > 1 -- Shows mismatches only ORDER BY o.OrderYear; Validation criteria for go-live: - Row counts match within 0.1% tolerance - Aggregated values match within 0.5% tolerance - Top 10 Power BI reports render correctly with identical numbers - ETL completion time is equal to or better than the old systemPerformance Optimization for Fabric LakehouseWarehouse performance tuning skills (indexes, partition functions, statistics) don't transfer directly to lakehouse. Instead, learn these Delta Lake optimization techniques:Z-Ordering (Multi-dimensional Clustering)# Optimize by customer_id and order_date for fast point lookups spark.sql(""" OPTIMIZE sales.orders ZORDER BY (customer_id, order_date) """) Use Z-ordering on columns that appear frequently in WHERE clauses and JOIN predicates.Clustering / table optimization patterns (verify current availability)Fabric and Delta Lake optimization capabilities evolve. Before standardising on a specific clustering feature or SQL syntax, validate support in Microsoft Fabric documentation and test it in the target workspace. For continuously-ingested tables, compare available clustering, compaction, and partitioning options using real query patterns rather than assuming one feature is universally available.Vacuum (Clean Up Old Files)# Remove files older than 7 days (default retention) spark.sql("VACUUM sales.orders RETAIN 168 HOURS") Run VACUUM after every major data load to reclaim storage space from old Parquet files.File Compaction# Compact small files into optimal 256MB-1GB files spark.sql("OPTIMIZE sales.orders") This is especially important for streaming or micro-batch ingestion that produces many tiny Parquet files.Migration Timeline (Realistic for a Malaysian Enterprise) Phase Duration Activities Assessment Week 1-2 Inventory tables, map data types, document ETL dependencies POC Week 3-4 Migrate one schema (e.g., Sales), validate with Power BI Core migration Week 5-10 Migrate remaining schemas, rebuild ETL pipelines Parallel run Week 11-14 Both systems running, daily validation Cutover Week 15 Redirect Power BI, decommission old warehouse connections Decommission Week 16 Archive old warehouse, remove old Synapse/SQL Server resources Common Pitfalls (and How to Avoid Them)1. Schema Evolution MismatchTraditional data warehouses have rigid, static schemas. Lakehouses support schema evolution (adding columns, changing types), but unplanned changes break downstream reports. Fix: Use Delta Lake schema enforcement (mergeSchema option) carefully. Add new columns with NULL defaults to avoid breaking existing queries.2. Performance ExpectationsEveryone expects lakehouse SQL endpoints to match Synapse Dedicated Pool performance. For some well-tuned workloads with appropriate file layout, table optimization, and semantic model design, it can perform well. For unoptimized tables (many small files, poor clustering/partitioning, or unsuitable query patterns), it may not. Fix: Run OPTIMIZE and VACUUM as part of your ETL pipeline, not as a separate manual step.3. Data Type Migration IssuesSome SQL Server data types don't have direct Delta Lake equivalents: - MONEY / SMALLMONEY → DECIMAL(19,4) - UNIQUEIDENTIFIER → STRING - HIERARCHYID → No direct equivalent (flatten before migration) - GEOGRAPHY → No direct equivalent (store as WKT string)Fix: Review data type mappings in a spreadsheet before migration. Flag unsupported types for manual handling.4. Security Model ReconstructionWarehouse security (schemas, roles, row-level security) doesn't migrate automatically. Fix: Document the entire security model before starting. Re-implement in Fabric using OneLake data access roles + SQL endpoint permissions + Power BI RLS.5. Underestimating ETL Rewrite EffortSSIS packages with complex control flow (loops, precedence constraints, event handlers) require significant rework in Fabric Data Factory. Fix: Budget 50-100% more time for ETL migration than for data migration.Key TakeawaysFabric lakehouse is a viable migration target when validated workload-by-workload — Microsoft documents lakehouse, OneLake, shortcuts, SQL/Spark access, Direct Lake, and Fabric capacity capabilities, but performance, security design, and endpoint behaviour must be proven against your own data model and reports.Start with a thorough assessment — you cannot migrate what you haven't documented. Inventory every table, ETL dependency, and BI report before touching anything in Fabric.Migrate incrementally, one schema at a time — start with a non-critical schema (e.g., Sales reporting), validate thoroughly with Power BI and business users, then expand. This de-risks the entire project.Learn Delta Lake optimization techniques — Z-ordering, liquid clustering, and VACUUM are your new performance toolkit. Traditional warehouse tuning (indexes, partitions) does not transfer. Allocate training time for your team.Run parallel for at least two business cycles — validate row counts, aggregated values, and report rendering before cutting over. A day-one data discrepancy erodes trust quickly.Budget 50% of total effort for ETL migration — data moves easily; pipeline logic is where the real work lives. SSIS → Fabric Data Factory rewrites require careful test planning.Fabric lakehouse migration is a strategic project that touches data engineering, BI, and governance. I advise Malaysian enterprises on migration roadmaps, capacity planning, and team upskilling. Connect on LinkedIn. --- ## Migrating Legacy .NET Apps to Azure Container Apps: A Step-by-Step Guide URL: https://wenfeng.my/migrating-legacy-net-apps-to-azure-container-apps-a-step-by-step-guide-2/ Author: admin Published: 2026-05-19 23:40:34 Modified: 2026-10-04 15:30:21 ⚡ TL;DRLegacy .NET applications are still common in Malaysian and Southeast Asian enterprises. Many of them run on Windows Server VMs with IIS, scheduled patching, manually managed certificates, and deployment processes that still depend on RDP or Web Deplo...The First Decision: Rehost or Modernise?Migration AssessmentStep-by-Step Migration Path to Azure Container AppsWhat If the App Must Stay on .NET Framework?Production Hardening ChecklistMigrating Legacy .NET Apps to Azure Container Apps: A Step-by-Step GuideLegacy .NET applications are still common in Malaysian and Southeast Asian enterprises. Many of them run on Windows Server VMs with IIS, scheduled patching, manually managed certificates, and deployment processes that still depend on RDP or Web Deploy.Azure Container Apps can be a strong target for modernised .NET applications, but there is an important boundary that must be clear before any migration starts:Azure Container Apps requires Linux-based x86-64 container images. It is not a lift-and-shift target for Windows Server Core containers running full .NET Framework.That means the right migration path depends on the application:If the application can be moved to .NET 8 or later and packaged as a Linux container, Azure Container Apps is a practical serverless container target.If the application must remain on full .NET Framework, IIS, COM components, or Windows-only dependencies, use a Windows-capable hosting option such as Azure App Service for Windows Containers, Azure Kubernetes Service with Windows node pools, or remain on Windows VMs until the application is modernised.This guide focuses on the practical assessment and migration path for moving legacy .NET applications toward Azure Container Apps safely, without pretending every .NET Framework workload can be dropped into Container Apps unchanged.The First Decision: Rehost or Modernise?Before writing a Dockerfile, decide whether the workload is a rehost candidate or a modernisation candidate. Situation Recommended target ASP.NET Core or .NET 8+ application that can run on Linux Azure Container Apps .NET Framework 4.x application requiring IIS and Windows Server Core Azure App Service for Windows Containers, AKS Windows node pools, or Windows VM Application with COM automation, Office Interop, GAC dependencies, or registry-heavy behaviour Windows VM or refactor first Stateless web API with externalised configuration and logging Strong Azure Container Apps candidate In-process session, file-based state, or local scheduler dependencies Refactor state and background work before Container Apps Azure Container Apps is excellent for serverless container operations, HTTP scaling, revision management, managed ingress, Dapr integration, and event-driven workloads. It is not the right first landing zone for an unchanged Windows-only .NET Framework application.Migration Assessment1. Runtime and Operating SystemCheck the target runtime first:Full .NET Framework 4.x requires Windows Server Core when containerised..NET 8 and later can target Linux containers, Windows Nano Server, or Windows Server Core depending on the application and base image.Azure Container Apps requires Linux-based x86-64 container images.If your application is still .NET Framework 4.x, the Azure Container Apps route usually starts with a code migration to .NET 8 or later, not with a direct Windows container deployment.2. Windows-Specific DependenciesLook for dependencies that block Linux containerisation:COM automation or Microsoft Office Interop.Windows registry access.GAC-registered assemblies.IIS modules that do not have a Linux/container equivalent.Windows-only graphics or printing libraries.Server-side WCF implementations.Local Windows services installed beside the web application.Some dependencies can be replaced. Others justify keeping the workload on a Windows-capable platform until the business case supports deeper refactoring.3. State StorageContainer Apps expects stateless, restartable containers. Review:In-process session state.Static variables used as shared state.Files written to local disk.Local cache directories.Scheduled jobs running inside the web server process.Recommended patterns:Store session state in Redis or a database.Store files in Azure Blob Storage or Azure Files where a file share is required.Move background jobs to Azure Container Apps jobs, Azure Functions, or a workflow engine.Send logs to stdout, Application Insights, or Azure Monitor instead of local files.4. Database and IdentityFor database access:Move SQL Server databases to Azure SQL Database or Azure SQL Managed Instance where appropriate.Replace hardcoded connection strings with configuration injected at deployment time.Prefer managed identity for Azure resource access where the target service supports it.Avoid embedding secrets in Docker images, YAML files, or source control.Step-by-Step Migration Path to Azure Container AppsStep 1: Port or Upgrade the ApplicationFor Azure Container Apps, the practical target is normally .NET 8+ running on Linux.Typical work items:Convert project files to SDK-style projects where possible.Replace .NET Framework-only packages with supported .NET packages.Replace System.Drawing.Common usage with cross-platform libraries such as ImageSharp or SkiaSharp where image manipulation is required.Replace Windows Authentication assumptions with Microsoft Entra ID, OpenID Connect, or an application-specific authentication model.Externalise configuration and secrets.Do not skip this step for .NET Framework applications. A Windows Server Core Dockerfile is not sufficient for Azure Container Apps because the platform requires Linux container images.Step 2: Build a Linux Container ImageExample Dockerfile for a modernised ASP.NET Core application:FROM mcr.microsoft.com/dotnet/aspnet:8.0 AS runtime WORKDIR /app EXPOSE 8080 ENV ASPNETCORE_URLS=http://+:8080 FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build WORKDIR /src COPY . . RUN dotnet restore RUN dotnet publish -c Release -o /out FROM runtime AS final WORKDIR /app COPY --from=build /out . ENTRYPOINT ["dotnet", "MyApp.dll"] Build and test locally:docker build -t my-modernised-app:1.0.0 . docker run --rm -p 8080:8080 my-modernised-app:1.0.0 Step 3: Push to Azure Container Registryaz acr create \ --resource-group rg-containerapps \ --name wfengcontainerregistry \ --sku Basic \ --admin-enabled false az acr build \ --registry wfengcontainerregistry \ --image my-modernised-app:1.0.0 \ . az acr build is useful when build agents do not have Docker available locally. For large enterprise applications, use CI/CD with a controlled build agent and repeatable version tags rather than manual image builds.Step 4: Create the Container Apps Environmentaz containerapp env create \ --name wfeng-env \ --resource-group rg-containerapps \ --location malaysiawest \ --logs-destination log-analytics \ --logs-workspace-id <workspace-id> \ --logs-workspace-key <workspace-key> Use workload profiles when you need dedicated compute characteristics, larger resource sizes, or isolation beyond the default consumption profile. Dedicated workload profiles include general-purpose profiles such as D4, D8, and D16. Do not describe these profiles as Windows-only; they are Container Apps workload profile sizes, not proof of Windows container support.Step 5: Deploy the Container Appaz containerapp create \ --name my-modernised-app \ --resource-group rg-containerapps \ --environment wfeng-env \ --image wfengcontainerregistry.azurecr.io/my-modernised-app:1.0.0 \ --target-port 8080 \ --ingress external \ --min-replicas 1 \ --max-replicas 5 For production, configure ACR access using managed identity or another approved identity pattern. Avoid enabling registry admin credentials unless there is a temporary migration reason and a compensating control.Step 6: Configure ScalingContainer Apps supports HTTP scaling and event-driven scaling patterns. For internal applications with predictable idle periods, scale-to-zero can reduce runtime consumption, but it introduces cold starts.az containerapp update \ --name my-modernised-app \ --resource-group rg-containerapps \ --scale-rule-name http-scale \ --scale-rule-type http \ --scale-rule-http-concurrency 100 \ --min-replicas 1 \ --max-replicas 10 Use min-replicas 0 only when cold starts are acceptable. For customer-facing portals or operational systems, keep at least one replica running unless the business has accepted the latency trade-off.Step 7: Configure Storage and SecretsFor persistent files, prefer application-level storage services such as Azure Blob Storage. If the application still needs a shared file system, Container Apps can mount Azure Files, but this should be treated as a transitional design rather than a permanent substitute for application refactoring.For secrets:Store secrets in Key Vault or Container Apps secrets.Inject configuration at deploy time.Do not bake secrets into container images.Rotate credentials after migration if they were previously embedded in web.config or deployment scripts.Step 8: Configure Custom Domain and TLSAzure Container Apps supports custom domains and free managed certificates for eligible publicly reachable domains. Certificate issuance and renewal depend on the domain continuing to meet the managed certificate requirements.A typical pattern is:az containerapp hostname add \ --name my-modernised-app \ --resource-group rg-containerapps \ --hostname app.example.com az containerapp hostname bind \ --name my-modernised-app \ --resource-group rg-containerapps \ --hostname app.example.com \ --environment wfeng-env Validate the current Azure CLI syntax in your build pipeline because Container Apps CLI extensions change over time.What If the App Must Stay on .NET Framework?If the application still requires full .NET Framework and Windows Server Core, do not force it into Azure Container Apps.Recommended interim options:Azure App Service for Windows ContainersGood for IIS-based web applications that can be packaged as Windows containers.Lower operational complexity than AKS.Azure Kubernetes Service with Windows node poolsSuitable when you already operate Kubernetes or need stronger orchestration control.Higher platform operations burden than App Service or Container Apps.Windows Server VM modernisation landing zoneSuitable when dependencies are too coupled for a near-term container migration.Still improve the platform with patching, monitoring, backup, deployment automation, and least-privilege access.Treat these as transition states. The long-term target should usually be .NET 8+ unless the application is near retirement.Production Hardening ChecklistBefore cutover:Confirm the application runs as a Linux container if the target is Azure Container Apps.Verify startup, liveness, and readiness behaviour.Move session state and file state outside the container.Configure Azure Monitor and Application Insights.Configure alerts for restarts, failed revisions, latency, and 5xx responses.Use managed identity where possible.Store secrets outside source code and images.Test scale-out and scale-in behaviour.Test cold-start impact if scale-to-zero is enabled.Validate rollback using Container Apps revisions.Confirm DNS, TLS, WAF, private networking, and outbound firewall requirements.Document the fallback path if the modernised container fails production validation.Cost ConsiderationsContainer Apps can reduce cost for variable or intermittent workloads because it can scale based on demand, including scale-to-zero for suitable applications. However, savings are not guaranteed.Cost depends on:Replica minimums.CPU and memory allocation.Workload profile type.Request volume and execution time.Logging volume.External dependencies such as Azure SQL, Redis, Storage, Key Vault, and Application Insights.Whether the app has been modernised enough to run efficiently as a Linux container.For steady 24/7 workloads, a VM, App Service plan, or AKS node pool may be cheaper. Use Azure Pricing Calculator with actual CPU, memory, traffic, and availability assumptions before committing to a migration business case.Key TakeawaysAzure Container Apps is not a direct Windows Server Core hosting target. It requires Linux-based x86-64 container images.Full .NET Framework usually means Windows Server Core. If the app cannot move to .NET 8+ or Linux, choose a Windows-capable Azure hosting option first.State externalisation is mandatory. Session state, files, cache, configuration, and background jobs must be moved out of the container or redesigned.Container Apps is a strong target after modernisation. Once the application runs as a Linux container, Container Apps provides managed ingress, scaling, revisions, Dapr integration, and simpler operations than a self-managed Kubernetes cluster.Do not sell cost savings without sizing. Variable workloads may benefit, but fixed workloads need a calculator-backed comparison.Legacy .NET migration is not a one-size-fits-all exercise. If you have a .NET Framework application on Windows Server and need a realistic Azure migration path, start with the dependency assessment first — then choose the platform that matches the technical facts, not the platform that sounds newest. --- ## Azure Disaster Recovery Architecture: Active-Active vs Active-Passive for Malaysian Enterprises URL: https://wenfeng.my/azure-disaster-recovery-architecture-active-active-vs-active-passive-for-malaysian-enterprises-2/ Author: admin Published: 2026-05-18 23:30:24 Modified: 2026-10-04 15:30:21 ⚡ TL;DRDisaster recovery on Azure requires a fundamental architectural decision: should you run your workload in one region with a standby (active-passive) or in multiple regions simultaneously (active-active)? For Malaysian enterprises, this choice is comp...Understanding the Regional Design ConstraintActive-Passive (Cold/Warm Standby) ArchitectureActive-Active (Geo-Redundant) ArchitectureMalaysian Enterprise ConsiderationsDecision Framework: Which Architecture for Which Scenario?Azure Disaster Recovery Architecture: Active-Active vs Active-Passive for Malaysian EnterprisesDisaster recovery on Azure requires a fundamental architectural decision: should you run your workload in one region with a standby (active-passive) or in multiple regions simultaneously (active-active)? For Malaysian enterprises, this choice is complicated by data residency requirements in Malaysia West, the available regional pair architecture, and the cost sensitivity typical of the local market.This article examines both architectures in detail, provides concrete implementation guidance, and helps you decide which is right for your specific RTO, RPO, and budget constraints.Understanding the Regional Design ConstraintAzure offers Malaysia West as a primary region, but Microsoft lists Malaysia West with no Azure paired region. Southeast Asia (Singapore) is still a common nearby DR candidate, but it should not be described as Malaysia West's formal paired region.This distinction matters because:Do not assume paired-region behavior — Microsoft documents benefits for formal Azure region pairs, such as staggered platform updates and service-specific geo-redundancy patterns, but those assumptions should be validated per service when the selected secondary is not a listed pair.Data residency is not automatic — Malaysia West to Southeast Asia is a cross-border DR design. Personal data replicated to Singapore may trigger Malaysia PDPA cross-border transfer obligations. You must classify the data, check contractual controls, and confirm whether consent or another permitted transfer basis applies.Latency must be measured for the workload — Malaysia-to-Singapore latency is often acceptable for web failover, but it should be benchmarked for each application. Avoid synchronous database write patterns across regions unless the application has been explicitly designed and tested for that latency.In short: treat Southeast Asia as a practical secondary region candidate, not as an automatic paired-region target. The design decision must be based on service support, compliance, latency, and recovery objectives.Active-Passive (Cold/Warm Standby) ArchitectureIn an active-passive architecture, one Azure region handles all production traffic. The DR region has infrastructure deployed and ready, but it remains idle — or running at minimal capacity — until failover is triggered.How It Works Malaysia West Southeast Asia (Singapore) ┌─────────────────┐ ┌──────────────────────┐ │ Active (100%) │ │ Passive (Standby) │ │ │ │ │ │ App Gateway │ │ App Gateway (off) │ │ App Service │──────>│ App Service (cold) │ │ SQL Database │──────>│ SQL Database (DR) │ │ Blob Storage │──────>│ Blob Storage (GRS) │ └─────────────────┘ └──────────────────────┘ Traffic flows to Malaysia West normally. Data is replicated to Southeast Asia asynchronously. During a disaster, you initiate failover: traffic is redirected, the standby resources are activated, and the application resumes from the DR region.RTO and RPO Expectations Component RTO RPO Azure SQL Database (Geo-restore) Minutes to hours Around 1 hour, depending on backup timing Azure SQL Database (Active Geo-replication / failover groups) Typically less than 60 seconds for database failover, excluding dependent application components Asynchronous replication; data loss can be greater than zero if recent changes have not replicated Azure VMs (Azure Site Recovery) Depends on recovery plan, VM boot time, DNS/routing, and validation steps Depends on configured replication and latest available recovery point Azure Blob/Storage (GRS or RA-GRS) Depends on account failover and application design Geo-replication is asynchronous; Microsoft documents Block Blob geo priority replication RPO as less than or equal to 15 minutes where that feature applies App Service (pre-provisioned DR deployment) Depends on traffic manager/front-door configuration and app warm-up Varies by application state and deployment model The ranges are wide because RTO and RPO depend on the specific replication technologies you choose. Azure SQL failover groups and active geo-replication can provide fast database failover, but they still use asynchronous replication and do not remove the need to design the application, identity, storage, networking, and DNS layers for recovery.Cost Modeling: Active-Passive for a Web ApplicationA typical three-tier web application (web tier, API tier, database) in an active-passive DR setup: Resource Primary Region DR Region (Passive) App Service / API compute Production sizing Stopped, scaled down, or pre-provisioned depending on RTO SQL Database Production sizing Geo-secondary or failover group secondary sized to recovery/read needs Azure Front Door / routing Global service Same global instance, with secondary origin configured Azure Site Recovery Protected VM scope Replication, storage, and recovery-plan configuration Storage Primary redundancy choice GRS/RA-GRS/GZRS or explicit replication pattern, subject to data residency Illustrative active-passive model: the DR footprint is usually materially cheaper than a fully active second region because some compute can be stopped, scaled down, or kept as infrastructure-as-code until failover. Treat any percentage as workload-specific rather than universal.For pricing, validate the actual SKU, region, licensing benefit, backup retention, data transfer, and support plan in Azure Pricing Calculator before using the figures in a proposal or budget.Implementation: Active-Passive with Azure Site RecoveryAzure Site Recovery (ASR) orchestrates the replication, failover, and failback of Azure VMs between regions.# Implementation note: # Azure Site Recovery for Azure-to-Azure VM replication is usually configured # from the Azure portal or automation generated after the Recovery Services # vault, fabric, protection container, policy, network mapping, and protected # item relationships exist. Do not treat the following as a copy-paste script. # Verify the Site Recovery extension is available in the installed Azure CLI. az extension add --name site-recovery az site-recovery --help # Recommended operational pattern: # 1. Create or select a Recovery Services vault in the DR region. # 2. Enable replication for each VM and confirm target network, subnet, # VM size, disk type, and availability-zone settings. # 3. Build a recovery plan grouping app, API, and database tiers. # 4. Run test failover into an isolated VNet at least quarterly. # 5. Record actual RTO/RPO from the test, not just design targets. Implementation: Active-Passive with Azure SQL Geo-ReplicationFor a better RPO than ASR, configure Active Geo-Replication on Azure SQL Database:-- On primary database in Malaysia West ALTER DATABASE [app-db] ADD SECONDARY ON SERVER [sql-srv-southeastasia] WITH (ALLOW_CONNECTIONS = ALL, SECONDARY_TYPE = GEO, SERVICE_OBJECTIVE = GP_Gen5_2); -- During failover, promote the secondary to primary -- (run this on the secondary server) ALTER DATABASE [app-db] FAILOVER; The SQL driver can also handle automatic failover redirection. Use ApplicationIntent=ReadOnly in your connection string to route read-only queries to the readable secondary.Active-Active (Geo-Redundant) ArchitectureIn an active-active architecture, both regions handle traffic simultaneously. Traffic is load-balanced across regions using Azure Traffic Manager or Azure Front Door. Each region runs identical infrastructure and serves user requests.How It Works ┌──────────────────────┐ │ Azure Front Door │ │ (Global Load Balancer)│ └──────┬───────────────┘ │ ┌────────────┴────────────┐ │ │ Malaysia West Southeast Asia ┌──────────────────┐ ┌──────────────────────┐ │ App Service │ │ App Service │ │ (50% traffic) │ │ (50% traffic) │ │ │ │ │ │ SQL (Read-Write) │<─────>│ SQL (Read-Only) │ │ Blob (LRS) │ │ Blob (LRS) │ └──────────────────┘ └──────────────────────┘ For active-active to work, the application must be stateless (session state must go to Redis or Cosmos DB), and the database must support multi-region writes or accept that writes happen in one region and are replicated to the other.RTO and RPO Expectations Component RTO RPO Azure Front Door routing Often fast when origins and health probes are already configured N/A Azure SQL active geo-replication / failover group Microsoft documents database disaster-recovery RTO as typically less than 60 seconds for customer-managed failover, excluding dependent components Asynchronous replication; possible data loss for unreplicated recent writes Application state Near-zero only if the application is stateless or uses a region-aware state store Depends on state-store replication model DNS propagation Avoided if Azure Front Door or another global routing layer remains the stable entry point N/A Active-active can reduce infrastructure recovery time because compute is already online in the secondary region. However, database failover, replication lag, identity, secrets, dependent services, and application consistency still determine the real business RTO/RPO.Cost Modeling: Active-Active for the Same Web Application Resource Primary Region DR Region (Active) App Service / API compute Production capacity Production or near-production capacity SQL Database Primary database Secondary database / failover group replica sized for recovery and read traffic Azure Front Door / routing Global service Same global instance with both origins active or priority-routed Redis / application state Primary state tier Region-aware state design or replicated/cache-warm strategy Storage Primary storage pattern Secondary storage pattern selected for consistency, latency, and compliance Illustrative active-active model: expect a materially higher monthly cost than active-passive because compute, observability, security, and operational testing run in both regions. The premium is workload-specific and should be priced in Azure Pricing Calculator before committing to a budget.The extra cost comes from running more resources continuously, not from a fixed universal percentage.Implementation: Active-Active with Azure Front Door and SQL Auto-Failover# Azure Front Door with priority routing (primary, then secondary) az afd profile create \ --resource-group rg-global \ --profile-name afd-wenfeng \ --sku Premium_AzureFrontDoor \ --location global az afd endpoint create \ --resource-group rg-global \ --profile-name afd-wenfeng \ --endpoint-name api-global \ --enabled-state Enabled az afd origin-group create \ --resource-group rg-global \ --profile-name afd-wenfeng \ --origin-group-name origins-southeastasia \ --probe-request-type GET \ --probe-protocol Http \ --probe-interval-in-seconds 30 \ --probe-path /health # Add Malaysia West as primary origin az afd origin create \ --resource-group rg-global \ --profile-name afd-wenfeng \ --origin-group-name origins-southeastasia \ --origin-name app-malaysiawest \ --host-name app-wenfeng-malaysiawest.azurewebsites.net \ --priority 1 # Primary # Add Southeast Asia as secondary origin az afd origin create \ --resource-group rg-global \ --profile-name afd-wenfeng \ --origin-group-name origins-southeastasia \ --origin-name app-southeastasia \ --host-name app-wenfeng-southeastasia.azurewebsites.net \ --priority 2 # Secondary (hot standby in active-passive, active in active-active) -- SQL Auto-Failover Group for database-level failover -- Managed via Azure CLI: az sql failover-group create \ --resource-group rg-db \ --server sql-malaysiawest \ --partner-server sql-southeastasia \ --name fg-app-db \ --failover-policy Automatic \ --grace-period 1 \ --add-db app-db -- Note: With --grace-period 1 (1 hour minimum), automatic failover -- takes at least 1 hour. Use forced failover (with data loss) for faster failover. -- The failover group provides a single read-write endpoint -- and a read-only listener endpoint: -- Read-Write: fg-app-db.database.windows.net -- Read-Only: fg-app-db.secondary.database.windows.net Malaysian Enterprise Considerations1. Data Residency and PDPAMalaysia's PDPA 2010 restricts transfers of personal data to places outside Malaysia unless an allowed transfer basis applies, such as consent, contractual necessity, or taking reasonable precautions and due diligence to ensure the data will not be processed in a way that would contravene the PDPA. This has direct implications:Active-passive with ASR — if data is replicated to Southeast Asia (Singapore) for DR purposes, you may be transferring personal data out of Malaysia. Mitigate by: (a) identifying which data is personal data and excluding it from replication where feasible, (b) confirming the lawful transfer basis with legal counsel, and (c) using appropriate Microsoft contractual, security, and organizational controls.Active-active with geo-replication — the same concern applies but is magnified because data is actively written in both regions. A stricter data classification exercise is required.Mitigation strategies:Use Azure Policy to tag resources containing personal data and exclude them from regional replication.Implement a data classification framework before designing your DR architecture — separate personal data from non-personal data at the application level.Consider keeping personal data in a separate database in Malaysia West only, while replicating non-personal application data for DR.2. Network LatencyMalaysia West to Southeast Asia (Singapore) latency should be measured from the actual application network path. For architecture planning, assume it can materially affect synchronous write paths and validate with Azure Network Watcher, application telemetry, or direct synthetic tests. Workload Type Cross-region latency impact Static web page serving Usually low if content is cached or served from the nearest edge API requests with multiple back-end round trips Can become noticeable; measure end-to-end transaction latency Real-time collaboration Potentially problematic without region-aware design Database synchronous replication Avoid unless the platform and application explicitly support the latency profile Video/voice conferencing Usually better handled with edge/media services rather than cross-region application round trips For active-active architectures, replication lag is a critical metric. Azure SQL active geo-replication is asynchronous, so design for possible lag and conflict/consistency behavior rather than assuming near-synchronous writes.3. Cost SensitivityMany Malaysian SMEs are cost-sensitive, so the total cost delta between active-passive and active-active is significant and should be modeled explicitly:Active-passive (warm standby): typically lower because some DR compute can be scaled down or kept inactiveActive-active: typically higher because both regions run production-grade capacity, monitoring, security, and operational processesAnnual difference: calculate from the actual workload, region, licensing, support, network egress, backup, and operational staffing assumptionsFor many SMEs, the active-active premium is only justified when the business impact of downtime clearly exceeds the additional run cost and operational complexity.4. Operational CapabilityActive-active requires more operational maturity. You must:Maintain identical infrastructure in two regions.Test and validate traffic routing regularly.Manage application state to be region-agnostic.Have staff available to respond to regional incidents.Monitor replication lag and DNS propagation.Few Malaysian SMEs have dedicated DR teams. Azure Site Recovery's one-click failover test capability is a significant advantage for teams with limited bandwidth.Decision Framework: Which Architecture for Which Scenario?Active-Passive Is the Right Choice When:Your RTO requirement is 15 minutes or more. Most Malaysian business applications can tolerate 30-60 minutes of downtime. Active-passive delivers this reliably.Your budget is constrained. Active-passive costs 28-40% of the primary region, not 100%. The savings are real.Your IT team is small (1-3 people). Active-passive is simpler to operate. You can test failover quarterly without deep architectural knowledge.Your workload is not latency-sensitive across regions. If users are primarily in Malaysia, having the DR region in Singapore is fine for failover scenarios.You need PDPA compliance. You can keep personal data in Malaysia West only and replicate only non-sensitive data to the DR region.Active-Active Is the Right Choice When:Your RTO requirement is under 1 hour for database failover. With forced failover (potential data loss), active-active can achieve sub-minute infrastructure failover. For automatic failover with data safety, expect at least 1 hour due to the grace period.Your workload has users in multiple regions. If you serve customers in Malaysia, Singapore, and Indonesia simultaneously, active-active serves them from the nearest region.Your SLA requires very high availability across regional failures. Active-active can support stronger regional resilience targets, but the actual SLA depends on the complete architecture, service SKUs, application design, and operational process.Your budget supports the additional run cost. If the business impact of downtime is high enough, the added active-active cost may be justified by avoided outage impact.Your team has DevOps capability. You need the operational maturity to maintain symmetric deployments across two regions.The Compromise: Active-Passive with Readable DRA hybrid approach that many enterprises overlook: configure active-passive but keep the DR database readable. Users who explicitly opt in (via a "read-only mode" link or during maintenance windows) can query the DR region without impacting production. This gives you:The cost structure of active-passive, subject to the actual secondary database and read workload sizingThe ability to run read-heavy reporting queries against the DR database without impacting productionWarmer standby — if failover is needed, the database is already online and can be promoted in minutesFailover Testing PlaybookRegardless of which architecture you choose, test your failover at least quarterly. Here is a practical playbook:Pre-Test (Week Before)Notify stakeholders: "We are conducting a scheduled DR test on [date]. Expected [n] hours of read-only mode."Take a full backup of critical databases (as a safety net).Review the last test's findings and confirm all remediation items are complete.Test DayActive-passive test:# Initiate planned failover via ASR # Trigger planned failover via ARM REST API (no direct CLI command for this operation) az rest --method POST \ --uri "https://management.azure.com/subscriptions/{subscription-id}/resourceGroups/rg-dr/providers/Microsoft.RecoveryServices/vaults/asr-vault-southeastasia/replicationRecoveryPlans/dr-plan-web-app/plannedFailover?api-version=2024-10-01" \ --body '{ "properties": { "failoverDirection": "PrimaryToRecovery" } } # Verify application is accessible from DR region curl -f https://dr-app-wenfeng.azurewebsites.net/health # Promote SQL secondary (for geo-replication setups) az sql failover-group set-primary \ --resource-group rg-db \ --server sql-southeastasia \ --name fg-app-db Run application smoke tests against the DR endpoint.Measure actual RTO and RPO against your SLAs.Post-TestFail back to the primary region.Document actual RTO/RPO achieved vs. targets.Identify gaps: "Restored VM had outdated SSL certificate," "DNS TTL caused 10-minute propagation delay."Fix gaps before the next test.Key TakeawaysActive-passive is the default for many Malaysian enterprises — it can provide practical recovery objectives at materially lower cost than fully active-active, provided failover is tested.Active-active is for high-availability requirements — it can reduce recovery time, but the cost premium and operational burden are workload-specific.Data residency complicates active-active — Malaysia's PDPA requires careful evaluation of what data crosses the border to Singapore.Test failover quarterly without exception — a DR plan never tested is not a DR plan. Use Azure Site Recovery's test failover in an isolated VNet.Azure Site Recovery simplifies active-passive — automated replication, one-click failover testing, and built-in recovery plans make it accessible to teams without dedicated DR expertise.The hybrid approach (active-passive with readable DR database) is a practical middle ground — warm standby cost structure with operational flexibility.DR architecture is not a one-size-fits-all decision. I review and design disaster recovery strategies for Malaysian enterprises — covering Azure Site Recovery, active-passive, active-active, and region-pair failover patterns. Message me on LinkedIn. --- ## Designing DR Between Malaysia West and Southeast Asia on Azure URL: https://wenfeng.my/mw-dr-southeast-asia/ Author: admin Published: 2026-05-17 22:52:17 Modified: 2026-10-04 15:30:31 ⚡ TL;DRMalaysia West changes the Azure architecture conversation for Malaysian enterprises. It gives organisations a stronger local-region option for workload placement, latency-sensitive systems, and data residency discussions.Practical Executive SummaryArchitecture ContextMalaysia West-Specific ConsiderationsDesign RecommendationsRisks and ConstraintsDesigning DR Between Malaysia West and Southeast Asia on AzureMalaysia West DR availability warning: Do not assume Azure Site Recovery, Recovery Services vault placement, Azure Backup, Azure SQL failover groups, Storage account failover, AKS, App Service, Key Vault, Private Link, Azure Firewall, VPN Gateway, ExpressRoute Gateway, Bastion, Front Door, Traffic Manager, Monitor, Sentinel, or zone-redundant options are available or feature-equivalent in Malaysia West until verified against current Microsoft regional availability and service documentation. Azure regional capabilities change over time; validate every service, SKU, quota, replication feature, and failover dependency before architecture sign-off.Malaysia West changes the Azure architecture conversation for Malaysian enterprises. It gives organisations a stronger local-region option for workload placement, latency-sensitive systems, and data residency discussions. But it does not remove the need for a deliberate disaster recovery design.Disaster recovery is not automatically solved by deploying into a newer local region. It is solved by understanding business impact, defining recovery targets, validating service capabilities, building recovery environments, and testing the actual failover path. For Malaysia-based organisations, a common question will be whether Azure Southeast Asia, located in Singapore, should be used as a recovery region for Malaysia West workloads.The short answer is: Southeast Asia can be evaluated as a customer-selected DR region, but it should not be described as the official paired region for Malaysia West unless Microsoft documentation explicitly confirms that at the time of design and publication. The current architecture position should be conservative: treat Malaysia West-to-Southeast Asia DR as an intentional cross-region design that requires service-by-service validation.For broader regional planning context, read the Malaysia West Azure Region architecture planning guide before finalising your disaster recovery strategy.Practical Executive SummaryFor Malaysian IT leaders, the key decision is not simply “Malaysia West or Southeast Asia?” The better question is: which workloads require regional recovery, what recovery objectives must be met, and which Azure services can support the required pattern in both regions?My recommendation is to use the following position as the baseline:Use Malaysia West as a primary region where it improves local workload placement, compliance posture, user experience, or commercial alignment.Evaluate Southeast Asia as a secondary recovery region only when business, compliance, network, service availability, and operational requirements support cross-region recovery.Do not assume Southeast Asia is Malaysia West’s official Azure paired region. Validate the current Microsoft Learn Azure regions list before making any paired-region statement.Validate every service and feature used for DR. Base service availability is not enough; replication, restore, failover, private networking, monitoring, backup, and zone-redundancy features may have different support boundaries.Define RTO and RPO per workload tier. Do not force all systems into one DR pattern.Test failover and failback. A DR design that has not been tested is only an architecture assumption.This matters because different workloads have very different recovery characteristics. A line-of-business application running on virtual machines may be a candidate for Azure Site Recovery if the region, vault, OS, disk, and network requirements are supported. A cloud-native application may be better recovered through infrastructure-as-code, container image replication, database replication, and DNS cutover. A reporting workload may only need backup and restore. A regulated system may not be allowed to replicate data outside Malaysia without compliance approval.Architecture ContextAzure disaster recovery architecture should start from business impact, not from tooling. Tools such as backup, replication, geo-restore, failover groups, traffic routing, and infrastructure automation are implementation mechanisms. They do not replace the need to classify workloads and define recovery objectives.A practical Azure DR model normally includes:Application tiering: Classify workloads by business criticality, dependency complexity, regulatory sensitivity, and acceptable downtime.Recovery objectives: Define recovery time objective (RTO) and recovery point objective (RPO) per workload tier.Recovery region selection: Choose a recovery location based on Microsoft regional availability, compliance acceptance, network path, operational readiness, and service capability.Data protection design: Decide how databases, files, object storage, VM disks, secrets, configurations, and logs will be protected and restored.Network failover design: Plan routing, firewall policy, private endpoints, private DNS, public ingress, outbound egress, and hybrid connectivity for both normal and recovery states.Identity and access design: Ensure Microsoft Entra ID, managed identities, Key Vault, RBAC, emergency access, and privileged operations are included in the runbook.Monitoring and evidence: Retain logs, backup reports, replication health, test results, and incident records.Failback strategy: Define how systems return to Malaysia West after recovery, including data reconciliation and business approval.For Malaysia West, the design should also align with the landing-zone foundation. A recovery environment is not just a second subscription with backups. It needs governance, policy, network segmentation, security monitoring, naming standards, cost management, and operational ownership. The Azure Landing Zone Design for Malaysia West article covers the platform foundation that should exist before DR becomes a production dependency.Malaysia West-Specific ConsiderationsMalaysia West introduces a local Azure region option for Malaysian workloads, but architects must avoid overclaiming regional capabilities. A region-level listing does not automatically mean every required service, SKU, availability zone feature, backup capability, replication feature, private endpoint capability, or DR function is available for a specific workload.The most important Malaysia West-specific considerations are:Region-pair status: Validate the Microsoft Learn Azure regions list before implementation. If Malaysia West is shown with no paired region, do not treat Southeast Asia as its official pair.Southeast Asia role: Southeast Asia should be positioned as a customer-selected recovery region, not an assumed Microsoft-designated pair for Malaysia West.Service availability: Check Azure Products by Region and service-specific Microsoft Learn documentation for every required service.Feature availability: Confirm the exact DR feature, not only the base service. For example, database availability does not automatically prove failover group support for the required configuration.Availability zones: Region-level zone support does not mean every service or SKU supports zone-redundant deployment. Validate zone capability by service.Data residency: Replication from Malaysia to Singapore/Southeast Asia may have regulatory, contractual, or internal policy implications. Treat this as a customer legal and compliance decision, not an architecture assumption.Network path: Validate VPN, ExpressRoute, private connectivity, latency, bandwidth, routing, DNS, and firewall behaviour. Do not assume provider availability or performance.Operational coverage: Ensure the operations team can support incidents, failover approval, recovery validation, communications, and failback across both regions.The design should be evidence-based. Before architecture sign-off, produce a service validation table that lists every workload component, the primary region requirement, the recovery region requirement, the Microsoft evidence source, and the decision owner.Design Recommendations1. Use workload tiers instead of one DR patternDo not apply the same recovery pattern to every application. A more practical model is:Tier 0 / Critical: Identity, network, security operations, core data platforms, and systems that other workloads depend on. These require the most disciplined recovery design and testing.Tier 1 / Business critical: Revenue, operations, customer, or regulatory workloads with defined RTO/RPO and active recovery procedures.Tier 2 / Important: Departmental systems or internal platforms where longer restoration is acceptable.Tier 3 / Recoverable: Systems that can tolerate backup-and-restore or rebuild from infrastructure-as-code.This avoids overengineering low-criticality systems and underprotecting critical dependencies.2. Select the DR pattern per workloadThe main patterns for Malaysia West-to-Southeast Asia DR are:Backup and restore: Lowest operational complexity. Suitable for workloads with acceptable downtime and data loss windows. Requires tested restore procedures.Pilot light: Minimal recovery infrastructure exists in Southeast Asia. Infrastructure-as-code, backups, images, and configuration are ready to activate. Good for controlled cost with moderate recovery expectations.Warm standby: A smaller recovery environment runs continuously and can scale during failover. Better RTO than pilot light, but higher cost and operational effort.Active-passive or active-active application design: Suitable only where the application and data platforms support it. This is the highest complexity model and requires careful consistency, traffic management, and testing.Azure Site Recovery for IaaS: A candidate pattern for supported VM workloads, subject to current Azure Site Recovery support matrix, vault region support, VM SKU, disk, OS, network, and region capability validation.Do not choose the pattern based only on technology preference. Choose it based on business criticality, data architecture, application dependency, supportability, and evidence that the required Azure services are supported.3. Design the recovery landing zone as a real operating environmentA recovery subscription should not be an empty placeholder. At minimum, define:Management groups, subscriptions, Azure Policy, RBAC, tagging, and budgets.Recovery hub or regional network topology.Firewall, NSG, route table, DDoS, and egress controls where required.Private DNS zones and Private Link behaviour for recovered services.Key Vault, certificates, managed identities, and secrets recovery.Log Analytics, Azure Monitor, Defender for Cloud, Sentinel integration if used, and alert routing.Backup vaults, recovery vaults, storage accounts, and repository locations where supported.Runbooks, automation accounts, deployment pipelines, and break-glass access.A DR region that is not governed and monitored will become a security and operations blind spot.4. Treat DNS and traffic cutover as first-class design itemsMany DR plans fail because the application can be recovered but traffic cannot be shifted cleanly. For each application, define:Public DNS cutover approach and time-to-live settings.Private DNS changes for internal workloads.Load balancing or traffic routing options, such as Azure Front Door, Traffic Manager, or external DNS, only where supported and validated.Certificate availability and renewal process in the recovery region.Inbound and outbound firewall rules.Third-party allowlists that may reference fixed IP addresses.User access, conditional access, and authentication dependencies.For network topology decisions, see Hub-and-Spoke vs Azure Virtual WAN for Malaysia-Based Enterprises. DR should be considered when choosing between centralised hubs, Virtual WAN, regional hubs, and hybrid connectivity patterns.5. Test failover without damaging productionA DR design must include test failover. The test should validate more than whether a VM boots. It should confirm:Application dependency order.Database consistency and recovery point.DNS, routing, firewall, and private endpoint behaviour.Secrets and certificates.Monitoring and alerting in recovery mode.User acceptance testing for business-critical functions.Security evidence and audit logs.Rollback or failback procedure.The test schedule should be agreed with business owners. Evidence should be retained because resilience is both an engineering and governance requirement.Risks and ConstraintsRisk: Incorrect paired-region assumptionImpact: The organisation may design governance, compliance, replication, or failover around a relationship that Microsoft does not document for Malaysia West.Recommendation: Validate current Azure region-pair documentation and state Southeast Asia as a customer-selected DR region unless official documentation says otherwise.Risk: Service availability gapsImpact: A design may depend on a feature that is unavailable or limited in Malaysia West, Southeast Asia, or the specific SKU required.Recommendation: Validate Products by Region and service-specific support matrices before approval.Risk: Compliance and data residency conflictImpact: Replicating data outside Malaysia may breach internal policy, contract terms, or regulatory expectations.Recommendation: Involve compliance and legal stakeholders before approving cross-region data replication.Risk: Unachievable RTO/RPOImpact: Business stakeholders may expect recovery times that the architecture cannot deliver.Recommendation: Define measurable RTO/RPO by workload, then test and document actual recovery results.Risk: Network and DNS failover failureImpact: Systems may be technically recovered but inaccessible to users or dependent systems.Recommendation: Include DNS, routing, firewall, hybrid connectivity, and certificate processes in the runbook.Risk: Recovery environment driftImpact: The recovery region becomes inconsistent with production, causing failover failure during an incident.Recommendation: Use infrastructure-as-code, policy, configuration management, and regular test failovers.Malaysia West-to-Southeast Asia DR Decision ChecklistUse this checklist before approving the architecture:Region facts: Confirm Malaysia West and Southeast Asia details in the current Microsoft Learn Azure regions list.Region-pair status: Confirm whether Malaysia West has an official paired region. Do not assume Southeast Asia is the pair.Service availability: Confirm each required Azure service in both regions.DR feature support: Confirm replication, backup, restore, failover, vault placement, and account failover features service by service.Availability zones: Confirm zone support at the service and SKU level, not just region level.RTO/RPO: Document target and tested recovery results for each workload tier.Data classification: Identify data that may replicate to Southeast Asia and obtain compliance approval.Network: Validate IP addressing, routing, firewall, VPN, ExpressRoute, Private Link, and private DNS.Identity and secrets: Confirm managed identities, Key Vault, certificates, RBAC, and emergency access.Operations: Define incident authority, runbooks, communications, test schedule, evidence retention, and failback.Cost model: Separate standby compute, storage, replication, backup, data transfer, monitoring, security, licensing, and testing cost.Support model: Confirm who owns failover decisions during business hours, after hours, and during regional incidents.DR Pattern Decision TableBackup and restore — Best for non-critical or lower-tier workloads. Lower cost, higher RTO/RPO, requires strong restore testing.Pilot light — Best when infrastructure can be rebuilt quickly and data is protected. Moderate complexity, controlled standby cost.Warm standby — Best for important workloads requiring faster recovery. Higher operating cost and stronger configuration management needed.Active-passive — Best for critical applications designed for regional failover. Requires application, database, DNS, and operations maturity.Active-active — Best only where business value justifies cost and complexity. Requires strong consistency design, traffic management, and failure-domain engineering.Azure Site Recovery for IaaS — Useful candidate for supported VM workloads. Must validate region, vault, OS, disk, VM SKU, network, and test failover support before committing.Related ReadingMalaysia West Azure Region: Architecture Planning Guide for EnterprisesAzure Landing Zone Design for Malaysia WestHub-and-Spoke vs Azure Virtual WAN for Malaysia-Based EnterprisesAzure Backup vs Veeam on Azure: Which DR Strategy for Malaysian SMEs?Azure Local for Malaysian Enterprises: When Hybrid Beats Cloud-OnlyCTA: Azure Architecture ReviewAzure architecture review CTA (architecture-cta)Planning disaster recovery for Malaysia West Azure workloads? I can review your RTO/RPO requirements, validate Malaysia West and Southeast Asia service availability, map workload dependencies, assess data residency risk, and produce a practical Azure DR architecture and failover runbook before you build.If you are evaluating Malaysia West as a primary Azure region, the best next step is a focused architecture review. The output should not be a generic DR diagram. It should be a workload-by-workload decision pack covering region evidence, service availability, recovery pattern, data protection, network failover, identity and secrets, operational runbooks, test schedule, and cost drivers.Closing ViewMalaysia West is strategically important for Malaysian Azure adoption, but resilience still requires disciplined architecture. Southeast Asia may be a practical recovery location for some organisations, especially where existing Singapore-based operations, skills, network patterns, or platforms already exist. However, it should be treated as a customer-selected DR region unless Microsoft documentation explicitly identifies it as Malaysia West’s official pair.The safest architecture approach is conservative and evidence-based: validate the region facts, verify every service and DR feature, involve compliance early, define realistic RTO/RPO, and test the recovery path before the business depends on it. --- ## Hub-and-Spoke vs Azure Virtual WAN for Malaysia-Based Enterprises URL: https://wenfeng.my/mw-hub-spoke-vwan/ Author: admin Published: 2026-05-14 22:51:07 Modified: 2026-10-04 15:30:32 ⚡ TL;DRMalaysia West changes an important planning question for Malaysian organisations: cloud network architecture is no longer only about extending to the nearest established regional hub.Practical executive summaryArchitecture context: what hub-and-spoke means on AzureArchitecture context: what Azure Virtual WAN providesMalaysia West-specific considerationsDesign recommendation 1: use hub-and-spoke when control is the priorityHub-and-Spoke vs Azure Virtual WAN for Malaysia-Based EnterprisesMalaysia West changes an important planning question for Malaysian organisations: cloud network architecture is no longer only about extending to the nearest established regional hub. It is now also about deciding how a local Azure region should fit into the enterprise network foundation, branch connectivity model, security inspection path, and disaster recovery design.For most enterprises, the decision is not simply “hub-and-spoke or Azure Virtual WAN?”. The better question is: which network operating model gives the organisation enough control, scale, security, and maintainability for the next three to five years?This article provides a practical architecture decision guide for Malaysia-based organisations evaluating a traditional Azure hub-and-spoke topology against Azure Virtual WAN. It is written for CIOs, infrastructure heads, network architects, platform teams, and enterprise architects planning Azure landing zones where Malaysia West may become the primary region.Malaysia West availability warning: Do not assume Azure Virtual WAN hubs, secured virtual hub, Azure Firewall, VPN Gateway, ExpressRoute Gateway, Bastion, Application Gateway/WAF, NAT Gateway, Private Link, specific NVA marketplace images, or zone-redundant options are available in Malaysia West until verified against current Microsoft regional availability and service documentation. Azure regional capabilities change over time; validate every service, SKU, quota, and feature before architecture sign-off.Practical executive summaryMy recommendation is straightforward:Start with hub-and-spoke when the organisation needs maximum design control, a simple regional topology, mature centralised inspection, direct route-table management, and close alignment with standard Azure landing-zone patterns.Consider Azure Virtual WAN when the organisation has many branches, factories, campuses, VPN sites, SD-WAN integrations, remote users, multiple ExpressRoute or VPN connectivity paths, or multi-region routing requirements where Microsoft-managed hub-scale networking can reduce operational complexity.Use coexistence where justified when an existing hub-and-spoke design already works but a branch-heavy or regional connectivity domain would benefit from Virtual WAN.Do not choose based on trend preference. Choose based on topology scale, routing model, security inspection requirements, operational skills, cost drivers, and Malaysia West service availability validation.For a single Malaysia West landing zone with a small number of workload spokes and controlled egress, hub-and-spoke is usually the safer starting point. It is explicit, understandable, and easier to govern through landing-zone design standards. For an enterprise with dozens or hundreds of sites, regional expansion, remote access, and mixed VPN/ExpressRoute connectivity, Azure Virtual WAN may become the more practical operating model—provided the required capabilities are available in the selected Azure regions.For broader regional planning, read the pillar article: Malaysia West Azure Region: Architecture Planning Guide for Enterprises. For landing-zone foundation design, see Azure Landing Zone Design for Malaysia West.Architecture context: what hub-and-spoke means on AzureA hub-and-spoke architecture places shared connectivity, routing, inspection, and platform services into a central hub virtual network. Application, data, and shared-service workloads are then placed into separate spoke virtual networks and connected to the hub through peering.A typical Malaysia West hub-and-spoke design may include:A hub VNet for shared network services.Central firewall or network virtual appliance inspection.VPN or ExpressRoute gateway placeholders, subject to service availability validation.Shared DNS resolution and private DNS integration.Monitoring, logging, security, and management services.Separate spokes for applications, data platforms, integration services, and management workloads.User-defined routes to control traffic inspection and egress.The strength of this model is control. Platform teams can define route tables, firewall policies, segmentation boundaries, DNS flows, and spoke onboarding standards explicitly. This fits organisations that want strong governance over each workload and clear separation between central platform ownership and application team ownership.The trade-off is operational responsibility. As the number of spokes, branches, regions, routes, firewalls, and exceptions grows, the design can become complex. Route propagation, asymmetric routing, firewall policy sprawl, peering limits, and manual onboarding processes need strong governance.Microsoft’s Azure Architecture Center describes hub-spoke as a recognised Azure topology pattern. It also encourages careful regional hub design rather than forcing unrelated cross-region traffic through a single hub. That advice matters for Malaysia West. If the organisation expects workloads in Malaysia West, Southeast Asia, and other Azure regions, the network team should plan regional hubs, routing boundaries, and DR paths deliberately rather than stretching one hub beyond its natural blast radius.Architecture context: what Azure Virtual WAN providesAzure Virtual WAN is Microsoft’s managed wide-area networking service. It brings together capabilities such as branch connectivity, site-to-site VPN, point-to-site VPN, ExpressRoute connectivity, virtual network connectivity, routing, and optional security integration through virtual hubs.The design pattern is still hub-like, but the hub is Microsoft-managed. Instead of building every routing and connectivity component inside a custom hub VNet, the organisation uses a Virtual WAN resource and virtual hubs to connect VNets, branches, remote users, and connectivity circuits.The potential value is scale and simplification:Centralised connectivity model for many branches or sites.Transitive connectivity between connected networks where supported.Simplified branch onboarding compared with building every connection manually in a custom VNet hub.More integrated handling of VPN, ExpressRoute, and virtual network connectivity.Optional secured virtual hub pattern where supported and validated.The trade-off is abstraction. Azure Virtual WAN reduces some operational burden, but it also constrains parts of the design. Teams that require very custom firewall placement, deep NVA integration, bespoke routing paths, or non-standard segmentation patterns must confirm that Virtual WAN can support those requirements before committing.Security is not automatically stronger in either model. A badly governed hub-and-spoke design can expose workloads. A poorly understood Virtual WAN design can also create routing and inspection gaps. The correct decision depends on implementation discipline, not the product name.Malaysia West-specific considerationsMalaysia West is important because it may allow Malaysian enterprises to place selected workloads closer to local users, facilities, and data residency expectations. However, proximity alone does not complete the architecture.Before approving either topology for Malaysia West, validate the following:Whether each required Azure networking service is available in Malaysia West.Whether the required SKUs are available, not just the service family.Whether availability-zone support exists for each service in scope.Whether the required gateway throughput, connection limits, and quotas are supported.Whether selected NVA marketplace images are deployable in Malaysia West.Whether ExpressRoute provider, peering, and last-mile assumptions are valid for the organisation’s locations.Whether operational tooling, monitoring, and diagnostics are regionally supported.Whether DR design depends on unsupported assumptions about official region pairing.Do not assume that “the region exists” means every networking feature is available. Do not assume a region-level availability-zone statement means every service can be deployed zone-redundantly. Do not assume Southeast Asia is the official paired region for Malaysia West unless current Microsoft documentation confirms it at the time of implementation.For many Malaysian organisations, the practical design may involve Malaysia West for production workloads, Southeast Asia or another validated region for DR, and carefully controlled connectivity between them. That is a DR and network architecture decision, not only a region selection decision.Design recommendation 1: use hub-and-spoke when control is the priorityHub-and-spoke is the preferred starting point when the cloud estate is still manageable and the organisation needs direct control over inspection, routing, and segmentation.It is a strong fit when:The organisation is building one primary landing zone in Malaysia West.Workloads are separated into a small number of well-governed spokes.Centralised security inspection is required through Azure Firewall or an NVA, subject to availability validation.The platform team wants explicit UDR control.DNS, private endpoints, and egress control require detailed design.The organisation already has Azure landing-zone governance based on management groups, policies, subscriptions, and regional hub VNets.Network changes are handled by a skilled central cloud platform or network team.A sensible Malaysia West hub-and-spoke reference design includes a hub VNet, firewall or inspection placeholder, gateway placeholders, private DNS integration, shared monitoring, shared security services, and workload spokes for application and data tiers. Each service should remain a placeholder until validated against current regional availability.This design is easier to explain to auditors, security teams, and operations teams because the traffic flow is visible: spoke-to-hub, hub-to-on-premises, hub-to-internet, and spoke-to-shared-services. It also aligns well with phased migration, where each workload is onboarded into a controlled spoke.The risk is that custom control can become custom complexity. If every application team requests unique routing, firewall exceptions, DNS patterns, and connectivity paths, the hub can become fragile. Strong design standards are mandatory.Design recommendation 2: consider Azure Virtual WAN when scale is the problemAzure Virtual WAN becomes attractive when the network problem is less about a few Azure spokes and more about many connectivity domains.It is a strong candidate when:The organisation has many Malaysian branches, campuses, factories, warehouses, clinics, schools, or retail outlets.SD-WAN or VPN aggregation is a major requirement.Remote user connectivity is part of the architecture.ExpressRoute and VPN need to coexist across multiple sites or regions.The organisation expects multi-region cloud growth.The network team wants to reduce custom hub routing operations.Transitive connectivity and managed hub routing are more valuable than highly customised VNet-level design.Examples include manufacturers with multiple plants, retailers with many outlets, logistics companies with depots, education groups with campuses, and healthcare networks with distributed sites. These organisations often need a repeatable branch connectivity model more than a handcrafted hub for each workload.However, Virtual WAN should not be selected before validating service availability, hub region support, gateway capabilities, security integration, quotas, and cost. In Malaysia West planning, the architecture must explicitly state whether the virtual hub is intended for Malaysia West or another validated region. If Malaysia West support is not confirmed, the design should label the virtual hub as “target region subject to availability validation”.Design recommendation 3: use coexistence when migration risk is highMany enterprises already have a working hub-and-spoke estate in Azure. Replacing it entirely with Virtual WAN may be unnecessary and disruptive.A coexistence approach can work when:Existing workload spokes and security controls are stable.The organisation wants Virtual WAN for branch or regional connectivity only.The network team needs a gradual migration path.Some applications require custom inspection while others can use managed hub routing.DR or regional expansion requires a new connectivity model without redesigning every spoke.In this model, hub-and-spoke remains the workload segmentation pattern, while Virtual WAN may serve as the broader connectivity fabric for branches, remote users, and inter-region routing. This must be designed carefully to avoid routing loops, inspection bypass, inconsistent DNS behaviour, and unclear operational ownership.Coexistence is not a shortcut. It is an architecture pattern that requires a routing authority, clear security inspection policy, documented failover paths, and monitoring across both models.Risks and constraintsRisk: unsupported Malaysia West service assumptionsImpact: The design may fail during deployment or require late-stage redesign.Recommendation: Validate every named service, SKU, quota, preview/GA status, and zone-redundant option before sign-off.Risk: unclear inspection pathImpact: Workloads may bypass firewall, DLP, IDS/IPS, or logging controls.Recommendation: Document north-south, east-west, on-premises, internet, and private endpoint traffic flows before implementation.Risk: routing complexity hidden behind abstractionImpact: Virtual WAN may simplify operations, but troubleshooting can become difficult if the team does not understand propagation, association, and route intent.Recommendation: Build a route design and operational runbook before production onboarding.Risk: over-customised hub-and-spoke designImpact: Manual UDRs, exceptions, and NVA dependencies can create brittle operations.Recommendation: Standardise spoke patterns, firewall policy structure, DNS rules, and deployment automation.Risk: cost comparison without traffic modelImpact: The organisation may choose the wrong pattern based on incomplete pricing assumptions.Recommendation: Model costs only after defining hubs, gateways, firewall requirements, data transfer, branch count, throughput, logging, and DR traffic.Decision checklistUse this checklist before selecting the topology:Topology scale: How many VNets, branches, remote users, and Azure regions are expected within three years?Primary region: Is Malaysia West the production region, DR region, or part of a multi-region design?Service availability: Are Virtual WAN, virtual hubs, VPN, ExpressRoute, Azure Firewall, Bastion, Private Link, NAT Gateway, Application Gateway/WAF, and required SKUs validated for the selected region?Security model: Where is traffic inspected? What is allowed to bypass inspection, if anything?Routing model: Who owns route design, propagation, route tables, and exception handling?Hybrid connectivity: Are VPN, ExpressRoute, SD-WAN, BGP, and last-mile provider assumptions documented?DNS and private access: How will private endpoints, Private DNS zones, conditional forwarding, and on-premises DNS integration work?Operations: Can the current team support the chosen model during incidents?DR: What happens during regional failure, gateway failure, firewall failure, or branch connectivity failure?Cost: Has the design been modelled using current Azure pricing and realistic traffic volumes?Decision table Requirement Hub-and-spoke starting point Azure Virtual WAN starting point Mandatory validation Single Malaysia West landing zone with controlled egress Strong fit Optional Firewall, gateway, DNS, and service availability in Malaysia West Many Malaysian branches or factory/retail sites Possible, but operationally heavier Strong fit Virtual hub support, VPN scale, SD-WAN interoperability Existing custom NVA inspection Strong fit Evaluate carefully NVA support, routing limits, marketplace image availability Multi-region estate Possible with one hub per region Strong fit for managed transitive model Hub region support and routing requirements Strict custom routing and segmentation Strong fit May be constrained Route propagation, association, inspection path Smaller enterprise with limited cloud network team Depends on skills May reduce operations if supported Cost, supportability, feature fit Regulated workload with audit scrutiny Strong fit if documented well Viable if controls are explicit Logging, firewall policy, evidence collection Related readingMalaysia West Azure Region: Architecture Planning Guide for EnterprisesAzure Landing Zone Design for Malaysia WestAzure Virtual WAN vs Hub-and-Spoke: Which Network Architecture for 2026?Azure Local for Malaysian Enterprises: When Hybrid Beats Cloud-OnlyCTA: Azure architecture review for Malaysia Westarchitecture-ctaPlanning Azure networking for Malaysia West? I can review your hub-and-spoke or Azure Virtual WAN design, validate Malaysia West service availability assumptions, assess hybrid connectivity, routing, security inspection, DNS, and DR dependencies, then produce a practical network architecture recommendation before you build.A good review should answer three questions clearly: what should be deployed in Malaysia West, what should remain in another validated region, and what must be changed before the design is safe for production.Final recommendationFor most Malaysia-based enterprises starting a local Azure platform, I would begin with a controlled hub-and-spoke landing zone unless branch scale, SD-WAN, remote access, ExpressRoute aggregation, or multi-region routing complexity clearly justifies Azure Virtual WAN.For branch-heavy organisations, Virtual WAN deserves serious evaluation, but only after regional availability, hub support, gateway requirements, security integration, cost model, and operational runbooks are validated. For existing Azure estates, coexistence may be the most practical path.The right answer is not the newest topology. The right answer is the topology your organisation can secure, operate, troubleshoot, fund, and evolve without creating hidden network risk. --- ## Microsoft Fabric RTI: Live Dashboards Without a Data Team (2026) URL: https://wenfeng.my/microsoft-fabric-real-time-intelligence-building-live-enterprise-dashboards-without-a-data-team-2/ Author: admin Published: 2026-05-13 23:31:09 Modified: 2026-10-04 15:33:40 ⚡ TL;DRFor most Malaysian enterprises, real-time analytics has been the domain of companies with dedicated data engineering teams, Kafka clusters, and Spark expertise.What Is Real-Time Intelligence in Fabric?Why This Matters for Malaysian EnterprisesStep-by-Step: Building a Real-Time IoT Monitoring DashboardMirrored Databases for Operational ReportingCost Model and Capacity PlanningFor most Malaysian enterprises, real-time analytics has been the domain of companies with dedicated data engineering teams, Kafka clusters, and Spark expertise. Microsoft Fabric's Real-Time Intelligence capabilities change this equation for many Microsoft-centric analytics teams. Business analysts, application developers, and operations teams can build streaming dashboards and live monitoring solutions without running Kafka or Spark infrastructure themselves — although production designs still need data ownership, governance, and operational accountability.This article walks through what Real-Time Intelligence in Fabric actually is, when it makes sense, how to use it with real KQL examples, and what Malaysian businesses should consider before adopting it.What Is Real-Time Intelligence in Fabric?Microsoft Fabric's Real-Time Intelligence hub is a set of integrated services that ingest, process, analyze, and visualize streaming data — all within the same SaaS platform as your data lake, data warehouse, and Power BI reports.The core components are:EventhouseEventhouse is a managed Kusto Query Language (KQL) database optimized for time-series, streaming, semi-structured, and log-style data. Think of it as a purpose-built database for "data that is happening right now" — IoT sensor readings, API logs, application traces, and security events. Microsoft positions Eventhouse for storing and analyzing streaming data, querying billions of events in seconds, and organizing data for fast search based on arrival time.Key capabilities: - Data is organized for fast searching based on when it arrived, which fits telemetry and log analytics patterns. - Materialized views for pre-computed aggregates. Microsoft documents materialized views as an optimization pattern; validate performance improvement against your own workload rather than relying on a universal percentage. - Governance through Fabric workspace/item permissions, auditability, lineage, and security patterns such as governed views/functions or downstream semantic-model controls where sensitive fields are involved. - Integration with Fabric and Power BI reporting experiences without building a separate streaming database stack.Real-Time DashboardsThese are KQL-powered dashboards that connect to KQL databases and can be configured with auto-refresh for operational monitoring. Unlike scheduled BI reports, Real-Time Dashboards are designed for low-latency exploration and live operations views; set refresh frequency based on query cost, CU consumption, and the latency the business actually needs.EventstreamsEventstreams is a visual, no-code editor for building streaming data pipelines. You connect a source (Azure Event Hubs, IoT Hub, Kafka, or an HTTPS endpoint), optionally apply transformations (filter, project, aggregate), and send the result to one or more destinations (Eventhouse, Lakehouse, or Real-Time Dashboard).Mirrored DatabasesMirroring in Fabric lets you create low-latency replicas of supported operational databases and external sources into Fabric OneLake. Microsoft documents support for sources such as Azure SQL Database, Azure SQL Managed Instance, Azure Cosmos DB, Azure Database for PostgreSQL, Azure Database for MySQL (preview), SQL Server, Snowflake, SAP, Oracle, and others. Treat replication as near real-time rather than guaranteed seconds: latency depends on source/destination region, change volume, network latency, and compute resources. This is useful for operational reporting because analytical queries can target the mirrored data rather than the production system.Why This Matters for Malaysian EnterprisesMalaysia has a strong manufacturing sector (IoT sensor monitoring), a growing fintech scene (real-time transaction monitoring), and increasing adoption of cloud-based ERP systems (Operational reporting). All of these generate streaming data, but most organizations lack the specialized skills to build and maintain traditional streaming infrastructure. Requirement Traditional Approach Fabric Real-Time Intelligence Infrastructure Kafka brokers/controllers, stream processing, monitoring SaaS analytics platform; no self-managed streaming cluster Skills required Data engineer, Kafka/platform operator, stream-processing developer KQL plus data modelling/governance skills Time to first dashboard 2-6 months 1-2 weeks Cost for 10K events/sec Highly variable: Kafka/stream processing, storage, support, and operations effort Start with a Fabric capacity estimate, then validate with Capacity Metrics; do not size only from event count Scaling Manual cluster resizing Auto-scale within capacity Step-by-Step: Building a Real-Time IoT Monitoring DashboardLet us walk through a concrete scenario: a Malaysian manufacturing company wants to monitor equipment temperature, humidity, and vibration from 200 IoT sensors on the factory floor.Step 1: Create an EventhouseIn the Fabric portal, navigate to Real-Time Intelligence, select "New" → "Eventhouse". Name it FactoryTelemetry and place the workspace/capacity in the approved data residency region for the organization. If Malaysia West is available for your tenant and Fabric workload, use it; otherwise validate Fabric regional availability and compliance requirements before deployment.// After creation, verify the Eventhouse is online.show databases// Output should show your Eventhouse databaseStep 2: Create an EventstreamEventstreams use a visual interface for connecting event sources, applying supported transformations, and routing events to destinations. The implementation flow is:Add source — choose Azure IoT Hub, Event Hubs, or the custom HTTPS endpoint. For the factory scenario, IoT Hub is the natural choice because it supports device authentication and twin management.Configure transformation — add a filter step to discard malformed messages, add a project step to select only needed fields.Add destination — select your Eventhouse and specify the target table.// Conceptual Eventstream configuration (illustrative only; validate against Fabric export/API support before treating as deployable IaC){"sources": [{"type": "iotHub","name": "factory-sensors","iotHubName": "iothub-factory-malaysia","consumerGroup": "fabric-ingest"}],"transformations": [{"type": "filter","name": "filter-in-range","expression": "temperature >= -20 and temperature <= 150"}],"destinations": [{"type": "eventhouse","name": "factory-telemetry","eventhouseName": "FactoryTelemetry","databaseName": "FactoryTelemetry","tableName": "IoTReadings"}]}Step 3: Create the Target Table with KQLBefore data can flow into Eventhouse, you must define the table schema:// Define the table schema for IoT sensor readings.create table IoTReadings (timestamp: datetime,deviceId: string,temperature: real,humidity: real,vibration: real,status: string,factoryZone: string)// Define JSON mapping for incoming data.create table IoTReadings ingestion json mapping 'IoTMapping''['' {"column":"timestamp", "path":"$.timestamp"},'' {"column":"deviceId", "path":"$.device.id"},'' {"column":"temperature", "path":"$.telemetry.temperature"},'' {"column":"humidity", "path":"$.telemetry.humidity"},'' {"column":"vibration", "path":"$.telemetry.vibration"},'' {"column":"status", "path":"$.device.status"},'' {"column":"factoryZone", "path":"$.metadata.zone"}'']'// Create a materialized view for hourly aggregates.create materialized-view HourlyAggregates on table IoTReadings{IoTReadings| summarizeavg_temp = avg(temperature),max_temp = max(temperature),avg_humidity = avg(humidity),device_count = dcount(deviceId)by bin(timestamp, 1h), factoryZone}Step 4: Create a Real-Time DashboardIn the Fabric portal, select "New" → "Real-Time Dashboard". Add tiles using KQL queries:// Tile 1: Live temperature chart (last 15 minutes)IoTReadings| where timestamp > ago(15m)| where status == "online"| summarize avg(temperature) by bin(timestamp, 10s), deviceId| render timechart// Tile 2: Anomaly detection — devices exceeding temperature thresholdIoTReadings| where timestamp > ago(5m)| where temperature > 60| project timestamp, deviceId, temperature, factoryZone| sort by timestamp desc// Tile 3: Device count by zone (live count)IoTReadings| where timestamp > ago(5m)| summarize distinct_devices = dcount(deviceId) by factoryZone| render columnchart// Tile 4: 5-minute moving average with anomaly bandslet timeWindow = 5m;IoTReadings| where timestamp > ago(1h)| summarizeavg_temp = avg(temperature),percentile_95_temp = percentile(temperature, 95),percentile_05_temp = percentile(temperature, 5)by bin(timestamp, 1m), factoryZone| render timechart with(ysplit=panels)The dashboard auto-refreshes without manual refresh. You can pin individual tiles to a Power BI report for embedding in executive dashboards.Step 5: Set Up AlertsFor alerting, use Fabric Activator or dashboard alerting patterns instead of assuming alert rules are authored directly inside Eventhouse. Activator can monitor streaming data or dashboard/report conditions and trigger actions such as email, Teams notifications, or Power Automate flows:// Query to identify overheating devicesIoTReadings| where timestamp > ago(2m) and temperature > 70| projectAlertTime = now(),DeviceId = deviceId,Temperature = temperature,Zone = factoryZone,Message = strcat("Device ", deviceId, " in zone ", factoryZone," is overheating at ", strcat(tostring(temperature), "°C"))Use this query as the condition behind the dashboard or Activator rule, then route notifications through Teams, email, Power Automate, or another approved workflow.Mirrored Databases for Operational ReportingThe Mirroring capability deserves special attention. It creates a low-latency replica of supported operational data in Fabric without building a separate ETL pipeline, but source prerequisites and latency must be validated.Setting Up a Mirrored Database for PostgreSQLFor Azure Database for PostgreSQL, use the Fabric mirroring workflow documented by Microsoft and confirm source prerequisites before production use. A safe implementation outline is:In Fabric, create a mirrored database for Azure Database for PostgreSQL.Create or select a secure connection to the source database.Select the databases/tables supported by the mirroring workflow.Validate replication health and latency under expected write volume.Query the mirrored data from Fabric analytics experiences such as SQL analytics endpoints, Power BI, notebooks, or other supported Fabric workloads.Example analytical query pattern after data is available in Fabric:SELECTDATEADD(hour, DATEDIFF(hour, 0, LastModified), 0) AS hour_bucket,OrderStatus,SUM(Amount) AS TotalRevenueFROM dbo.ordersWHERE LastModified > DATEADD(day, -1, SYSUTCDATETIME())GROUP BY DATEADD(hour, DATEDIFF(hour, 0, LastModified), 0), OrderStatus;Use cases for mirrored databases in Malaysia: - ERP reporting with reduced production impact — mirror a supported operational source into Fabric and run analytical queries against the mirrored data instead of the production database. Validate source support, licensing, and replication overhead for systems such as Dynamics, SAP, SQL Server, or PostgreSQL. - Cross-database joins — join mirrored CRM data with mirrored inventory data to build a real-time order fulfillment dashboard. - Compliance queries — auditors need access to historical data without touching production. Point them at the mirror.Cost Model and Capacity PlanningFabric uses Capacity Units (CU). Real-Time Intelligence consumes CU for:Ingestion — each MB of data ingested consumes CU based on the source and transformations.Querying — each KQL query consumes CU proportional to the data scanned.Storage — Eventhouse data is stored in Fabric OneLake, billed separately from CU.Capacity Sizing Guide Workload Events/Second Recommended Capacity Compute price reference Small (50 IoT devices, basic dashboards) 100 F2 pilot Microsoft listed Malaysia West pay-as-you-go compute at USD 0.38/hour at time checked; about USD 277/month before storage, tax, network, and currency effects Medium (200 devices, 5 dashboards, alerting) 1,000 F4 pilot/production validation USD 0.76/hour; about USD 555/month before storage, tax, network, and currency effects Large (2,000 devices, 15 dashboards, anomaly detection) 10,000 F8 starting estimate USD 1.52/hour; about USD 1,110/month before storage, tax, network, and currency effects Enterprise (10,000+ devices, complex ML models) 50,000+ F16+ after benchmark F16 listed at USD 3.04/hour; about USD 2,219/month before storage, tax, network, and currency effects Note: The estimates below are based on Fabric pay-as-you-go capacity pricing. Actual costs depend on CU consumption patterns, data ingestion volume, and query complexity. Monitor usage with the Fabric Capacity Metrics app.Cost Optimization StrategiesUse materialized views. Instead of scanning raw data for every dashboard query, pre-compute hourly or daily aggregates. Treat materialized views as a workload-specific optimization: measure ingestion cost, query latency, and refresh behaviour before declaring the improvement.// Compare: raw query (expensive)IoTReadings| where timestamp > ago(7d)| summarize avg(temperature) by bin(timestamp, 1h)// This scans 7 days of raw data per refresh// vs. materialized view (cheap)HourlyAggregates| where timestamp > ago(7d)// This scans 168 rows instead of millionsSet data retention limits. Kusto databases support retention and caching policies. For the factory use case, consider keeping raw hot-path data for 7 days and longer-term aggregates for 1 year, then validate against regulatory and operational requirements:In the Fabric Eventhouse UI, you can configure retention and caching policies per table: - IoTReadings: Set retention to 7 days (raw telemetry, short-lived) - HourlyAggregates: Set retention to 365 days with a 30-day hot cacheThese settings are configured through the Eventhouse table management interface rather than KQL commands.Use caching policies. Configure hot-cache policy deliberately for frequently queried data. Do not assume a universal default is right for every workload; tune cache duration against query latency and capacity consumption.Comparison with Traditional Streaming SolutionsBefore Fabric Real-Time Intelligence, Malaysian organizations building real-time dashboards typically used one of these approaches:Apache Kafka + Spark StreamingPros: Maximum flexibility, battle-tested, works with any data source.Cons: Requires Kafka operations skills, broker/controller management, stream-processing skills, and a separate monitoring stack unless you use a managed service. VM-based cost varies widely by sizing, storage, replication factor, support model, and region before any stream-processing compute.Best for: Organizations with existing Kafka investment or complex stream processing needs (joins across multiple streams, stateful aggregations, ML inference in-stream).Azure Stream AnalyticsPros: SQL-based, managed service, native connections to Azure services.Cons: Job-oriented (not dashboard-oriented), you still need a separate visualization layer. Output must go to a database or Power BI.Best for: Simple ETL patterns where you need to filter and route streaming data to multiple destinations.Fabric Real-Time IntelligencePros: SaaS analytics platform, no self-managed streaming cluster for common scenarios, KQL-based, built-in dashboards, unified with the rest of Fabric.Cons: Tied to Microsoft ecosystem, limited to Fabric's CU model for costing, some advanced processing patterns (exactly-once semantics, stateful joins) require careful design.Best for: Organizations that want a unified analytics platform and need to get a real-time dashboard running in weeks, not months.Security and Governance ConsiderationsFor Malaysian enterprises, data governance is not optional. Fabric Real-Time Intelligence provides:Access control and least privilege — use Fabric workspace roles, item permissions, and governed sharing rather than giving broad workspace access to every dashboard user.Sensitive-field minimization — avoid exposing personally identifiable or commercially sensitive columns in operational dashboards. Use governed views/functions, semantic-model security, or separate curated datasets where appropriate.// Example: expose a governed view that omits sensitive telemetry fields.create-or-alter function ReadOnlyUserView () {IoTReadings| where timestamp > ago(7d)| project timestamp, deviceId, status // Exclude temperature and humidity columns}Row-level and object-level security where supported — apply the relevant Fabric/Power BI/Mirrored Database security feature for the item type being served.Audit logging and lineage — use Fabric audit and lineage features to understand who accessed data and how data moved from source to dashboard.Common Pitfalls to AvoidPitfall 1: Treating Eventhouse like a data lake. Eventhouse is optimized for time-series queries over recent data. Do not use it as the primary storage for data older than your retention policy. Archive historical data to the Fabric Lakehouse and use Eventhouse for the hot path only.Pitfall 2: Over-provisioning capacity upfront. Start with F2 or F4 capacity. Monitor CU consumption in the Capacity Metrics app for two weeks before scaling up. Most workloads consume less CU than expected.Pitfall 3: Ignoring data structure for mirrored databases. Mirrored databases replicate the source schema exactly, including any messy column names, nullable fields, or missing indexes. Plan for data cleanup in your KQL queries rather than trying to enforce schema changes on the source.Pitfall 4: Real-time for the sake of real-time. Not every dashboard needs sub-second refresh. If your business decisions are made daily, a scheduled Power BI report or batch dashboard is usually cheaper than a Real-Time Dashboard. Use Real-Time Intelligence only for the subset of dashboards that genuinely need it.Key TakeawaysReal-Time Intelligence reduces the need to operate dedicated streaming infrastructure — no self-managed Kafka or Spark cluster for common dashboard scenarios, but production ownership and data governance remain necessary.Eventhouse is the core engine — KQL databases with native time-series optimizations, capable of querying billions of events in seconds.Fabric capabilities are improving, but validate item-level support — materialized views, source connectors, governance controls, and mirrored databases are useful patterns, but validate each feature in your target region, tenant, capacity, and source system.Start with one use case — pick a single high-value dashboard (IoT monitoring, API performance, or real-time sales), build it completely, prove the value, then expand.Monitor CU consumption — use the Fabric Capacity Metrics app to track costs. Use materialized views and retention policies to optimize.Mirroring can simplify operational reporting — supported sources can replicate into Fabric with near real-time behaviour, but latency and source support must be validated for each workload.Building real-time intelligence on Fabric is a capability I help Malaysian enterprises design and implement. If your organisation needs live dashboards without dedicated data engineering headcount, let's discuss on LinkedIn. Related Reading Microsoft Fabric Real-Time Intelligence: Eventhouse, KQL, and Streaming Analytic Microsoft Fabric Lakehouse Migration: Moving from Traditional Data Warehouses to FabCon and SQLCon 2026: Why SQL and Fabric Are Becoming One Operational Data Pla --- ## Azure Landing Zone Design for Malaysia West URL: https://wenfeng.my/mw-landing-zone-design/ Author: admin Published: 2026-05-12 22:51:29 Modified: 2026-10-04 15:30:32 ⚡ TL;DRCampaign: Malaysia West Azure ArchitectureStatus: Draft for review — validate current Azure regional availability before implementation or publication.Practical Executive SummaryArchitecture Context: What an Azure Landing Zone Must ProvideMalaysia West-Specific ConsiderationsDesign RecommendationsMalaysia West Landing-Zone Decision ChecklistCampaign: Malaysia West Azure ArchitectureStatus: Draft for review — validate current Azure regional availability before implementation or publication.Practical Executive SummaryMalaysia West gives Malaysian enterprises a stronger reason to revisit their Azure landing-zone design. It should not be treated as a simple region selection change. A landing zone is the operating foundation for cloud adoption: identity, subscriptions, networking, policy, security, logging, backup, and workload onboarding. If this foundation is weak, the first migration may still succeed technically, but the second, fifth, and twentieth workload will create governance debt.For enterprises planning to use Malaysia West as a primary Azure region, my recommendation is straightforward: use the standard Azure landing-zone architecture pattern, but add a Malaysia West-specific validation gate before design approval. The core pattern should remain consistent with the Microsoft Cloud Adoption Framework: management groups, platform landing zones, application landing zones, policy-based governance, central connectivity, central monitoring, and clear separation between platform and workload responsibilities.The Malaysia West-specific work is the validation layer. Before any production deployment, architects must validate the current availability of each required Azure service, each SKU, availability-zone support, quota, capacity, data residency behaviour, backup and replication options, and network connectivity path. Do not assume that a service, feature, or replication model available in Southeast Asia is automatically available in Malaysia West. Do not assume that Malaysia West is paired with Southeast Asia unless Microsoft documentation confirms it at the time of design. Treat Southeast Asia as a candidate secondary region to evaluate, not as an automatic pair.A good landing-zone design for Malaysia West should answer these questions before workload migration:Which management groups and subscriptions will host platform, connectivity, management, security, sandbox, and workload resources?Which regions are allowed for production, DR, backup, logging, and shared services?Which services are approved for Malaysia West after verification against Microsoft regional availability sources?How will traffic enter, leave, and move across Azure, on-premises sites, and remote users?How will privileged access, policy, monitoring, backup, and incident response operate after go-live?What is the fallback plan if a required service or SKU is not available in Malaysia West at the required scale?The right outcome is not just a diagram. It is an approved operating model that lets teams deploy safely, consistently, and with evidence-backed regional assumptions.Architecture Context: What an Azure Landing Zone Must ProvideAn Azure landing zone is the enterprise foundation for cloud workloads. In practical terms, it is the controlled environment into which applications, data platforms, integration services, virtual machines, containers, and security tools are deployed. It is not a single subscription. It is not only a virtual network. It is not only a Terraform repository. It is a combination of architecture decisions, governance controls, identity model, automation, and operating process.For most enterprises, the landing-zone model should separate two responsibilities.The first is the platform landing zone. This is where the central cloud platform team manages shared capabilities such as connectivity, security tooling, management, monitoring, policy, and sometimes identity-adjacent services. These components should be designed once, operated carefully, and reused across workloads.The second is the application landing zone. This is where business workloads run. An application landing zone may be a subscription or a set of subscriptions assigned to a product team, business unit, environment, or workload portfolio. It inherits governance from the management-group structure, connects through approved network patterns, and follows the standard security and monitoring baseline.Malaysia West does not change these fundamentals. What it changes is the importance of regional validation. A mature Azure landing-zone pattern can be deployed into different regions, but every region has its own service availability, SKU availability, zone support, quota, capacity, and network considerations. That is why I would not design a Malaysia West landing zone by copying an existing Southeast Asia deployment without a structured review.The correct architecture approach is:Keep the enterprise control plane consistent.Validate the regional data plane carefully.Use policy and automation to enforce decisions.Document exceptions before workloads are migrated.This is especially important for regulated, latency-sensitive, or data-residency-driven organisations in Malaysia. Malaysia West may support important business objectives, but compliance and resilience outcomes still depend on the exact services, configuration, data flows, backup locations, and operating controls selected by the customer.For the broader regional planning context, read: Malaysia West Azure Region: Architecture Planning Guide for Enterprises.Malaysia West-Specific ConsiderationsThe most important Malaysia West design principle is conservative validation. Architects should use careful wording and careful design gates until Microsoft’s current documentation and the customer’s tenant-level availability confirm the implementation path.1. Service Availability Is Not a Generic StatementA landing-zone diagram often includes components such as firewall, application gateway, private endpoints, DNS, monitoring, key management, backup, and security posture management. However, for Malaysia West, each service named in the design must be checked against current Microsoft regional availability sources before implementation. Regional availability can vary by service, SKU, feature, and preview or general availability state.This means the architecture pack should include a service availability matrix, not only a high-level diagram. For each required service, record:Required Azure service and SKU.Target region, normally Malaysia West for primary deployment.Availability status from current Microsoft documentation.Availability-zone support, if required.Quota or capacity constraints.Known feature limitations.Approved fallback region or alternative design.This matrix should be reviewed before deployment, before commercial commitment, and again before production cutover.2. Availability Zones Must Be Validated by ServiceAvailability-zone support should not be discussed as a blanket region claim. Even where a region supports zones, individual Azure services may have different zone-redundant, zonal, or non-zonal behaviours. The design should state the resilience requirement first, then verify whether the target service supports the required pattern in Malaysia West.For example, a workload may require zone-redundant compute, zone-redundant storage, zone-aware database deployment, or zonal ingress. These requirements need to be validated service by service. If a required service does not support the needed resilience pattern in Malaysia West, the design may need to use a different SKU, a different service, active-passive deployment, application-level resilience, or a secondary region.3. Region Pairing and DR Must Not Be AssumedDo not assume that Southeast Asia is the Microsoft paired region for Malaysia West. Treat Southeast Asia as a practical candidate secondary region to evaluate because it is a mature nearby Azure region, but validate official region-pair and replication behaviour before any DR design is approved.This matters because Azure service replication behaviour varies. Storage redundancy, database replication, backup vault placement, disaster recovery orchestration, and service-managed failover do not all behave the same way. Some services require explicit target-region selection. Some features may not support every region combination. Some services are global or geo-scoped rather than strictly regional.The landing zone should therefore define DR as a requirement-driven design, not a default region-pair assumption.4. Data Residency Is an Architecture Objective, Not an Automatic Compliance OutcomeMalaysia West may support data residency objectives for Malaysian enterprises, but no region selection alone guarantees compliance. Architects still need to map data stores, logs, backups, telemetry, security tooling, identity flows, support processes, third-party integrations, and cross-region replication.A good landing-zone design should define approved data locations for:Production application data.Database backups and snapshots.Log Analytics and security logs.Key management and secrets.Replicated or exported data.Dev/test copies of production data.Third-party monitoring, ITSM, and backup platforms.If data residency is a board-level or regulatory driver, the landing-zone design must be reviewed with security, risk, legal, and compliance stakeholders before implementation.Design Recommendations1. Use a Clear Management-Group HierarchyStart with a management-group structure that supports policy inheritance and operational separation. A practical enterprise model should include:A tenant root group managed only by a small platform governance team.A platform management group for shared services.A landing zones management group for production and non-production workloads.A sandbox management group for controlled experimentation.A decommissioned or quarantine group for subscriptions being retired or investigated.Under the platform group, separate connectivity, management, and security where operationally justified. Smaller organisations can consolidate some platform functions, but they should still keep a clear boundary between platform responsibilities and application team responsibilities.Avoid placing all workloads directly under one flat structure. It becomes difficult to apply policies, separate cost ownership, delegate access, or operate consistent controls.2. Separate Subscriptions by Responsibility and LifecycleThe subscription model should reflect ownership, risk, and lifecycle. For Malaysia West landing zones, I normally recommend the following baseline:Connectivity subscription for hub networking, firewall or network security services, DNS forwarding, private connectivity, and central routing.Management or operations subscription for logging, monitoring, automation, update management, and operational tooling.Security subscription if the security operations model requires separation for SIEM, security posture management, or central security integrations.Identity or shared services subscription only if there are specific Azure-hosted identity-adjacent services; Microsoft Entra ID itself is tenant-scoped, not subscription-scoped.Application landing-zone subscriptions separated by workload, environment, or business unit.Sandbox subscriptions with strict policy, budget, and expiry controls.This model supports cleaner RBAC, cost attribution, policy assignment, and operational accountability.3. Build Identity Around Least PrivilegeIdentity is not a regional Azure design area, but it is one of the most common sources of cloud risk. The landing zone should use Microsoft Entra ID groups, role-based access control, privileged identity management, conditional access, managed identities, and break-glass accounts.Key recommendations:Avoid permanent subscription owner assignments for engineers and vendors.Use Privileged Identity Management for elevated access where licensing and operating model allow it.Assign roles to groups, not individual users, wherever possible.Use managed identities for workloads instead of secrets stored in code or pipelines.Maintain two monitored break-glass accounts with strong controls and documented procedures.Separate platform admin roles from workload contributor roles.For Malaysia West adoption, identity design should be completed before workload teams are onboarded. Retrofitting RBAC after multiple migrations is operationally painful.4. Choose Hub-and-Spoke Unless the WAN Requirement Justifies Virtual WANFor many Malaysian enterprises, a hub-and-spoke topology remains the most understandable baseline. It provides central ingress and egress, inspection, shared DNS, private connectivity, and workload segmentation. Workload virtual networks connect as spokes and inherit central controls.Azure Virtual WAN can be a better option where the enterprise has many branches, complex regional connectivity, large-scale SD-WAN integration, or a global network operating model. However, it should be selected because the connectivity requirement justifies it, not because it is newer.For Malaysia West, do not assume any specific ExpressRoute location, peering behaviour, Virtual WAN hub capability, or gateway SKU availability without verification. The network design should include placeholders for:On-premises data centres and branches.Site-to-site VPN or private connectivity.Central firewall or network virtual appliance options.DNS forwarding and private DNS resolution.Private Link and private endpoint patterns.Egress control and inspection.Segmentation between production, non-production, and shared services.For a deeper comparison, read: Azure Virtual WAN vs Hub-and-Spoke: Which Network Architecture for 2026?.5. Enforce Governance Through Azure PolicyGovernance should not rely on manual review alone. The landing zone should use policy initiatives to enforce baseline decisions consistently.Practical policy controls include:Allowed regions, with Malaysia West as the preferred primary region where validated.Approved secondary regions for DR, logging, or shared services where required.Mandatory tags for owner, cost centre, environment, application, data classification, and support model.Denial or audit of public IP creation where not explicitly approved.Required diagnostic settings for supported services.Required encryption and secure transfer settings.Restriction of unsupported SKUs or preview services in production.Defender for Cloud and security baseline enablement where applicable.Private endpoint controls for sensitive data services.For Malaysia West, the allowed-region policy must be designed carefully. If it is too strict, it may block required shared services, DR, or logging patterns that are not yet available in Malaysia West. If it is too loose, teams may deploy data and workloads into unapproved regions. The right design is usually a policy initiative with controlled exceptions, not a single hard-coded rule applied everywhere.6. Design Monitoring, Backup, and DR Before MigrationOperations must be part of the landing zone, not a post-migration task. The platform should define logging, alerting, backup, recovery, incident response, patching, vulnerability management, and service health monitoring before the first production workload goes live.At minimum, define:Where activity logs and resource diagnostic logs are sent.Whether central Log Analytics workspaces are regional, shared, or workload-specific.Which security events are monitored by the SOC.Backup vault placement and retention requirements.Recovery objectives by workload tier.DR target region candidates and service-specific replication options.Alert routing into ITSM, email, chat, or managed services operations.Runbooks for platform incidents, connectivity failures, identity incidents, and backup recovery tests.For hybrid constraints or workloads not ready for public cloud, see: Azure Local for Malaysian Enterprises: When Hybrid Beats Cloud-Only.Malaysia West Landing-Zone Decision ChecklistUse this checklist before approving the design.Region and service validationConfirm Malaysia West availability for every required Azure service.Confirm SKU availability, quota, capacity, and feature limitations.Confirm availability-zone support by service, not only by region.Confirm preview versus generally available status.Record evidence and review date.Management and subscription designDefine management-group hierarchy.Separate platform and application subscriptions.Define production, non-production, sandbox, and decommissioning controls.Assign cost ownership and tagging standards.Identity and accessUse group-based RBAC.Enable privileged access workflows where applicable.Define break-glass accounts and monitoring.Remove unnecessary permanent owner assignments.Use managed identities for workloads and automation.Network and connectivitySelect hub-and-spoke or Virtual WAN based on actual WAN requirements.Validate Malaysia West support for required network services and SKUs.Define ingress, egress, DNS, private endpoint, and routing patterns.Document on-premises connectivity assumptions and fallback options.Security and governanceApply Azure Policy initiatives for region, tagging, diagnostics, public exposure, and security baseline.Define Defender for Cloud and posture management scope where available and licensed.Require encryption, key management, and private access patterns for sensitive workloads.Document exception handling and approval workflow.Operations, backup, and DRDefine monitoring workspace strategy and log retention.Validate backup and recovery service availability.Define RTO and RPO tiers.Evaluate Southeast Asia or another region only as a candidate secondary region until validated.Test restore and failover procedures before production dependency.Risks and ConstraintsRisk: Unverified service availability in Malaysia WestImpact: Design components may not be deployable, or may require different SKUs, regions, or patterns.Recommendation: Maintain a service-by-service validation matrix and update it before design sign-off.Risk: Overly restrictive allowed-region policyImpact: Required DR, backup, logging, or shared services may fail deployment if they need an approved secondary region.Recommendation: Use policy initiatives with controlled exceptions and documented region purposes.Risk: Assuming Southeast Asia is the automatic DR pairImpact: DR design may not match Microsoft-supported replication behaviour or customer recovery requirements.Recommendation: Treat Southeast Asia as a candidate secondary region and validate each service’s replication model.Risk: Subscription-owner sprawlImpact: Excessive privileged access increases security exposure and weakens operational control.Recommendation: Use least privilege, group-based RBAC, privileged access workflows, and regular access reviews.Risk: Landing zone becomes a one-time build instead of an operating modelImpact: Governance drift, inconsistent workload onboarding, and poor incident response will appear over time.Recommendation: Define ownership, runbooks, monitoring, policy lifecycle, and change management from day one.Related ReadingMalaysia West Azure Region: Architecture Planning Guide for EnterprisesAzure Landing Zones for AI WorkloadsAzure Virtual WAN vs Hub-and-Spoke: Which Network Architecture for 2026?Azure Local for Malaysian Enterprises: When Hybrid Beats Cloud-OnlyAzure Backup vs Veeam on Azure: Which DR Strategy for Malaysian SMEs?CTA: Azure Architecture Review for Malaysia Westarchitecture-cta — Need to validate your Malaysia West Azure landing zone before migration?I can review your target workloads, regional availability assumptions, subscription model, network topology, governance baseline, security controls, monitoring design, backup strategy, and DR options. The outcome is a practical landing-zone readiness plan for Azure Malaysia West, including design risks, validation actions, and implementation priorities.Closing ViewMalaysia West should be approached as an architecture opportunity, not just a new deployment location. The best landing-zone designs will combine standard Azure governance patterns with disciplined regional validation. That means strong management groups, clean subscription boundaries, least-privilege access, controlled network architecture, policy-based governance, security baselines, and operational readiness.The final design decision should be evidence-based. Validate what is available today, document what is assumed, and decide how the platform will handle exceptions. If that discipline is applied early, Malaysia West can become a strong foundation for local cloud adoption rather than another source of technical debt. --- ## Azure Backup vs Veeam: Best DR for Malaysian SMEs (2026) URL: https://wenfeng.my/azure-backup-vs-veeam-on-azure-which-dr-strategy-for-malaysian-smes-2/ Author: admin Published: 2026-05-11 23:30:54 Modified: 2026-10-04 15:33:29 ⚡ TL;DRSmall and medium enterprises in Malaysia face a specific challenge: they need enterprise-grade disaster recovery on a budget, but the options can be overwhelming.The Malaysian SME DR RealityAzure Backup: The Native OptionVeeam on Azure: The Enterprise OptionReal Cost Comparison: A Malaysian SME ScenarioThe Hybrid Approach Many SMEs MissSmall and medium enterprises in Malaysia face a specific challenge: they need enterprise-grade disaster recovery on a budget, but the options can be overwhelming. Azure Backup and Veeam on Azure are the two most common paths, and while both protect your data, they serve fundamentally different use cases, budgets, and operational models. Getting this choice wrong means either overpaying for capabilities you don't need or — worse — discovering during an actual outage that your backup solution can't recover what matters.This article provides a practical, Malaysia-specific comparison to help you decide.The Malaysian SME DR RealityBefore comparing tools, it is worth understanding the constraints most Malaysian SMEs operate under:IT teams of 1-3 people — there is no dedicated backup administrator. Whoever sets up backups also manages the network, the ERP system, and the office Wi-Fi.PDPA compliance is mandatory but enforcement is growing — Malaysia's Personal Data Protection Act 2010 applies if you process personal data, even if you are a small business. Regulators are becoming more active.Hybrid infrastructure is the norm — most SMEs run some workloads on-premises (ERP, accounting) and some in Azure (web apps, email, file shares). The backup solution must bridge both.Ringgit sensitivity — every dollar spent on backup is a dollar not spent on growth. The total cost of ownership must be predictable and justifiable.With these constraints in mind, let us examine the two contending solutions.Azure Backup: The Native OptionAzure Backup is Microsoft's fully managed backup-as-a-service. It is designed to protect Azure-native workloads with minimal configuration and zero infrastructure to maintain.What It CoversAzure Backup natively protects:Azure VMs — full VM-level backups with application-consistent snapshots for Windows (VSS) and file-consistent for Linux.SQL Server on Azure VMs — log, differential, and full backups with point-in-time restore.SAP HANA on Azure VMs — full support including backint integration.Azure File Shares — share-level snapshots with soft delete.On-premises Windows servers — via the Microsoft Azure Recovery Services (MARS) agent.Azure Blob storage — operational backup with point-in-time restore for supported blob scenarios.Azure Database for PostgreSQL — vaulted backup support is available for supported PostgreSQL deployments; confirm the exact server type and region before relying on it for long-term retention.Strengths in the Malaysian ContextZero infrastructure overhead. You do not need to provision, patch, or monitor any backup VMs. The backup engine runs on Microsoft infrastructure. For an IT team of two, this is invaluable — there is simply no time to manage yet another server.Pay-per-use pricing. Azure Backup pricing is based on protected instances plus backup storage consumed. There are no upfront commitments, but the actual monthly cost depends on instance size, retention, storage redundancy, archive tier usage, and whether cross-region restore is enabled.Built-in long-term retention. Azure Backup stores backup data in Microsoft-managed backup storage and supports archive tiering for eligible long-retention recovery points. Use Azure Pricing Calculator for the selected region, redundancy, retention, and archive assumptions rather than relying on a fixed RM/GB estimate.Cross-Region Restore (CRR). If you enable geo-redundant storage and CRR for supported Azure VM backups, Azure Backup lets you restore vault-tier recovery points in the secondary Azure paired region. Do not assume an arbitrary secondary region for new or unpaired regions; confirm CRR support and the available target region for the selected Azure region before using it as the DR design basis.Soft delete protection. Accidentally deleting a backup vault does not lose your data — soft delete holds deleted backup data for 14 days. This prevents the most common data loss scenario: human error.Limitations You Must KnowAzure Backup is not a universal backup platform. The key gaps:No application-aware backup for non-Microsoft applications. Oracle, MySQL (self-managed), PostgreSQL (self-managed), and custom applications on Linux are backed up at the file or VM level only. You cannot do log-truncation-aware Oracle backups through Azure Backup alone.Limited granularity. Restore is VM-level or file-level (for Windows). There is no item-level restore for Exchange, SharePoint, or third-party applications without additional tooling.No built-in cross-cloud restore. You cannot natively restore an Azure Backup to AWS, GCP, or an on-premises hypervisor other than Hyper-V.Immutable vault support available. Azure Backup now supports immutable vault settings with WORM (write once, read many) storage. You can enable and lock immutability on Recovery Services vaults to prevent any operations that could lead to loss of recovery points — even administrators cannot delete backup data once the vault is locked. However, immutability applies at the vault level, not per individual backup item, and operational backups (blobs, files, disks) are not covered. For granular per-item immutability policies, Veeam still offers more flexibility.Veeam on Azure: The Enterprise OptionVeeam Backup for Microsoft Azure is a purpose-built solution for protecting Azure workloads with the full Veeam feature set. It runs as a marketplace application in your Azure subscription.What It CoversVeeam on Azure protects:Azure VMs (Windows and Linux) — agentless, application-consistent backups.Azure SQL Database and Managed Instance — native integration for database backups.Azure Files, Azure SQL, Cosmos DB, and selected Azure-native resources — coverage depends on the Veeam Backup for Microsoft Azure version and licensed edition.Cross-subscription and separate repository designs — useful when you need backup storage separated from the production subscription or management boundary.Hybrid scenarios — unified management with on-premises Veeam deployments via a single console.Strengths for Malaysian SMEs with Complex RequirementsTrue immutability. Veeam supports Azure Blob immutability (WORM policy) at the storage account level. Once a backup is written to immutable storage, not even an administrator with full Azure rights can modify or delete it for the retention period. This is the gold standard for ransomware protection. For Malaysian SMEs worried about the rising threat of ransomware attacks targeting local businesses — including the 2024 wave targeting Malaysian manufacturing firms — this is a significant advantage.Application-aware and scriptable processing. Veeam supports guest processing and application-aware options for supported workloads, and can use pre-freeze/post-thaw scripting patterns for Linux or custom applications. Treat Oracle, MySQL, PostgreSQL, and bespoke application consistency as design items to validate, not as automatic coverage for every deployment.Cross-account restore. You can back up production VMs in one Azure subscription and restore them into a completely separate subscription or account. This protects against the scenario where an attacker compromises your production subscription — your backups live in a separate trust boundary.Unified hybrid platform. If your organization runs Veeam on-premises today (common among Malaysian enterprises that have been using Veeam for years), Veeam Backup for Azure integrates into the same management console. One pane of glass for on-premises, Azure, and even AWS backup. This reduces training overhead and operational complexity.Free edition cap. Veeam Backup for Microsoft Azure Free Edition protects up to 10 instances free of charge, subject to edition limitations. You still pay for the Azure infrastructure, repositories, worker resources, and storage consumed.LimitationsInfrastructure overhead. Veeam deploys worker VMs in your subscription to orchestrate backups. A typical deployment requires at least one worker VM (Standard_B2s or similar, approximately RM 200-300/month) plus storage accounts for backup data. This is operational overhead that Azure Backup does not require.Higher learning curve. Veeam's feature depth means more configuration options — more places to make mistakes. Policies, repositories, proxies, gateways, and backup chains all need to be understood and maintained.Licensing complexity. Veeam licensing has moved to a per-workload instance model with subscription tiers. Understanding what you are paying for — especially across hybrid environments — requires careful reading of the licensing terms. The Free Edition has documented feature limitations, including limitations around certain advanced capabilities such as Azure Files indexing, virtual network configuration backup, and Cosmos DB protection. Validate the current licensing guide before assuming a feature is included.Real Cost Comparison: A Malaysian SME ScenarioThe figures below are an illustrative planning scenario, not a vendor quote. Confirm current prices in Azure Pricing Calculator, Veeam licensing guidance, and the selected Azure region before making a purchasing decision.Let us model a typical Malaysian SME with:6 Azure VMs (4 Windows, 2 Linux) — 2-4 vCPUs each1 SQL Server Standard instance on Azure VM500 GB of protected data across all VMsRetention policy: 30 daily, 12 monthly, 3 yearly1 TB of Azure File SharesAzure Backup Costs (Monthly) Item Cost (RM) VM Backup (6 instances) ~75 SQL Server Backup ~30 Azure File Share Backup ~20 Storage (Standard tier) ~85 Total ~210 Veeam on Azure Costs (Monthly) Item Cost (RM) Veeam Free Edition license 0 Worker VM (B2s) ~90 Managed disks for worker ~30 Backup storage (Standard + archival) ~120 Immutable blob storage ~40 Total ~280 The difference is approximately RM 70/month — not enough to be the deciding factor for most SMEs, but enough that it compounds over three years (approximately RM 2,500).The Hybrid Approach Many SMEs MissThe most pragmatic strategy for Malaysian SMEs is often a hybrid approach:Azure Backup for routine protection of all Azure VMs and File Shares. Set it up in one afternoon, configure CRR, and forget about it.Veeam for critical workloads only — the single SQL Server or ERP database that drives the business. Protect it with application-aware processing and immutable storage.This gives you the simplicity of Azure Backup for 80% of your workloads and the enterprise resilience of Veeam for the 20% that genuinely require it. Total cost: approximately RM 260/month, which is only RM 50 more than Azure Backup alone.Common Pitfalls to AvoidPitfall 1: Never testing restores. Both Azure Backup and Veeam can fail in subtle ways — corrupted restore points, missing network permissions, changed firewall rules. Run a quarterly restore test using a separate Azure subscription or isolated VNet. Schedule it on your calendar now.Pitfall 2: Choosing by feature count instead of workflow. The tool with the longest feature list is not the tool you will actually use every week. If your team is comfortable with Azure Portal and PowerShell, Azure Backup will generate fewer operational surprises. If you already have Veeam on-premises, extending to Azure is the lower-risk choice.Pitfall 3: Ignoring outbound data transfer costs. Restoring large backups from Azure to on-premises or to another region incurs egress charges at approximately RM 0.50/GB (Azure egress from Asia to internet at $0.12/GB). A 500 GB restore could cost approximately RM 250 in egress alone. Account for this in your DR budget.Pitfall 4: Underestimating RPO with Azure Backup. Azure Backup for VMs runs once per day by default. If your SQL database grows by 20 GB daily, the RPO is 24 hours. For many SMEs this is acceptable. For e-commerce platforms processing orders around the clock, it may not be. Veeam can support more frequent policy schedules for selected workloads, but the achievable RPO depends on workload type, snapshot frequency, repository design, and operational constraints.Pitfall 5: Overprovisioning Veeam infrastructure. The Veeam worker VM sizing guide tends to over-recommend. Start with the vendor sizing guidance and a small pilot, then scale worker resources only after measuring backup window, restore time, and repository throughput.Step-by-Step: Setting Up a Basic Azure Backup Policy# Connect to Azure Connect-AzAccount -Subscription "your-subscription-id" # Create a Recovery Services Vault $vault = New-AzRecoveryServicesVault ` -ResourceGroupName "RG-Backup" ` -Name "Vault-MalaysiaWest" ` -Location "malaysiawest" # Set storage replication to Geo-Redundant Set-AzRecoveryServicesBackupProperty ` -Vault $vault ` -BackupStorageRedundancy GeoRedundant # Configure a daily backup policy with 30-day retention $policy = Get-AzRecoveryServicesBackupProtectionPolicy ` -VaultId $vault.ID ` -WorkloadType "AzureVM" $schPolicy = Get-AzRecoveryServicesBackupSchedulePolicyObject ` -WorkloadType "AzureVM" $retPolicy = Get-AzRecoveryServicesBackupRetentionPolicyObject ` -WorkloadType "AzureVM" $retPolicy.DailySchedule.DurationCountInDays = 30 New-AzRecoveryServicesBackupProtectionPolicy ` -VaultId $vault.ID ` -Name "Daily30Day-Retention" ` -WorkloadType "AzureVM" ` -SchedulePolicy $schPolicy ` -RetentionPolicy $retPolicy Step-by-Step: Deploying Veeam Backup for Azure# Deploy Veeam Backup for Azure via Azure Marketplace # Step 1: Open https://portal.azure.com/#create/veeam.veeam-backup-for-azure # Step 2: Select the resource group and VM size for the worker # Step 3: Configure storage and networking # Step 4: Deploy and connect via Veeam UI # # For automated deployment, see the official Veeam documentation: # https://helpcenter.veeam.com/docs/vbazure/guide/deploying_vb.html # After deployment, configure via Veeam UI: # 1. Add Azure account credentials # 2. Create backup policy with immutable storage target # 3. Select VMs to protect # 4. Enable application-aware processing PDPA Compliance: What Both Solutions OfferUnder Malaysia's PDPA, data principals have rights to access, correct, and withdraw consent for their personal data. For backup and DR, the key compliance requirements are:Data residency must be explicitly configured — PDPA does not make a backup tool compliant by default. Select Malaysia West where residency is required, understand whether geo-redundancy or CRR sends data to another jurisdiction, and confirm any cross-border transfer basis with legal counsel.Data lifecycle must be governed — backup retention, legal hold, immutability, and deletion workflows must be aligned with privacy obligations and business recovery requirements. Do not promise instant deletion if immutable retention is enabled.Data must be protected from unauthorized access — use Azure platform encryption, role-based access control, MFA, private endpoints where appropriate, key management, and operational monitoring. Tool configuration matters more than product selection alone.For a deeper dive into PDPA requirements for cloud backup, consult Jabatan Perlindungan Data Peribadi (JDP/PDP) and your legal advisor. No backup tool substitutes for a proper data governance framework.Conclusion: Making the Call Criteria Azure Backup Veeam on Azure Setup time 2-3 hours 4-8 hours Operational overhead Minimal Moderate Ransomware protection Soft delete + immutable vault (vault-level) Full immutability (storage-level) Application-aware backup Strong for supported Microsoft/Azure workloads such as Azure VM, SQL Server on Azure VM, and SAP HANA on Azure VM Broader guest-processing and scripting options, but validate each database/application Cross-cloud restore No Yes Monthly cost (6 VMs) Illustrative ~RM 210 Illustrative ~RM 280 Hybrid management Separate console Unified with on-prem Choose Azure Backup if: all your workloads are Azure-native, you have fewer than 10 VMs, your budget is tight, and you want set-and-forget protection. It is the sensible default for most Malaysian SMEs.Choose Veeam on Azure if: you run hybrid infrastructure, need immutable backups for ransomware protection, must back up non-Microsoft databases, or already use Veeam on-premises. The additional complexity is justified by the additional protection.Consider the hybrid approach if: you want the best of both — Azure Backup for broad coverage, Veeam for critical workloads. It keeps complexity manageable while providing enterprise-grade protection where it matters most.Key TakeawaysAzure Backup is the sensible default — fully managed, pay-per-use, covers 80% of scenarios. Start here.Veeam provides enterprise-grade protection — immutability, application-aware, cross-account restore.The cost difference may be small or material depending on retention and storage design — treat the RM figures above as a worked example, not a quote.Test your restores quarterly — a backup is only as good as its last successful recovery.The hybrid approach works well — Azure Backup for broad coverage, Veeam for critical workloads.Backup and DR architecture is one of the areas where I see Malaysian SMEs either overpaying or under-protecting. If you want a second pair of eyes on your DR strategy — whether Azure-native, Veeam-based, or hybrid — I review these setups regularly. Message me on LinkedIn. --- ## How I Built a Multi-Agent Content Pipeline with Hermes Agent and Ghost Blog URL: https://wenfeng.my/how-i-built-a-multi-agent-content-pipeline-with-hermes-agent-and-ghost-blog-2/ Author: admin Published: 2026-05-10 23:30:57 Modified: 2026-05-10 23:30:57 ⚡ TL;DRRunning a personal brand blog while working full-time as a cloud architect in Malaysia is a time problem I never fully solved — until I stopped trying to do everything myself.The Problem: Content Consistency at ScaleArchitecture OverviewThe Ghost Blog SetupContent Library StrategyThe Hard PartsRunning a personal brand blog while working full-time as a cloud architect in Malaysia is a time problem I never fully solved — until I stopped trying to do everything myself.I have ideas, expertise, and opinions accumulated from years of Azure consulting, migration projects, and solution architecture. But I don't have the 4-6 hours per article it takes to research, write, edit, optimize images, create social posts, and distribute content consistently. The ideas pile up, the drafts languish, and the publishing cadence suffers.So I built a multi-agent content pipeline that does most of the repeatable operational work autonomously. A set of specialized scheduled jobs handles topic research, drafting, SEO review, fact-checking, social media creation, backups, and posting — all orchestrated by Hermes Agent. The blog runs on Ghost (self-hosted on an Azure VM), and I keep the ability to intervene for strategic review while the pipeline enforces automated pre-publish checks.This article walks through the full architecture, the agent responsibilities, the infrastructure setup, the hard-won lessons from production, and the actual cost breakdown.The Problem: Content Consistency at ScaleThe content creation workflow has more steps than most people realise:Research — find trending topics worth writing aboutDecide — pick the right topic for your audience and scheduleWrite — produce a well-structured, technically accurate articleOptimize — SEO titles, meta descriptions, headings, tags, internal linksCreate visuals — diagrams, screenshots, code blocksSocial posts — platform-specific copy for LinkedIn, X/Twitter, and the newsletterPublish — format, preview, schedule, publishDistribute — post to social channels, email subscribersTrack — monitor analytics, update strategyDoing all nine steps manually takes 4-6 hours per article for me. The bottleneck is not the writing itself — it is everything around the writing. Research eats 45 minutes, SEO tweaks eat 30 minutes, and social media creation and scheduling eat another hour.The solution: automate the repeatable work, then gate publishing with review and validation controls.My pipeline automates topic discovery, SEO checks, social content preparation, backups, and distribution. Topic selection and drafting are assisted by agents, while publishing is gated by scheduled validation jobs so an article can be corrected before it goes live. I retain control over creative direction and can override the schedule when needed.Architecture OverviewThe pipeline consists of specialized scheduled jobs, each with a single, well-defined responsibility. No job should overlap heavily with another's job — this is critical for debugging and maintaining the system.Agent Roster Agent Schedule (MYT) Responsibility Topic Researcher Daily, 12:00 AM Adds new content ideas to the topic bank so the draft library stays stocked. Research Scout Daily, 12:15 AM Checks current signals across AI, Azure, cloud migration, Fabric, LLMs, DevOps, and backup/DR. Outputs scored topic suggestions with source URLs. Content Strategist Daily, 1:15 AM Assigns drafted topics from the bank to editorial calendar dates. Rotates between content pillars such as AI Agents, Cloud Architecture, Data Platform, and DevOps. Blog Writer Daily, 2:15 AM The heavy lifter. Writes full 1500-2500 word articles in markdown and updates Ghost drafts through the Admin API. SEO Reviewer Daily, 3:15 AM Audits drafts for title optimization, meta description quality, heading structure, internal linking opportunities, and tag relevance. Fact Checker Daily, pre-publish Validates high-risk technical claims against authoritative sources and patches drafts before publication. Social Writer / Sync Daily, 4:15-5:00 AM Creates and reconciles platform-specific social posts for each editorial entry — LinkedIn, X/Twitter, and Facebook. Ghost Backup Daily, 4:00 AM Exports a MySQL 8 dump to a GitHub repository for disaster recovery. Buffer Poster Daily, 5:30 AM Posts scheduled social items via the Buffer GraphQL API. Retries safely on rate limits. Analytics Reporter Weekly, Sunday 3:00 AM Generates a traffic and SEO performance report for the past week. Most agents run during off-peak hours (12:00 AM - 5:30 AM MYT) using cost-efficient models, with pre-publish validation later in the morning. The staggered schedule prevents API rate-limit collisions — the Blog Writer, SEO Reviewer, Fact Checker, and Publisher do not all mutate the same draft at the same time.Data FlowTopic Researcher / Research Scout -> Topic Bank (file) -> Content Strategist -> Editorial Calendar -> Blog Writer -> Ghost Draft (via Admin API) -> SEO Reviewer -> Fact Checker -> Publisher -> Social Writer / Sync -> Social Calendar -> Buffer Poster -> Buffer API -> Social Platforms Each step depends on the previous one completing. If the Research Scout fails, the Strategist works from the existing topic bank. If the Blog Writer fails, the SEO Reviewer skips that article. This "fail-soft" design was essential for building a system I can trust while I sleep.The Ghost Blog SetupGhost runs as a Docker container on an Azure B-series VM with MySQL 8 as the database container on the same host. Nginx handles reverse proxy and SSL termination via Let's Encrypt, using Cloudflare DNS challenge for certificate renewal. I avoid hard-coding the VM size as a universal recommendation because sizing depends on traffic, backup load, and background agent activity.Why Ghost?Ghost provides the best combination of:Visual editor — clean, distraction-free writing experience when I do manual editsBuilt-in newsletter — no extra Mailchimp or ConvertKit subscription neededFull Admin API — this is the killer feature for agent-driven content creation. Every agent interaction goes through the REST API.Self-hosting — full control over the data, theme, and deployment pipelineMembership system — free and paid subscription tiers built inAlternatives I evaluated included WordPress, Hashnode, and Substack. For my workflow, Ghost was the cleanest fit because its Admin API and self-hosted model make draft creation, metadata updates, image handling, and publishing automation practical without giving up editorial control.Agent-to-Ghost API FlowThe Blog Writer agent generates markdown, converts it to HTML, and sends it to Ghost using the Admin API's HTML source import path. A simplified request looks like this:POST /ghost/api/admin/posts/?source=html Authorization: Ghost {JWT} Content-Type: application/json { "posts": [{ "title": "...", "html": "<p>Article body...</p>", "tags": [{"name": "AI Agents"}, {"name": "Hermes Agent"}], "status": "draft", "feature_image": "...", "meta_title": "...", "meta_description": "..." }] } For updates, Ghost requires optimistic locking via the post's latest updated_at value, so the SEO Reviewer and Fact Checker read the draft first and then issue a PUT with the current timestamp plus the corrected fields. Publishing is a status transition from draft to published after the pre-publish checks pass.I built a small Python helper library that handles: - JWT token generation with the correct hex-decoded Admin API secret - Markdown-to-HTML conversion for Ghost's ?source=html import path - Retry logic with exponential backoff for API rate limits - Image upload via the Ghost Admin API image endpointContent Library StrategyThe pipeline maintains a content library of 20-30+ pre-written articles in draft status. This solves the "what do I publish this week?" problem permanently.Research accumulates topics in a topic bank file. The Content Strategist assigns priorities based on: - Which content pillar hasn't had a post recently - Seasonal relevance (e.g., Microsoft Ignite recap in November) - Topic freshness (trending vs evergreen)The Blog Writer drafts articles from the bank, the library grows, and by the time a publish date arrives, the article has already gone through SEO review and pre-publish fact-checking. On publish day, the publisher can move the validated draft live and the social posts fire automatically.Why 20-30 articles ahead? Buffer. If I am sick, travelling for a client engagement, or buried in a migration project, the pipeline keeps publishing from the library. I don't miss a week, and the site keeps generating traffic.Topic Bank LifecycleResearched -> In Bank -> Assigned -> Draft Created -> SEO Reviewed -> Published Every topic carries full metadata: source URL, suggested title, target pillar, estimated word count, and creation date. This makes the system auditable — I can see exactly where each article came from and how it progressed.The Hard PartsNothing worked on the first try. Here are the biggest gotchas I hit in production and how I resolved them.1. Ghost Admin API Authentication (The Hex vs Base64 Trap)Ghost Admin API uses hex-encoded secrets for JWT generation — not base64. This is the most common gotcha, and the Ghost documentation could be clearer about it.# WRONG — this silently fails with 401 import base64 secret = base64.b64decode(ghost_admin_key) # RIGHT — hex decode secret = bytes.fromhex(ghost_admin_key.split(':')[1]) Use bytes.fromhex() in Python or xxd -r -p in bash. The JWT itself is standard HS256, but the input secret must be raw bytes from hex, not base64.2. Ghost Editor Content FormatGhost's current Admin API represents post content around a Lexical editor field, and the API also supports an HTML import workflow with ?source=html. That means I do not need to hand-build editor JSON for normal article drafts.The safer pattern for agents is:POST /ghost/api/admin/posts/?source=html with an html field in the JSON payload. Ghost converts the HTML into its internal editor representation. This keeps the agent code simpler and avoids brittle conversions whenever Ghost changes editor internals.3. Cloudflare Bot Fight ModeWhen Cloudflare Bot Fight Mode is enabled, all programmatic API calls receive 403 challenges — even valid Ghost Admin API requests from the agent. WAF skip rules do not override it.The fix: disable Bot Fight Mode for the Ghost subdomain. Alternatively, route API calls through an SSH tunnel to bypass Cloudflare entirely, but that adds latency and complexity.4. Content Quality at ScaleThe biggest risk with automated content pipelines is quality degradation. If the system publishes mediocre articles, it damages your brand faster than not publishing at all.My mitigations: - Require real code examples — every article must include at least one CLI command, config file, or code sample - Emphasize practical content — no fluff, no filler paragraphs - Human review gate — no article publishes without my review - SEO Reviewer agent — catches structural issues before they reach me - Random spot-checks — I read 1 in 5 articles end-to-end to audit quality driftCost BreakdownHere is the actual monthly cost of running this pipeline: Component Cost Hermes Agent (cron scheduler) $0 (open source, self-hosted) Ghost hosting (Azure VM, storage, backup overhead) Roughly low tens of USD/month; verify with Azure Pricing Calculator for the selected region and VM size Domain (wenfeng.my/) ~$12/year (~$1/mo) Cloudflare (free tier) $0 Buffer (free tier) $0 LLM inference (Hermes-configured cost-efficient model, currently mimo-v2.5 in my environment) ~$5-8/mo Total Usually low tens of USD/month for this small self-hosted setup, excluding labour and optional paid social tools The important point is not the exact monthly number; it is the cost profile. The core platform is inexpensive to run compared with manual production time, but the final figure depends on VM sizing, region, model usage, backup retention, and whether paid social scheduling features are required.What's NextI am currently working on:Image generation agent — generates featured images and diagrams, then uploads approved assets to GhostTranslation agent — translates select articles to Malay and Chinese for regional audiencesPerformance scraper — tracks Google Search Console data weekly to refine the SEO Reviewer agent's rulesKey TakeawaysSingle-responsibility agents — each agent has one job, no overlap, no shared context. Debugging is trivial because the failure is always in one agent.The topic bank is the most important component — invest in research automation first. A pipeline with great research and mediocre writing outperforms one with great writing and no research.Ghost Admin API is the integration point — programmatic draft creation and updates via the API make agent-driven blogging practical. Use the documented HTML import path or Lexical field rather than relying on outdated editor assumptions.Automate the repeatable, validate the critical — research, drafting, SEO, and distribution can be automated. Strategic direction and high-risk technical claims still need review or automated fact-checking gates.Off-peak scheduling saves money — agents run while you sleep. Staggered schedules prevent API collisions and keep costs minimal.Fix Bot Fight Mode before you deploy — this single issue cost me two nights of debugging. Disable it on your Ghost subdomain from day one.Build fail-soft — each agent should work with whatever state the previous agent left, even if that state is "nothing new today."The goal is not to replace human creativity — it is to remove the operational friction that prevents consistent publishing. Agents handle research, drafts, SEO, fact-checking, and distribution. I handle strategic direction and intervene where judgement matters. The blog has gone from 1 post per month to 4-6 posts per month, and the quality has improved because attention goes to the posts that need actual thought rather than the mechanics of publishing.I build multi-agent content pipelines and AI automation systems for Malaysian enterprises. If your organisation is exploring agent-based automation or wants to streamline content operations, let's talk on LinkedIn. --- ## FabCon and SQLCon 2026: Why SQL and Fabric Are Becoming One Operational Data Platform URL: https://wenfeng.my/fabcon-and-sqlcon-2026-why-sql-and-fabric-are-becoming-one-operational-data-platform/ Author: admin Published: 2026-05-09 23:30:00 Modified: 2026-10-04 15:30:32 ⚡ TL;DRFor years, enterprise data architecture has been split into separate worlds. SQL teams managed operational databases. Analytics teams built warehouses and lakehouses. BI teams maintained semantic models and reports.The old operating model is too fragmentedWhat an integrated SQL and Fabric platform meansArchitecture decisions that matterExample: customer risk platformMigration approach for existing SQL estatesFor years, enterprise data architecture has been split into separate worlds. SQL teams managed operational databases. Analytics teams built warehouses and lakehouses. BI teams maintained semantic models and reports. AI teams copied data into vector stores or notebooks. Each world had its own tools, security model, performance assumptions, and operating rhythm.The direction signalled by FabCon and SQLCon 2026 is that this separation is becoming harder to justify. Microsoft is positioning SQL and Fabric as parts of one operational data platform: transactional SQL data, analytical storage in OneLake, governed semantic models, real-time intelligence, and AI agents that can use business context from the same foundation.That does not mean every SQL Server, Azure SQL Database, Synapse workspace, and Fabric item should be merged overnight. It means architects should stop designing SQL modernisation, lakehouse analytics, and AI enablement as unrelated programmes. The practical goal is not a bigger platform diagram. The goal is to reduce duplicated data movement, inconsistent security, and disconnected business definitions.The old operating model is too fragmentedA typical enterprise pattern looks like this:Operational SQL systems -> nightly ETL -> warehouse or lake -> semantic model -> dashboard -> analyst export -> AI proof of concept This works, but it is slow and brittle. The business asks a question, the data team checks whether the pipeline contains the right data, the BI team checks whether the metric is defined, and the AI team asks for another copy. By the time the answer arrives, the operational situation may already have changed.The fragmentation also creates security problems. A customer record may be protected in SQL, exported into a lake with different controls, reshaped into a Power BI dataset with another permission model, and then copied into a notebook or agent tool. Each copy is a new exposure path. Each layer needs lineage, retention, ownership, and audit.The better pattern is to make SQL, Fabric, and AI share a governed data plane wherever practical.What an integrated SQL and Fabric platform meansAn integrated platform does not mean SQL disappears. SQL remains the system of record for many enterprise applications. Fabric does not replace every database. Instead, Fabric becomes the analytical and contextual layer around the operational estate.A practical reference model is:Business Applications - ERP, CRM, portals, custom apps - Azure SQL / SQL Server / SQL MI | | mirroring, shortcuts, pipelines, events v Microsoft Fabric - OneLake data products - Lakehouse / warehouse / SQL database in Fabric - Semantic models and shared metrics - Real-Time Intelligence and Eventhouse - Fabric IQ / context layer | v Consumers - Power BI reports - Data science notebooks - Data agents - Operations agents with approval gates The key design principle is that operational data should become analytically useful without creating uncontrolled copies. Mirroring and OneLake integration are valuable because they reduce the gap between the operational source and the analytical surface. Semantic models are valuable because they define business meaning once. Real-time capabilities are valuable because many operational decisions cannot wait for yesterday's refresh.Architecture decisions that matterThe first decision is ownership. A SQL database may be owned by an application team, but the Fabric data product derived from it needs a named data owner, technical owner, classification, quality expectation, and support model. Without ownership, integrated platforms become dumping grounds.The second decision is data movement. Not every table needs to be mirrored. Start with domains where the business value is clear: customer, subscription, invoice, case, order, asset, telemetry, and cost. For each domain, decide whether the data should be mirrored near real time, loaded in batches, exposed through a shortcut, or kept inside the source system with query federation. Use the lowest-complexity pattern that satisfies the requirement.The third decision is security inheritance. A user who cannot access sensitive salary data in SQL should not get it through a Fabric report, notebook, or agent. Design row-level security, column-level security, workspace boundaries, sensitivity labels, and audit logging before scaling consumption.The fourth decision is semantic consistency. If revenue means one thing in SQL, another thing in Power BI, and another thing inside an agent prompt, the platform will lose trust. Define core metrics centrally and reuse them.Example: customer risk platformConsider a Malaysian enterprise trying to identify customers at risk of churn. The relevant information may sit across SQL databases, CRM, support systems, invoice ageing, and product usage telemetry. The old approach is to export everything into a warehouse and build dashboards. That is useful, but not always timely.A Fabric-centred approach would mirror selected operational tables, stream usage events into Real-Time Intelligence, curate a customer data product in OneLake, define a semantic model for active customer, recurring revenue, open incident, SLA breach, and risk score, then expose that model to reports and agents.The agent should not directly query every raw table. It should use curated data products and governed metrics. If it recommends action, such as creating a CRM task or notifying an account manager, that action should pass through a policy or approval gate.SQL customer + invoice data Support cases Usage events | v Curated customer data product in Fabric | +--> Power BI risk dashboard +--> Data agent answer with evidence +--> Operations workflow recommendation This is where SQL and Fabric become operationally interesting. The platform is not only reporting what happened. It is helping the business respond with context.Migration approach for existing SQL estatesDo not start by moving every database. Start with a portfolio assessment.Group SQL workloads into four categories. First, systems of record that should remain operational databases. Second, reporting-heavy databases that need better analytical scale. Third, legacy marts that can be rationalised into Fabric data products. Fourth, experimental copies that should be retired.For each candidate domain, document source owner, data classification, refresh requirement, consumer list, integration pattern, and target security controls. Then build one end-to-end slice: operational source to OneLake, semantic model, report, and agent-safe consumption surface.A simple implementation checklist:1. Select one high-value business domain. 2. Identify source SQL tables and data owner. 3. Classify sensitive columns before ingestion. 4. Choose mirroring, pipeline, or shortcut pattern. 5. Define curated tables and semantic metrics. 6. Apply row/column security and sensitivity labels. 7. Validate with one report and one agent-safe query path. 8. Monitor freshness, failures, usage, and cost. This phased approach avoids a common mistake: launching Fabric as a new central platform while leaving business definitions and security controls behind.Pitfalls to avoidThe first pitfall is treating Fabric as just another destination for SQL exports. If the governance model does not improve, the organisation only gets a newer silo.The second pitfall is ignoring operational latency. Some use cases are fine with daily refresh. Others require event-driven updates. Match the ingestion pattern to the decision being supported.The third pitfall is giving agents raw database access. Agent access should be mediated through curated data products, semantic models, and purpose-built tools. Raw access increases the chance of data leakage and incorrect interpretation.The fourth pitfall is underestimating cost ownership. Mirroring, storage, compute, real-time processing, and agent queries all have cost profiles. FinOps should be part of the design from the beginning.Key takeawaysSQL and Fabric should be planned as one enterprise data platform, not separate modernisation tracks.Keep operational SQL systems where they belong, but expose selected data into governed Fabric data products.Security, semantic consistency, and ownership matter more than the ingestion technology.AI agents should consume curated and governed context, not uncontrolled database copies.Start with one business domain and prove the operating model before scaling. --- ## OneLake Security GA: Designing Row-Level, Column-Level, and Agent-Safe Access for Fabric URL: https://wenfeng.my/onelake-security-ga-designing-row-level-column-level-and-agent-safe-access-for-fabric/ Author: admin Published: 2026-05-08 23:30:00 Modified: 2026-10-04 15:30:42 ⚡ TL;DRFabric adoption often begins with ingestion, lakehouses, Power BI reports, and executive dashboards. Security tends to arrive later, when more teams ask for access and the platform owner realises that workspace permissions alone are not enough.Why workspace permissions are not enoughRow-level and column-level thinkingReference design for Fabric domainsAgent-safe access is differentPractical design stepsFabric adoption often begins with ingestion, lakehouses, Power BI reports, and executive dashboards. Security tends to arrive later, when more teams ask for access and the platform owner realises that workspace permissions alone are not enough. That sequence is risky. Once sensitive data is already copied, modelled, and consumed by many reports, retrofitting governance becomes difficult.OneLake security becoming generally available is important because it pushes Fabric closer to an enterprise-grade security model. Row-level and column-level controls at the data layer make it easier to expose shared data products without giving every consumer the same broad access. That matters even more now that data agents and AI workflows can sit on top of Fabric.The main point for architects is simple: do not design Fabric security only for human report readers. Design it for analysts, engineers, applications, notebooks, semantic models, and AI agents from the beginning.Why workspace permissions are not enoughWorkspace-level access is useful for collaboration, but it is too coarse for many enterprise data domains. A finance workspace may contain revenue, margin, payroll allocation, customer credit terms, and forecast data. A sales manager may need customer revenue for her region, but not payroll data. A data scientist may need aggregated patterns, but not personally identifiable customer details. An agent may need to answer questions about trends, but not reveal confidential rows to a user who is not allowed to see them.If the only control is workspace membership, platform teams often respond by creating many duplicated workspaces and datasets. That fragments ownership and creates inconsistent definitions. Fine-grained controls are a better answer.A mature model combines several layers:Tenant and capacity controls -> domain workspaces -> lakehouse / warehouse permissions -> OneLake roles -> row-level and column-level policies -> semantic model security -> application or agent access controls No single layer is enough. The goal is defence in depth.Row-level and column-level thinkingRow-level security answers: which records can this identity see? Examples include region, business unit, legal entity, customer portfolio, project, or data residency boundary.Column-level security answers: which attributes can this identity see? Examples include salary, national ID, email address, margin, bank account, medical notes, or contract pricing.Object-level controls answer: which tables, files, folders, or data products can this identity use at all?These controls should map to business policy, not only technical convenience. If the business cannot explain who should see a field and why, the platform team cannot reliably secure it.Reference design for Fabric domainsFor Malaysian enterprises, I normally recommend a domain-oriented pattern. Each major domain has a clear workspace boundary, data owner, classification, and consumption model.Fabric Domain: Customer - Raw/source-aligned zone: restricted to data engineering - Curated data product: governed access for analytics - Semantic model: approved business metrics - Agent-safe view: reduced fields, evidence references, policy checks The raw zone should be tightly restricted. Most users and agents should not need it. The curated zone should expose business-ready tables with consistent naming, data quality checks, and documented ownership. The semantic model should define metrics. Agent-safe views should remove unnecessary sensitive columns and include only the data needed for the use case.This pattern reduces two common risks: analysts accidentally using raw data, and agents being granted broad access because nobody built a safer surface.Agent-safe access is differentAI agents change the security discussion. A human analyst usually asks for access to a dataset or report. An agent may receive a natural-language question, decide which tool to call, query data, summarise results, and possibly trigger an action. That creates a wider path for accidental disclosure.A user should not be able to bypass permissions by asking an agent. If the user cannot see customer margin in a report, the agent should not reveal it in a generated answer. This means agent access must consider both the agent identity and the end-user identity.A safer pattern is:User -> Agent Gateway -> user identity and policy check -> approved Fabric tool -> agent-safe view / semantic model -> answer with source references Do not give the agent a superuser identity and rely on prompt instructions to hide data. Prompts are not access controls. Use Entra ID, Fabric permissions, data-layer policies, logging, and approval gates.Practical design stepsStart with classification. Identify confidential, regulated, personal, commercial, and public fields. Do this before broad onboarding. Sensitive data that is not identified cannot be protected consistently.Next, define access personas. For example: data engineer, domain analyst, executive reader, regional manager, external auditor, data scientist, application service, and AI agent. Each persona needs a clear reason to access each data product.Then build policy tables or attributes for row-level filtering. Common examples include region, branch, department, tenant, legal entity, and customer ownership. Avoid hardcoding security filters in many reports. Centralise the mapping where possible.For column controls, create safe views where needed. Even when column-level security exists, safe views make consumption easier and reduce mistakes.Finally, instrument access. Log who queried what, through which tool, and from which workload. For agent workloads, log user identity, agent identity, requested operation, data source, and generated response metadata.Example policy modelA simple policy mapping could look like this:{ "principal": "sales-manager-my-north", "allowed_regions": ["MY-North"], "allowed_columns": ["customer_id", "customer_name", "revenue", "case_count", "renewal_date"], "restricted_columns": ["gross_margin", "credit_limit", "national_id"], "agent_allowed": true, "write_actions_allowed": false } The exact implementation may use Entra groups, Fabric roles, security tables, semantic model rules, or application policy. The architecture principle is more important than the file format: policy must be explicit, reviewable, and testable.Verification and testingSecurity configuration is not complete until it is tested. Create test identities for each persona and validate expected access. Use negative tests: confirm that a regional user cannot see another region, that sensitive columns are masked or blocked, and that an agent cannot reveal restricted fields indirectly.Test through every consumption path. A policy that works in a report but fails in a notebook or Spark job is incomplete. A policy that works for direct users but fails for service principals is also incomplete.A practical checklist:- Test direct lakehouse or warehouse access. - Test Power BI semantic model access. - Test notebook or Spark access where used. - Test API/application access. - Test agent access with allowed and restricted prompts. - Review audit logs after each test. Pitfalls to avoidThe first pitfall is overusing workspace duplication. It may feel simple at first, but it creates long-term data sprawl.The second pitfall is granting agents broad read access. Build agent-specific views and tools instead.The third pitfall is ignoring service identities. Pipelines, notebooks, applications, and agents often use service principals or managed identities. They need the same review as human users.The fourth pitfall is failing to document business ownership. Security teams can enforce controls, but the business must decide who is allowed to see sensitive data.Key takeawaysOneLake security makes fine-grained Fabric governance more practical, but it does not replace architecture discipline.Use workspace, object, row, column, semantic model, and application controls together.Design agent-safe access surfaces instead of giving agents raw or privileged data access.Test security through every consumption path, not only Power BI reports.Treat data ownership and classification as production prerequisites, not later documentation. --- ## Malaysia West Azure Region: Architecture Planning Guide for Enterprises URL: https://wenfeng.my/mw-azure-architecture-planning-guide/ Author: admin Published: 2026-05-08 00:39:20 Modified: 2026-10-04 15:30:42 ⚡ TL;DRMalaysia West changes the Azure architecture conversation for Malaysian enterprises. For many years, Malaysian organisations commonly designed Azure workloads around Southeast Asia, East Asia, Australia, or another regional hub, then justified the tr...Practical Executive SummaryArchitecture ContextMalaysia West-Specific ConsiderationsDesign RecommendationsRisks and ConstraintsMalaysia West changes the Azure architecture conversation for Malaysian enterprises. For many years, Malaysian organisations commonly designed Azure workloads around Southeast Asia, East Asia, Australia, or another regional hub, then justified the trade-offs around latency, data residency, support operations, and disaster recovery. A local Azure public cloud region in Kuala Lumpur gives architects a stronger in-country option.That does not mean every workload should move immediately. It also does not mean every Azure service, SKU, feature, database tier, AI capability, GPU family, SAP-certified VM, or analytics platform is automatically available in Malaysia West. Region selection is an architecture decision, not a portal dropdown selection.This guide is written for CIOs, CTOs, infrastructure leaders, cloud architects, and platform engineering teams planning enterprise workloads on Azure in Malaysia. The focus is practical: what to validate, what to design, and what not to assume.Implementation note: validate current Azure Malaysia West service availability, SKU availability, quota, availability-zone support, and feature behaviour directly against Microsoft products-by-region and service-specific Microsoft Learn pages before implementation or publication of a final design.Practical Executive SummaryMalaysia West is a significant improvement for Malaysian Azure architecture because it can support local residency objectives, reduce dependency on cross-border hosting for some workloads, and provide a closer platform for Malaysia-based users and operations teams. Microsoft Learn lists Malaysia West with programmatic name malaysiawest, physical location Kuala Lumpur, geography Malaysia, availability-zone support at the region level, and paired region as N/A. Microsoft has also announced Malaysia West as its first cloud region in Malaysia; for architecture purposes, use the Microsoft Learn region list as the authoritative source for current availability-zone status.The executive takeaway is straightforward:Use Malaysia West when local residency, Malaysia user latency, regulatory posture, or business preference justifies an in-country Azure region.Do not assume all services are available. Malaysia West service and SKU availability must be checked service by service.Treat availability zones as an architecture capability, not automatic high availability. Each service still needs validation for zonal or zone-redundant deployment support.Do not assume paired-region disaster recovery. Microsoft Learn lists the paired region as N/A, so DR region selection must be deliberate.Build an Azure landing zone first, then onboard workloads. Direct production deployment without governance, identity, network, security, monitoring, backup, and DR foundations is a weak operating model.My recommendation: for enterprise workloads, use Malaysia West as the primary region only after completing a region-fit assessment, landing zone review, service availability validation, and DR design. For critical workloads, design and test failover to an approved secondary region such as Southeast Asia, East Asia, Australia East, or another region selected based on compliance, latency, service availability, and business footprint.Architecture ContextA regional Azure deployment is not only about compute and storage placement. It affects identity, connectivity, DNS, security boundaries, data flows, monitoring, backup, recovery, procurement, support, and operational ownership.For a Malaysian enterprise, the typical architecture questions are:Which workloads genuinely need to run in Malaysia West?Which workloads can remain in Southeast Asia or another existing region?Which data must remain in Malaysia, and which data can be replicated cross-border under approved controls?What is the approved DR region if Malaysia West has no paired region listed?Which Azure services and SKUs are confirmed available in Malaysia West today?Which platform services should be centralised, and which should be workload-specific?How will security, monitoring, backup, and incident response work during a regional incident?The correct starting point is an Azure landing zone. Microsoft’s Cloud Adoption Framework positions Azure landing zones as a structured approach covering billing and tenant decisions, identity and access management, management groups and subscriptions, network topology, security, management, governance, and platform automation.For Malaysia West, that landing zone needs explicit regional policy. Common examples include:Allowed regions: Malaysia West plus approved DR and shared-services regions.Required tags: owner, environment, cost centre, data classification, RTO, RPO, business criticality.Diagnostic settings: required where supported for critical resources.Private endpoint policy: required for sensitive PaaS services where technically feasible.Public IP policy: denied by default except approved ingress, firewall, NAT, or platform services.Backup and retention policy: based on workload criticality and regulatory requirements.This prevents Malaysia West adoption from becoming an uncontrolled regional sprawl exercise.Malaysia West-Specific Considerations1. Availability zones exist at the region level, but service support is still specificA common mistake is to treat region-level availability-zone support as if every workload automatically becomes highly available. That is not how Azure works. Some services support zonal deployment, some support zone-redundant deployment, some support both, and some depend on tier, SKU, configuration, or region-specific availability.Architects should validate:Whether the required service supports availability zones in Malaysia West.Whether the required SKU or tier supports zonal or zone-redundant deployment.Whether the application can tolerate zone failure through retries, health probes, stateless scaling, queueing, and data replication.Whether dependencies such as firewall, load balancer, NAT, database, storage, DNS, and secrets management are also resilient.Availability zones reduce certain single-region risks. They do not replace application resilience or disaster recovery.2. Paired region is listed as N/AMicrosoft Learn lists Malaysia West paired region as N/A. That does not make Malaysia West unsuitable, but it changes the DR conversation. Region pairs are used by some Azure services for platform sequencing or geo-replication behaviour, but they do not automatically provide a complete DR solution. Many regions are nonpaired, and resilient architectures can still be built using paired or nonpaired regions.For Malaysia West, the secondary region should be chosen deliberately. Typical candidates for evaluation include Southeast Asia, East Asia, Australia East, or another region aligned to the organisation’s operating footprint. The right answer depends on:Data residency and regulatory interpretation.RTO and RPO requirements.Network latency and replication behaviour.Application architecture and database replication support.Service availability in both primary and secondary regions.Cost of standby capacity, data transfer, and operational testing.Do not write “Malaysia West DR goes to Singapore by default” as an architecture rule. Southeast Asia may be a practical candidate for many Malaysian enterprises, but it must still be validated.3. Service availability is the gating itemThe most important Malaysia West planning step is service validation. A high-level Azure region page is not enough. Before committing a workload, confirm the exact service, SKU, tier, feature, quota, redundancy option, and support limit.Examples that require explicit validation include:VM families, GPU families, and quota limits.AKS versions, node pool VM families, and networking modes.App Service, Functions, and integration features.Azure SQL Database, SQL Managed Instance, PostgreSQL Flexible Server, and MySQL options.Storage redundancy choices and premium performance tiers.Azure Firewall, Application Gateway, VPN Gateway, ExpressRoute Gateway, and NAT Gateway.Azure Monitor Logs, Application Insights, Azure Backup, and Azure Site Recovery.Azure AI services, Azure OpenAI, Azure Machine Learning, Microsoft Fabric, and SAP-certified infrastructure.Treat any unverified Malaysia West service claim as verify_before_publish or verify_before_implementation.4. Compliance is not automaticMalaysia West can support in-country data residency objectives, but deploying into Malaysia West does not automatically satisfy PDPA, Bank Negara Malaysia, government, healthcare, telecommunications, or internal governance requirements. Compliance depends on architecture, data classification, contracts, encryption, access control, logging, operational procedures, support model, third-party access, and cross-border data flows.Architects should involve legal, compliance, risk, and data protection stakeholders before using Malaysia West as a compliance conclusion.Design RecommendationsStart with landing zone foundationsI do not recommend placing production workloads directly into a manually created subscription. The baseline should include:Management group hierarchy aligned to platform, landing zones, sandbox, and decommissioned scopes.Separate subscriptions for connectivity, management/security, and application landing zones where scale or control boundaries require it.Azure Policy guardrails for region, public access, diagnostics, private endpoints, tagging, and encryption.Role-based access control mapped to platform teams, workload teams, security teams, and operations.Infrastructure as Code for repeatable deployment and change control.For most enterprises, a practical structure is:Platform connectivity subscription for hub network, firewall, VPN or ExpressRoute gateways, DNS, and routing.Platform management subscription for monitoring, automation, update management, security tooling, and backup governance.Application landing zone subscriptions separated by production, non-production, and regulated workloads.DR subscriptions or resource groups in the selected secondary region where recovery isolation is required.Design the network deliberatelyMalaysia West deployments should normally use either hub-and-spoke or Azure Virtual WAN. The choice depends on scale, number of sites, routing complexity, firewall model, and operating team maturity.Hub-and-spoke remains a strong pattern when the environment has a manageable number of VNets, centralised inspection, and predictable routing. Azure Virtual WAN becomes more attractive when there are many branches, multiple regions, SD-WAN integration, or a need for managed global transit.Key networking decisions include:Private IP address plan that avoids overlap with on-premises and DR regions.DNS forwarding between on-premises, Azure private DNS zones, and workload VNets.Private Link design and private endpoint ownership.Internet ingress pattern: Application Gateway, Azure Front Door, firewall, or partner WAF.Egress control: Azure Firewall, NAT Gateway, NVA, or controlled direct egress.VPN versus ExpressRoute based on criticality, throughput, SLA, lead time, and cost.For deeper network pattern comparison, see Azure Virtual WAN vs Hub-and-Spoke: Which Network Architecture for 2026?.Build security and identity into the first releaseThe minimum security baseline should include:Microsoft Entra ID privileged access model.MFA, Conditional Access, Privileged Identity Management, and monitored break-glass accounts.Subscription-level RBAC aligned to operating responsibilities.Defender for Cloud baseline and secure score governance.Key Vault with managed identities and restricted access.Private endpoints for sensitive PaaS services where feasible.Diagnostic logs, activity logs, and security alerts routed to the agreed monitoring platform.Encryption controls and key management decisions based on data classification.Do not defer these controls to “phase two” if production workloads are in scope.Treat DR as a design workstream, not an appendixFor critical workloads, the DR approach should be defined before production go-live. The design should include:Approved secondary region.Workload dependency map.Database replication or restore pattern.Application deployment approach in secondary region.DNS or traffic failover method.Secrets, certificates, and identity availability.Backup vault strategy and restore testing.Runbook ownership, testing frequency, and acceptance criteria.If the workload has strict RTO/RPO, backup-only recovery is usually not enough. Consider active-passive or active-active patterns where the application and cost model justify it.Validate cost by environmentMalaysia West cost planning should not be a single monthly estimate. Build the BOM by environment:Production.UAT or staging.Development.DR or standby.Shared platform services.Security, monitoring, and backup.The main cost drivers are compute family, reservations or savings plans, storage redundancy, cross-region replication, logging ingestion, backup retention, network gateways, firewall throughput, and standby DR capacity. Pricing can vary by region and change over time, so use the Azure Pricing Calculator and current Microsoft pricing before proposal or procurement.Risks and ConstraintsRisk: assuming all Azure services are available in Malaysia WestImpact: architecture rework, project delay, unsupported designs, or forced region changes.Recommendation: make service availability validation a formal architecture gate.Risk: treating availability zones as automatic HAImpact: applications may still fail because dependencies, state, health checks, or deployment patterns are not zone-resilient.Recommendation: validate service-level AZ support and test failure behaviour.Risk: no listed paired regionImpact: platform defaults may not match DR expectations.Recommendation: choose a secondary region deliberately and design application-level or service-level recovery.Risk: compliance overstatementImpact: false assurance to executives, auditors, or regulators.Recommendation: position Malaysia West as supporting residency objectives, not guaranteeing compliance.Risk: operational under-designImpact: the environment may deploy successfully but fail under incident, audit, backup, or change-management pressure.Recommendation: define monitoring, backup, security operations, access review, and DR testing before production.Checklist and Decision TableUse this as a minimum readiness checklist before committing production workloads to Malaysia West:Confirm required Azure services are available in Malaysia West.Confirm SKU, tier, feature, and quota availability, not only the service name.Confirm availability-zone support for each critical service and dependency.Confirm the approved DR region and replication pattern.Confirm whether any data may replicate outside Malaysia and under what controls.Confirm landing zone management group and subscription design.Confirm network topology: hub-and-spoke, Azure Virtual WAN, VPN, ExpressRoute, DNS, and egress.Confirm identity, RBAC, privileged access, and break-glass process.Confirm monitoring, alerting, log retention, backup, restore testing, and DR runbooks.Confirm cost estimate by production, non-production, DR, and shared services.Decision table:Use Malaysia West as primary when Malaysia residency, local user latency, stakeholder preference, or regulatory posture justifies it and required services are validated.Use Southeast Asia or another region as primary when required services, SKUs, or operational maturity are not yet validated for Malaysia West.Use Malaysia West plus secondary DR region when the workload is business-critical and has defined RTO/RPO.Use single-region Malaysia West only for workloads with acceptable outage tolerance, tested backup restore, and no strict cross-region recovery requirement.Delay migration when service availability, compliance interpretation, network connectivity, or DR ownership is unresolved.Related ReadingAzure Virtual WAN vs Hub-and-Spoke: Which Network Architecture for 2026?Azure Local for Malaysian Enterprises: When Hybrid Beats Cloud-OnlyAzure Landing Zones for AI WorkloadsMicrosoft Fabric vs Azure Synapse: When to Use Which in 2026Microsoft Learn: List of Azure regionsAzure products available by regionMicrosoft Learn: Azure landing zonesCTA: Azure Architecture Review for Malaysia WestPlanning workloads for Azure Malaysia West? Get an architecture review before committing production workloads.I can help validate:Region fit and workload placement.Azure landing zone design.Hub-and-spoke or Azure Virtual WAN topology.Service, SKU, feature, quota, and availability-zone assumptions.Security, identity, monitoring, backup, and operational readiness.DR strategy and secondary-region selection.Cost model by environment and support model.The strongest enterprise approach is simple: landing zone first, service validation second, workload architecture third, and DR testing before production. Malaysia West gives Malaysian organisations an important new platform option, but the architecture still needs to be designed, validated, and operated with discipline. --- ## LLM Deployment on Azure: Azure OpenAI vs. Self-Hosted Open-Source Models — A Cost and Architecture Decision Framework URL: https://wenfeng.my/llm-deployment-on-azure-azure-openai-vs-self-hosted-open-source-models-a-cost-and-architecture-decision-framework/ Author: admin Published: 2026-05-07 23:31:20 Modified: 2026-10-04 15:30:43 ⚡ TL;DREvery enterprise AI architect I speak with in Malaysia is asking the same question: "Should we pay for Azure OpenAI or run open-source models ourselves on Azure GPU VMs?"IntroductionThe Problem: Two Very Different Paths to the Same DestinationAzure OpenAI Pricing (2025 Realities)Self-Hosted Cost RealityThe Decision MatrixIntroductionEvery enterprise AI architect I speak with in Malaysia is asking the same question: "Should we pay for Azure OpenAI or run open-source models ourselves on Azure GPU VMs?"It sounds straightforward, but the answer is anything but. The decision doesn't just live in a spreadsheet — it ripples through your data governance posture, your latency profile, your team's headcount requirements, and ultimately your ability to move fast when the market shifts.In this article, I'll walk through a structured decision framework I've developed over the past 18 months working with Malaysian enterprises deploying LLMs on Azure. You'll get concrete cost estimates (in MYR, using approximate mid-2025 exchange rates because regional pricing matters), architecture patterns you can adapt, and a decision matrix that cuts through the FUD.---The Problem: Two Very Different Paths to the Same DestinationLet's start by defining the two options clearly.Azure OpenAI Service is Microsoft's managed offering. You get API access to GPT-4o, GPT-4o-mini, the o-series reasoning models, and embeddings models. Microsoft handles hardware provisioning, patching, load balancing, and model updates. You consume it as a managed API with a pay-per-token pricing model. Self-hosted open-source LLMs on Azure GPU VMs means you provision your own GPU instances (NC-series, ND-series), deploy an inference server (vLLM, TGI, llama.cpp), and run models like Llama 3, Mistral, Qwen, or Phi. You pay for compute by the hour regardless of usage, and you are responsible for the full operations stack.These two paths produce the same output — text from a large language model — but the economics, operational burden, and architectural implications could not be more different.---Azure OpenAI Pricing (2025 Realities)As of mid-2025, here are the approximate pricing tiers for Azure OpenAI in Southeast Asia (Singapore region, since Azure OpenAI is typically deployed in the Southeast Asia (Singapore) region for Malaysian customers): Model Input (per 1M tokens) Output (per 1M tokens) Cached Input GPT-4o $2.50 / ~RM11 $10.00 / ~RM44 $1.25 / ~RM5.50 GPT-4o-mini $0.15 / ~RM0.66 $0.60 / ~RM2.64 $0.075 / ~RM0.33 o3-mini $1.10 / ~RM4.85 $4.40 / ~RM19.40 $0.55 / ~RM2.42 text-embedding-3-small $0.02 N/A N/A At first glance, this looks reasonable. GPT-4o-mini at RM0.66 per million input tokens is cheap enough to use for almost any classification or extraction task. But here's the trap: GPT-4o itself is expensive at scale. If your application processes 10 million tokens per month in both directions on GPT-4o, you're looking at roughly RM550 per month just in API costs.Scale that to production — 500 million tokens per month per direction — and you're at RM27,500 per month. At that point, you should be asking: "What else could I buy for RM27,500 a month?"---Self-Hosted Cost RealityLet's look at the self-hosted path. I'll use Azure ND A100 v4-series (the workhorse for LLM inference) as our baseline. VM SKU GPU vCPUs RAM Monthly Cost (pay-as-you-go, SG) Monthly Cost (1-year reserved) Standard_NC16as_T4_v3 1x T4 16GB 16 110 GiB ~RM7,800 ~RM3,200 Standard_NC48ads_A100_v4 1x A100 80GB 48 440 GiB ~RM30,700 ~RM20,000 Standard_ND40rs_v2 8x V100 40 672 GiB ~RM98,000 ~RM48,000 Note: Pricing is approximate (Azure Retail Prices, SE Asia, Linux, mid-2025 list rates) and varies by region, OS, and offer type. The ND96amsr_A100_v4 (8x A100) is not available in the SE Asia region. Verify with the Azure Pricing Calculator.Now, the key insight: a high-memory GPU VM like the ND40rs_v2 (8x V100) or a multi-GPU A100 configuration can serve a 70B parameter model like Llama 3.1-70B with vLLM and handle hundreds of thousands of requests per day. Let's say you run 10 million tokens per day through that VM (300 million per month). Your cost with Azure OpenAI on GPT-4o would be about RM16,500 per month. Your cost on a reserved ND40rs_v2 would be about RM48,000 per month.Wait — that looks worse for self-hosted, right?Not so fast. That same GPU VM can serve multiple models simultaneously. You could run Llama 3.1-70B for complex reasoning, a fine-tuned Qwen 2.5-32B for your domain-specific tasks, and a small embeddings model — all on the same hardware using vLLM's multi-LoRA or model multiplexing. The economics improve when you amortise compute across workloads.The break-even analysis looks roughly like this (actual thresholds vary by GPU SKU, reservation term, and workload mix):- Below 200M tokens/month: Azure OpenAI (GPT-4o-mini) is cheaper. Don't bother self-hosting.- 200M–500M tokens/month on small models (< 8B): Azure OpenAI still wins. The T4 VM is borderline but operational overhead argues against it.- 500M–1B tokens/month on large models (70B+): Self-hosting starts to break even, especially with reserved instances and multiple workloads sharing the GPU.- Above 1B tokens/month on any model size: Self-hosting wins decisively, assuming you have the team to run it.---The Decision MatrixHere is the structured framework I use with clients. It evaluates across five dimensions: Criterion Weight Azure OpenAI Self-Hosted (open-source) Unit economics at low volume High ✅ Wins ❌ Paying for idle GPU Unit economics at high volume High ❌ Expensive per token ✅ Wins, amortised compute Data residency & sovereignty Critical ✅ Microsoft compliance, but data leaves your network ✅ Full control, data never leaves your Azure VNet Model choice & customisation Medium ❌ Limited to Microsoft's catalog ✅ Any HuggingFace model, fine-tuned variants Latency & throughput control Medium ❌ Shared infra, rate limits, variable latency ✅ Full control, predictable P50/P99 Operational overhead High ✅ Zero ops, managed SLA ❌ Need MLOps, monitoring, patching, GPU infra skills Compliance & auditability Critical ✅ SOC 2, ISO, contractual (Microsoft liability) ✅ Your infra, your audit (but your responsibility) Scaling agility Medium ✅ Auto-scale via provisioned throughput ❌ Must pre-provision, scale-out is slower Fine-tuning capability Low ⚠️ Limited to Azure OpenAI fine-tuning (few models) ✅ Full LoRA/QLoRA/full fine-tuning freedom How to use this matrixScore each criterion from 0–3 for each option. Multiply by the weight factor (Critical=3, High=2, Medium=1, Low=0.5). Sum both sides. The higher total is your starting recommendation.A real example from a Malaysian fintech client:- Data sovereignty = Critical → Self-hosted +3- Operational overhead = High → Azure OpenAI +2- Volume (100M tokens/month) → Azure OpenAI +2- Model customisation = Medium → Self-hosted +1Result: Split deployment. Sensitive data workflows (customer PII processing) go self-hosted. General chat and summarisation go through Azure OpenAI. This "hybrid" pattern is where most enterprises land.---Architecture Patterns for ProductionPattern 1: Pure Azure OpenAI (The "No-Ops" Pattern)[Client App] → [API Gateway / APIM] → [Azure OpenAI] → [Azure AI Search (RAG)]Best for: Teams without MLOps capability, POCs, low-to-medium volume, general-purpose chat. Key considerations:- Use Azure API Management in front for throttling, caching, and key rotation- Enable content filtering and abuse detection (may cause false positives — tune carefully)- Provisioned Throughput Units (PTUs) give you reserved capacity at a discount (reserve minimum 1 month)- Data processing stays in Azure boundary but passes through Microsoft's inference infrastructurePattern 2: Pure Self-Hosted (The "Full Control" Pattern)[Client App] → [Azure Load Balancer] → [AKS Node Pool (GPU)] → [vLLM / TGI] → [Vector DB] ┌──────────────────────┐ │ Model Registry (ACR) │ Best for: High-volume workloads, custom fine-tuned models, strict data sovereignty, predictable latency requirements. Key considerations:- Deploy on AKS with GPU node pools and cluster-autoscaler (cold start is real — budget 3-5 min for node provisioning)- Use vLLM for inference serving — it supports PagedAttention, continuous batching, and tensor parallelism across GPUs- Implement GPU sharing with KServe or Knative for multi-model serving- Monitor GPU utilisation with Prometheus + Grafana dashboards- Plan for spot instance fallback (Azure Spot VMs can cut costs 60-80% but have eviction risk)Pattern 3: Hybrid (The "Best of Both" Pattern)[Client App] → [Router / Orchestrator] ├── ⬇️ Azure OpenAI (GPT-4o-mini for simple, GPT-4o for complex) Best for: Most production deployments. Route simple/high-volume tasks to self-hosted, complex/occasional tasks to GPT-4o, sensitive data to self-hosted. Implementation strategy:- Use a lightweight router (a simple Python service or an LLM-based router like Martian or Portkey)- Self-host your small embedding model (BGE-M3 or Snowflake Arctic Embed) for RAG — this is almost always cheaper than Azure OpenAI embeddings at scale- Reserve Azure OpenAI PTU only for your GPT-4o complex-task throughput- Fine-tune a 7B–14B parameter model (Qwen 2.5, Llama 3.1, or Phi-3) on your domain data for the 80% case; route the remaining 20% to GPT-4o---Real-World Cost Comparison: Enterprise Chatbot in MalaysiaLet me walk through a concrete scenario. A Malaysian insurance company wants a customer-facing chatbot handling policy queries, claim status checks, and product recommendations.Workload profile:- 50,000 conversations per day (~1.5M/month)- Average 500 input + 150 output tokens per conversation- 80% simple (policy lookups) + 20% complex (claim escalation, ambiguous queries)- Data sovereignty: Must remain in Malaysia/Southeast Asia- Latency target: P99 < 3 secondsOption A: Pure Azure OpenAI Item Monthly Cost (RM) GPT-4o-mini (95% of tokens) ~RM 2,400 GPT-4o (5% of tokens) ~RM 3,800 Embeddings (text-embedding-3-small) ~RM 500 Azure AI Search (S2 tier) ~RM 1,600 API Management ~RM 800 Total ~RM 9,100/month Option B: Self-Hosted (Hybrid Strategy) Item Monthly Cost (RM) NC48ads_A100_v4 (1x A100, 1-yr reserved, 80% of inference) ~RM 20,000 NC16as_T4_v3 (1x T4 spot, embeddings & re-ranking) ~RM 1,100 Azure Kubernetes Service ~RM 800 Azure Container Registry + Storage ~RM 300 MLOps tooling (MLflow, custom monitoring) ~RM 600 Azure OpenAI GPT-4o (only 5% complex queries) ~RM 1,900 Total ~RM 24,700/month The turning pointAt this volume, the pure Azure OpenAI path is significantly cheaper on raw compute. The hybrid path costs more but gives the insurer:1. Full data residency for policy documents (no data sent to external inference endpoints)2. A fine-tuned model that understands Malaysian insurance vernacular (instead of generic English)3. Predictable latency with no rate-limit surprises during peak hours4. The ability to swap models without vendor negotiationIf this company scales to 5x the volume (250K conversations/day), the Azure OpenAI-only option balloons to roughly RM38,000/month while the hybrid architecture scales more linearly — the GPU reserved cost stays flat and only additional inference capacity is needed. The economics increasingly favour self-hosted at very high volumes, but the crossover point depends on actual GPU pricing and workload mix.---Pitfalls to Avoid1. Underestimating the MLOps TaxSelf-hosting requires someone who understands GPU operators, CUDA driver compatibility, vLLM configuration, HPA tuning, and model storage optimisation. If your team doesn't have this expertise, budget 3-6 months to build it — or hire for it. The "free" open-source model can cost you RM200,000+ a year in MLOps engineering salary before it even answers a single query reliably in production.2. Ignoring Token Caching EconomicsAzure OpenAI charges less for cached input tokens (50% discount). Self-hosted solutions like vLLM also support KV-cache-based prefix caching. Many workloads (conversations with a shared system prompt or RAG context) see 40-60% cache hit rates. Factor caching into your cost model — it changes the break-even point significantly.3. Over-Provisioning for Peak LoadOne mistake I see repeatedly: teams provision GPU VMs for peak traffic and pay for idle GPUs 60-70% of the time. Use Azure's cluster-autoscaler with GPU node pools, configure pod-level HPA based on request queue depth, and always set a minimum replica of 1 with a generous cooldown. Consider Azure Spot GPU VMs for stateless inference workloads.4. Treating All Models as InterchangeableGPT-4o is genuinely better than open models at complex reasoning, instruction following, and multilingual nuance (especially Bahasa Malaysia and Mandarin mixed contexts which we deal with daily in Malaysia). Don't sacrifice accuracy on the altar of cost savings. Route with intelligence — let the difficult queries hit GPT-4o and serve the rest from your fine-tuned open model.5. Skipping the Caching LayerA Redis-backed response cache (with semantic similarity matching or exact match) can eliminate 20-40% of your inference calls on frequently-asked questions. This is the single highest-ROI optimisation you can make, and it works for both Azure OpenAI and self-hosted paths. Implement it from day one.---Key Takeaways1. There is no universal winner. The right choice depends on your volume, data sensitivity, team capability, and latency requirements. Use the decision matrix in this article — don't guess.2. The hybrid pattern wins in production. Route simple and high-volume tasks through self-hosted open models; reserve managed Azure OpenAI for complex reasoning, fallback, and multilingual edge cases. This gives you cost control without sacrificing capability.3. Self-hosting pays off at scale, but only if you have the team. Below ~500M tokens per month, the operational overhead rarely justifies the savings. Above that, the economics swing in favour of self-hosted — especially with reserved instances and multi-model GPU sharing.4. Data sovereignty is a legitimate driver, not an excuse. For Malaysian financial institutions, insurers, and government-linked companies subject to Bank Negara or PIDM regulations, self-hosting may be mandatory. Build the compliance case into your framework from the start.5. Optimise the caching layer before optimising the model. A response cache with 30% hit rate effectively gives you a 30% discount on inference costs — and it takes a weekend to implement, not a month.---The LLM deployment landscape on Azure is rich with options, and the gap between managed and self-hosted paths is narrowing every quarter. Microsoft continues adding open models to Microsoft Foundry (Llama, Mistral, Phi, Cohere), while the open-source ecosystem improves inference efficiency through techniques like speculative decoding and 4-bit quantisation.The decision framework above will serve you well today. But revisit it every six months — because this market moves faster than any architecture decision you'll ever make.Have questions about your specific Azure LLM deployment? I work with Malaysian enterprises on data and AI architecture — from model selection and GPU sizing to hybrid inference architectures. Connect on LinkedIn or drop me a message.---Law Wen Feng is a Principal Solution Architect based in Malaysia, specialising in cloud-native data platforms, AI/ML infrastructure on Azure, and enterprise-scale LLM deployment strategies. --- ## Microsoft Fabric vs Azure Synapse: When to Use Which in 2026 URL: https://wenfeng.my/microsoft-fabric-vs-azure-synapse-when-to-use-which-in-2026-2/ Author: admin Published: 2026-05-06 23:30:51 Modified: 2026-10-04 15:30:43 ⚡ TL;DRMicrosoft Fabric and Azure Synapse Analytics serve overlapping but distinct roles in the Microsoft data platform ecosystem.What Is Microsoft Fabric?What Is Azure Synapse Analytics?Head-to-Head ComparisonWhen to Choose FabricWhen to Choose SynapseMicrosoft Fabric and Azure Synapse Analytics serve overlapping but distinct roles in the Microsoft data platform ecosystem. Choosing between them — or deciding to use both — requires understanding not just their capabilities today, but where Microsoft is investing and where each platform is heading.I have been working with both platforms since their early days: Synapse since its 2019 launch (as Azure SQL Data Warehouse before the rebrand), and Fabric since the public preview in 2023. In 2026, the landscape has shifted. Fabric is Microsoft's strategic bet for the data platform, but Synapse remains the right choice for specific workloads. This guide breaks down the decision with real cost models, architecture patterns, and migration considerations.What Is Microsoft Fabric?Fabric is Microsoft's unified analytics platform — a SaaS experience that combines data engineering, data warehousing, data science, real-time analytics, and business intelligence into a single, integrated product. It is built on OneLake, a single, unified data lake that serves as the foundation for all Fabric workloads, with multi-cloud shortcuts for external data access.Key characteristics:SaaS — fully managed, no infrastructure to provision or patch. You create a capacity and start working.OneLake — a single copy of data, accessible by all workloads (warehouse, lakehouse, notebooks, Power BI, real-time analytics). No data duplication, no ETL between workloads.Built-in Power BI — Direct Lake mode connects Power BI directly to OneLake data without moving it into an import or dataset. This eliminates the traditional dataset refresh bottleneck.Fabric Capacities — compute is purchased as Fabric Capacity (F SKUs), measured in Capacity Units (CUs) per hour. You pay for the capacity, not per-query or per-TB-scanned.Mirrored databases — real-time replication from Azure SQL Database, Cosmos DB, and other sources into OneLake, enabling near-real-time analytics without ETL pipelines.Data Activator — a no-code tool for detecting patterns in streaming data and triggering actions (emails, Power Automate flows, Teams messages).When it works: Fabrics' sweet spot is organisations that want a single platform for all analytics workloads — data engineering, BI, real-time, and data science — without managing separate services or duplicating data.When it struggles: Fabric's capacity-based pricing can be unpredictable for bursty workloads. Large, steady-state data warehouse workloads may cost more than Synapse Dedicated SQL Pools. And Fabric is still maturing — some advanced features (like materialised views with aggregations in the SQL analytics endpoint) are newer and less battle-tested than Synapse equivalents.What Is Azure Synapse Analytics?Synapse is Microsoft's cloud data warehousing and big data analytics service. It provides three compute engines under a single management surface:Dedicated SQL Pool — the classic data warehouse. Provision a pool with DWUs (Data Warehouse Units), get predictable performance. This is the evolution of Azure SQL Data Warehouse.Serverless SQL Pool — query over Parquet, Delta, or CSV files in ADLS Gen2 without provisioning compute. Pay per TB scanned. Ideal for ad-hoc exploration and data lake queries.Apache Spark Pool — managed Spark clusters for data engineering, ETL, and machine learning. Supports Python, Scala, C#, and SQL.Key characteristics:PaaS — more control over infrastructure than Fabric. You choose the pool size, configure networking, manage Spark configurations, and tune performance.ADLS Gen2 integration — data lives in your own Azure Data Lake Storage Gen2 accounts. You control the storage, the directory structure, the lifecycle policies.Synapse Link — real-time replication from Cosmos DB, SQL Server, and Dataverse into Synapse for near-real-time analytics.Dedicated SQL Pool with DWU pricing — predictable cost that scales linearly with DWUs. A DW1000c pool costs ~$12,000/month and delivers consistent performance.Pipeline orchestration — built-in data integration (SSIS-compatible) for orchestrating ETL/ELT workflows.When it works: Synapse excels for organisations with existing data warehouse investments, dedicated SQL workloads that need predictable performance, and teams that need granular control over infrastructure and query tuning.When it struggles: Synapse is complex to manage compared to Fabric. The separation of compute and storage is less integrated — you manage ADLS, Synapse pools, pipelines, and Power BI as separate resources. There is no unified data lake like OneLake, so data engineering notebooks and warehouse queries operate on different copies of data.Head-to-Head Comparison Criteria Microsoft Fabric Azure Synapse Analytics Service model SaaS (Fabric Capacities) PaaS (Dedicated/Serverless/Spark pools) Compute model CU hours (F2 to F2048) DWU hours (Dedicated) or TB scanned (Serverless) Data store OneLake (unified, single copy) ADLS Gen2 (flexible, multiple copies) Data warehouse Fabric Warehouse (T-SQL, auto-optimised) Dedicated SQL Pool (T-SQL, manually tuned) Data engineering Notebooks + Data Factory pipelines Synapse Spark + Pipelines Data science Notebooks + ML model registry Synapse Spark + MLflow (custom setup) Real-time analytics Eventhouse + KQL (first-class) Limited (Spark streaming only) BI integration Native Direct Lake (no dataset refresh) Separate Power BI (import/DirectQuery) Cost model Capacity-based (can spike with concurrency) DWU-based (predictable for Dedicated) Cost at small scale F2 ~$250/mo (all workloads included) Serverless ~$5/TB scanned (unpredictable) Cost at large scale F64 ~$8,500/mo (includes Power BI) DW500c ~$6,000/mo (Power BI extra) Maturity Growing (GA since November 2023, rapidly evolving) Mature (since 2019, stable feature set) Multi-cloud Azure only Azure only Control Minimal (SaaS abstractions) High (networking, tuning, config) Security Workspace RBAC, shortcuts, SQL analytics VNet injection, private endpoints, firewall When to Choose Fabric1. You Want SaaS SimplicityIf your team lacks dedicated data engineers or platform administrators, Fabric's managed experience removes operational overhead. There are no servers to patch, no capacity planning for Spark executors, no VNet configurations for private endpoints (Fabric handles networking). A small team can stand up a complete analytics platform in hours, not weeks.2. OneLake Is a Compelling ValueIf your organisation struggles with data duplication — the same data in the warehouse, the data lake, Power BI datasets, and data science feature stores — OneLake eliminates this. One copy of the data, accessible by all workloads. No ETL between lake and warehouse. No stale Power BI datasets waiting for refresh.The architecture pattern:[Source Systems] -> OneLake (Delta/Parquet)|+------------+------------+| | |Fabric Data Fabric Power BIWarehouse Notebooks Direct Lake| | |(T-SQL) (Python/SQL) (No import)3. Real-Time Analytics Is a PriorityFabric's Eventhouse (built on Azure Data Explorer/KQL) provides first-class real-time analytics. You can ingest streaming data (IoT telemetry, clickstreams, application logs), run KQL queries, and visualise in real-time dashboards — all within the same capacity as your batch analytics.Synapse's real-time capabilities are limited to Spark Structured Streaming, which requires more engineering effort and separate infrastructure.4. Power BI Is Your Primary BI ToolFabric's Direct Lake mode is a game-changer for Power BI users. It connects Power BI directly to OneLake data with no import delay and no DirectQuery latency. Dataset refreshes become instant — the data is already in OneLake.In Synapse, Power BI requires either import mode (with scheduled refreshes) or DirectQuery (with query-time latency). Direct Lake in Fabric eliminates both trade-offs.When to Choose Synapse1. You Need Predictable Query PerformanceDedicated SQL Pools with DWU-based scaling provide consistent, predictable performance. If you run the same analytics workload every day — same queries, same data volume, same concurrency — Synapse Dedicated pools deliver stable response times.Fabric Warehouse optimises based on CU consumption, which can vary with concurrency. A busy hour where many users run complex queries can consume more CUs than a quiet hour, causing performance variability. This is manageable with monitoring and capacity scaling, but it is less predictable than dedicated DWU compute.2. You Already Have Significant Synapse InvestmentIf your team has existing Synapse pipelines, Spark notebooks, dedicated SQL pools, and monitoring tooling built around Synapse, migrating to Fabric needs a solid business case. The migration is not trivial — it involves:- Recreating pipeliines in Fabric Data Factory- Re-pointing Spark notebooks to OneLake paths- Converting Synapse SQL views and stored procedures to Fabric Warehouse- Retraining the team on Fabric's management surfaceFor a mature Synapse deployment with 20+ pipelines and 50+ views, budget 2-4 weeks for migration plus another month for stabilisation.3. You Need Maximum ControlSynapse's PaaS model gives you:- VNet injection for network isolation- Custom firewall rules and IP whitelisting- SQL Server Auditing (not just workspace-level auditing)- Resource class management (for workload isolation in Dedicated pools)- Manual statistics management for query optimisation- Custom Spark configurations and library managementFabric abstracts most of this away. If your compliance or security team requires network-level controls, Synapse may be the only option.4. Cost Predictability MattersDWU-based pricing is simple: a DW500c pool costs ~$6,000/month, every month, regardless of how many queries run. Budgeting is straightforward.Fabric's CU-based model is less predictable. Your CU consumption depends on:- Number of concurrent users- Query complexity- Data volume scanned- Background operations (index maintenance, statistics updates)- Real-time ingestion throughputMicrosoft provides the Fabric Capacity Estimator, but actual consumption can vary day-to-day. For organisations that need fixed monthly costs, Synapse Dedicated pools are easier to budget.5. Specific Source Control and CI/CD RequirementsSynapse supports:- Synapse Workspace Git integration (GitHub, Azure DevOps)- ARM template deployment for infrastructure-as-code- Synapse Pipeline CI/CD with Azure DevOpsFabric's CI/CD story is improving but less mature. Fabric deployment pipelines were added in late 2024, but they are workspace-level, not the granular control that Synapse users get with ARM templates.The Hybrid ApproachIn 2026, many enterprises will use both platforms. This is not a compromise — it is using each platform for what it does best.Pattern: Fabric for New Workloads, Synapse for Legacy[New Workloads] [Existing Workloads]| |Fabric Synapse(OneLake) (Dedicated Pool + ADLS)| |+-------- Mirrored ------+DatabaseNew analytics workloads — real-time dashboards, self-service BI, data science experiments — run on Fabric where the SaaS model and OneLake provide agility. Existing data warehouse workloads stay on Synapse where predictable performance and control matter.Fabric's mirrored databases can replicate data from Synapse Dedicated SQL Pools into OneLake, providing a bridge between the two platforms. This enables Power BI users to query Synapse data through Fabric's Direct Lake mode without impacting Synapse performance.Pattern: OneLake as the Data FoundationUse OneLake as the single source of truth, then choose the compute engine per workload:OneLake|+------+------+------+------+| | | | |Fabric Synapse Synapse Power AzureWarehouse Serverless Serverless BI MLSQL SparkFabric Warehouse for BI and operational reportingSynapse Serverless SQL for ad-hoc data lake queriesSynapse Serverless Spark for custom data engineeringPower BI Direct Lake for self-service dashboardsAzure ML for model training (reading from OneLake)This pattern gives you the best of both worlds: OneLake's unified storage with the flexibility to choose the right compute engine for each job.Cost Modeling: Real-World ScenariosLet me share cost data from a client deployment in Southeast Asia — a mid-size financial services company managing ~5 TB of analytics data.Scenario A: Data Warehousing (5 TB, 20 concurrent users) Platform Configuration Monthly Cost Synapse Dedicated DW500c (dedicated pool) ~$6,000 Fabric F64 capacity ~$8,500 Winner Synapse ~$2,500 cheaper For dedicated warehouse workloads, Synapse is cheaper because you pay only for the warehouse compute. Fabric includes all workloads in the capacity — you pay for data engineering and real-time even if you only use the warehouse.Scenario B: Self-Service BI + Data Engineering (10 GB, 50 users) Platform Configuration Monthly Cost Synapse Serverless SQL + Spark (pay-per-TB) ~$1,200 + Power BI ~$500 = ~$1,700 Fabric F2 capacity ~$250 (includes Power BI) Winner Fabric ~$1,450 cheaper For lighter workloads with Power BI, Fabric's all-in-one capacity is dramatically cheaper because Power BI Pro licenses are included in the F64+ SKUs (for F2-F32, the Fabric capacity handles the compute but you still need Power BI Pro licenses for consumers).Migration ConsiderationsFrom Synapse to FabricThe migration path is well-documented:OneLake integration — OneLake supports the same file formats as ADLS Gen2 (Parquet, Delta). If your Synapse data is already in Parquet/Delta format, the data layer migration is straightforward.Pipeline migration — Synapse Pipelines map directly to Fabric Data Factory pipelines. The mapping is close to one-to-one, though some Synapse-specific activities (like Synapse Notebook activities) need recasting.SQL migration — Dedicated SQL Pool T-SQL objects (views, stored procedures, functions) need testing against Fabric Warehouse. Fabric Warehouse supports standard T-SQL but has some differences in execution plans.Estimated effort: 2-4 weeks for a medium-complexity Synapse deployment.From Fabric to SynapseLess common, but possible if Fabric's cost model or limitations don't work:OneLake data can be accessed via ADLS Gen2-compatible APIs (Azure Data Lake Storage Gen2-compatible)Fabric pipelines need recreation in SynapseFabric SQL analytics endpoint views need conversion to Synapse SQL viewsPower BI Direct Lake datasets need conversion to Import or DirectQueryThe 2026 VerdictFabric is the future of Microsoft's data platform. New features, investment, and engineering resources are flowing into Fabric. Synapse is no longer the primary investment area for Microsoft — it will continue to run existing workloads reliably, and Microsoft continues to support it, but major new capabilities ship to Fabric first.My recommendation:Greenfield data platform: Start with Fabric. The SaaS model, OneLake, and built-in Power BI integration make it the clear choice for new deployments. Use F2-F32 capacities for smaller workloads.Existing Synapse deployment under 5 TB: Plan a migration to Fabric within 12-18 months. The cost savings from unified capacity and eliminated data duplication will offset the migration effort.Existing Synapse deployment over 5 TB: Evaluate carefully. Run the Fabric Capacity Estimator with your actual workloads. If Fabric's F64+ cost exceeds your current Synapse spend, stay on Synapse for the warehouse layer but consider Fabric for new workloads.Hybrid is not a failure — it is the pragmatic approach for most mid-to-large enterprises. Fabric for new SaaS-friendly workloads, Synapse for existing predictable performance workloads. OneLake bridges the gap.Key TakeawaysFabric is the future of Microsoft's data platform — new features and investment are flowing into Fabric. Start evaluating it now, even if you are not ready to migrate.Synapse is not going away — it remains the better choice for predictable performance at scale, existing investments, and organisations that need granular control over infrastructure.OneLake is the biggest differentiator — if unified data access across all analytics workloads matters to your organisation, Fabric wins. The elimination of data duplication alone can justify the migration.Cost comparison requires real workloads, not list prices — use the Fabric Capacity Estimator and Azure Pricing Calculator with your actual query patterns. The answer is often counter-intuitive.The hybrid approach is valid and recommended — use Fabric for new SaaS-friendly workloads, keep Synapse for existing predictable workloads. OneLake and mirrored databases provide the bridge.Real-time analytics is Fabric's hidden advantage — Eventhouse and KQL provide capabilities that Synapse cannot match. If real-time is a requirement, Fabric is the clear choice.Plan your migration by workload, not by platform — start with the workloads that benefit most from Fabric (self-service BI, real-time, data science experiments) and leave steady-state warehouse workloads on Synapse until there is a clear cost or capability reason to move. --- ## Azure Virtual WAN vs Hub-and-Spoke: Which Network Architecture for 2026? URL: https://wenfeng.my/azure-virtual-wan-vs-hub-and-spoke-which-network-architecture-for-2026-2/ Author: admin Published: 2026-05-05 23:33:16 Modified: 2026-10-04 15:30:43 ⚡ TL;DRAzure Virtual WAN and Hub-and-Spoke are the two dominant network architectures for enterprises on Azure. The right choice depends on your scale, complexity, operational maturity, and — most importantly — where you expect to be in three years.The Hub-and-Spoke ModelAzure Virtual WANDecision MatrixMigration Path: Hub-and-Spoke to Virtual WANReal-World Recommendation for 2026Azure Virtual WAN and Hub-and-Spoke are the two dominant network architectures for enterprises on Azure. The right choice depends on your scale, complexity, operational maturity, and — most importantly — where you expect to be in three years. This guide breaks down the decision with real architecture diagrams, cost models, migration paths, and the trade-offs I have seen across dozens of Azure engagements in Southeast Asia.If you are starting a new Azure landing zone in 2026, the default answer is probably Virtual WAN. But that "probably" carries a lot of nuance. Let me walk through both options so you can make an informed decision for your specific context.The Hub-and-Spoke ModelHub-and-spoke is the traditional Azure networking pattern. A central hub VNet hosts shared services (Azure Firewall, VPN Gateway, DNS, Azure Bastion) and connects to spoke VNets via VNet peering.ArchitectureThe corrected architecture places the VPN/ExpressRoute Gateway inside the Hub VNet, with Azure Firewall acting as the central inspection point. Shared services such as Private DNS, Bastion, and monitoring sit in the hub. Each workload spoke is individually peered to the hub and uses user-defined routes where inspected traffic must traverse Azure Firewall; spokes are not chained together.Each spoke is a separate VNet with its own subnets, NSGs, and route tables. Traffic between spokes must traverse the hub, where Azure Firewall enforces inspection. This design is well-understood, well-documented, and has been the recommended pattern since Azure first supported VNet peering.When Hub-and-Spoke WinsSmall to medium deployments — under 40-50 spoke VNets. VNet peering limits (500 per VNet in a region) mean you eventually hit scale limits.You need granular routing control — custom user-defined routes (UDRs), per-spoke firewall policies, and fine-grained traffic engineering. Virtual WAN abstracts much of this away.Your team has deep networking expertise — hub-and-spoke requires understanding BGP, UDRs, peering transitive routing limitations, and Azure Firewall policy hierarchies.Cost sensitivity — no Virtual WAN hub charges. At small scale, hub-and-spoke is cheaper by $200-600/month.Single region — multi-region hub-and-spoke is possible but complex. Each region needs its own hub with inter-hub peering or ExpressRoute Global Reach.Cost ModelHub-and-spoke costs are straightforward: Component Cost Estimate VNet peering (within region) ~$0.01/GB ingress/egress (usually negligible) Azure Firewall ~$1.25/hr + $0.065/GB processed VPN Gateway (Active-Active) ~$0.25/hr ExpressRoute Gateway ~$0.70/hr (1 Gbps) Typical monthly (small) ~$600-2,000 depending on firewall throughput No per-VNet platform fee. The cost scales linearly with spoke count through peering data transfer charges.Limitations I Have Hit in ProductionPeering limit: You can peer a maximum of 500 VNets to a single hub VNet. Beyond that, you need a multi-hub architecture.Non-transitive peering: VNet peering is non-transitive. Spoke A cannot route to Spoke B through the hub unless you add UDRs on each spoke pointing to the firewall as the next hop. This works, but it is manual to configure at scale.Complex multi-region: Each region needs its own hub. Connecting hubs requires additional peering or ExpressRoute circuits.Firewall bottleneck: All inter-spoke traffic flows through the single hub firewall. At scale, this becomes a throughput bottleneck.Azure Virtual WANVirtual WAN is Microsoft's managed wide-area networking service. It provides automated hub deployment, built-in routing, integrated VPN/ExpressRoute gateways, and centralized management through a single control plane.ArchitectureThe corrected Virtual WAN architecture uses a Microsoft-managed control plane to operate regional virtual hubs. Each hub provides managed routing, gateway scale units, optional secured hub integration through Azure Firewall Manager, spoke VNet connections, and branch or SD-WAN connectivity over the Microsoft backbone.Each regional hub is a managed infrastructure that includes:VPN gateway for site-to-site and point-to-site connectivityExpressRoute gateway for private connectivityAzure Firewall Manager integration for secured hubsSD-WAN integration via NVA partners such as Fortinet, Cisco, and Palo AltoAutomatic routing tables and route propagationWhen Virtual WAN WinsLarge scale — 50+ spoke VNets across multiple regions. Virtual WAN handles routing at scale without hitting peering limits.Multi-region deployments — hub-to-hub connectivity is automatic. ExpressRoute circuits in different regions are connected without manual peering.Branch office connectivity — built-in SD-WAN integration with partners like VMware SD-WAN, Cisco vEdge, and Fortinet SD-WAN.Limited networking team — Virtual WAN abstracts away most routing complexity. Azure manages the hub infrastructure.Global transit — Virtual WAN acts as a global transit network. Traffic between East Asia and Southeast Asia hubs routes through the Microsoft backbone, not the public internet.Cost ModelVirtual WAN pricing is different from hub-and-spoke: Component Cost Estimate Virtual WAN hub (standard) ~$0.25/hr per hub VPN Gateway (in-hub) ~$0.36/hr (per scale unit) + $0.05/hr (per connection unit) ExpressRoute Gateway (in-hub) ~$0.42/hr (2 Gbps per scale unit) Data transfer between hubs Standard inter-region rates Typical monthly (3 hubs, 60 spokes) ~$1,500-3,500 The key insight: Virtual WAN has a higher base cost but better per-unit economics at scale. At 60+ spokes, Virtual WAN is typically cheaper than an equivalent hub-and-spoke setup because you eliminate redundant gateway VMs and manual peering management.What Virtual WAN Does Not Do WellGranular routing control: Virtual WAN's routing model is simpler than hub-and-spoke but less flexible. You work with route tables and connections — no custom UDRs per spoke.Small deployments: Under 20 spokes, you are paying hub charges for minimal benefit.Non-standard topologies: If you need exotic routing patterns (e.g., selective inspection for specific spokes, asymmetric routing), hub-and-spoke gives you more control.Migration complexity: Moving from hub-and-spoke to Virtual WAN requires careful planning. The attach-then-decomission approach works but has a period of dual-running costs.Decision Matrix Factor Hub-and-Spoke Virtual WAN Scale ceiling ~500 VNets per hub (peering limit) 1000+ VNets across hubs Regions Single region (multi-region complex) Built-in multi-region Team expertise required High (routing, BGP, UDRs) Medium (routing tables, connections) Cost (20 spokes, 1 region) ~$800-1,200/mo ~$1,000-1,500/mo Cost (100 spokes, 3 regions) ~$4,000-7,000/mo ~$2,500-4,500/mo Branch/SD-WAN integration Manual (VPN GW + third-party) Built-in via Virtual WAN partners Firewall integration Azure Firewall in hub VNet Azure Firewall Manager in hubs Routing model UDRs + BGP Route tables + connections Deployment speed Days-weeks (manual config) Hours (managed deployment) SLA for control plane None (self-managed) 99.95% for hubs Migration Path: Hub-and-Spoke to Virtual WANMost organisations start with hub-and-spoke and migrate to Virtual WAN when they hit multi-region or scale requirements. The migration is well-documented and non-disruptive if done correctly.Step-by-Step MigrationPhase 1: Deploy Virtual WAN alongside existing hub-and-spoke Phase 2: Attach spokes to Virtual WAN hubs (dual connectivity) Phase 3: Route traffic through Virtual WAN Phase 4: Decommission self-managed hub Phase 1 — Deploy Virtual WAN# Create Virtual WAN az network vwan create \ --name wenfeng-vwan \ --resource-group rg-networking \ --type Standard \ --location southeastasia # Create regional hub az network vhub create \ --name seasia-hub \ --resource-group rg-networking \ --vwan wenfeng-vwan \ --address-prefix 10.100.0.0/24 \ --location southeastasia Phase 2 — Attach spokes# Connect existing spoke VNet to Virtual WAN hub az network vhub connection create \ --name spoke-app-conn \ --resource-group rg-networking \ --vhub-name seasia-hub \ --remote-vnet /subscriptions/.../resourceGroups/rg-app/providers/Microsoft.Network/virtualNetworks/app-vnet \ --internet-security true Phase 3 — Route migrationGradually shift traffic by updating route tables. Keep the existing hub active until all spokes are migrated. This is the safest approach — you can roll back a spoke at a time if something breaks.Phase 4 — DecommissionOnce all spokes are attached and traffic flows through Virtual WAN: 1. Remove VNet peerings from the old hub 2. Delete the old hub resources (Azure Firewall, VPN Gateway, DNS VMs) 3. Delete the hub VNetThe critical rule: never delete the old hub until all spokes are confirmed working through Virtual WAN.Common Migration PitfallsRoute propagation delays: After attaching spokes, route propagation can take 2-5 minutes. Schedule migrations during maintenance windows.Firewall policy drift: If you use Azure Firewall with custom rules in hub-and-spoke, those rules need porting to Azure Firewall Manager policies before the migration.IP address overlap: If your hub VNet addresses overlap with Virtual WAN hub addresses, you need re-addressing before migration.VPN/ExpressRoute disconnect: When you transition site-to-site VPN connections from the old hub gateway to the Virtual WAN gateway, expect a brief disconnect. Use BGP to minimise disruption.Real-World Recommendation for 2026Greenfield (Starting Fresh)Use Virtual WAN from day one unless you are: - A single-region deployment with under 20 spokes - A small team with legacy networking tooling - Running a proof-of-concept or MVP with minimal networking requirementsFor 2026 greenfield deployments, Virtual WAN is Microsoft's strategic direction. New features (like vWAN Integrated Secure Hubs and SD-WAN enhancements) typically ship on Virtual WAN first, while hub-and-spoke continues to be supported for existing deployments.Brownfield (Existing Hub-and-Spoke)Stay on hub-and-spoke if you are under 40 spokes and single-region. You have working infrastructure, and the migration cost exceeds the operational benefit.Plan the migration to Virtual WAN if: - You are expanding to a second region - You are approaching 50 spokes - You are adding branch/SD-WAN connectivity - Your networking team is shrinking (or being asked to do more with less)The Southeast Asia ContextIn Malaysia and Southeast Asia, two factors tilt the decision toward Virtual WAN:Multi-region is common: Companies operate across Singapore, Malaysia, Indonesia, and Thailand. Virtual WAN's multi-region hub-to-hub connectivity is a significant operational win over stitching together regional hub-and-spoke deployments with peering.Branch connectivity: Regional banks, retailers, and logistics companies have dozens of branches across the region. Virtual WAN's SD-WAN integration simplifies what would otherwise be a complex VPN mesh.Key TakeawaysHub-and-spoke is the default for single-region, small-to-medium deployments. It gives full control at lower cost. Do not migrate prematurely.Virtual WAN wins at scale — multi-region, 50+ VNets, branch offices, or when you want managed networking. The managed control plane is a significant operational win.Start with hub-and-spoke, migrate when you need to. The migration path is well-established. Use the attach-then-decomission approach to minimise risk.Cost comparison requires real sizing. Use the Azure Pricing Calculator with your actual VNet count, throughput, and gateway requirements. At 40-60 spokes, do the math both ways — the answer is not always obvious.Both support the same security patterns — Azure Firewall, NSGs, Private Link, and Private Endpoints work with either architecture. Do not let security concerns drive the decision.Your team's networking expertise matters more than the technology choice. Virtual WAN reduces operational burden but still requires understanding of routing, VPNs, and firewall policies. Hub-and-spoke demands deeper expertise but gives more control.Plan for the scale you will have in 3 years, not the scale you have today. The cost of migrating from hub-and-spoke to Virtual WAN outweighs the early cost savings if you outgrow hub-and-spoke within 18 months. --- ## Azure Local for Malaysian Enterprises: When Hybrid Beats Cloud-Only URL: https://wenfeng.my/azure-local-formerly-azure-stack-hci-for-malaysian-enterprises-when-hybrid-beats-cloud-only/ Author: admin Published: 2026-05-05 01:10:19 Modified: 2026-10-04 15:30:43 ⚡ TL;DRIn fifteen years architecting infrastructure for Malaysian enterprises, countless meetings go like this:IntroductionThe Problem with Cloud-Only in MalaysiaAzure Local: What Actually ChangedDecision Framework: When Should Malaysian Enterprises Go Hybrid?Reference Architecture: Three Patterns for Malaysian EnterprisesIntroductionIn fifteen years architecting infrastructure for Malaysian enterprises, countless meetings go like this:CIO: "The board wants us in the cloud. Everything. By next year." Me: "What about your core banking system?" CIO: "...we'll figure that out."The cloud-first mandate has swept through Malaysian boardrooms from KL to Penang to Johor. And for good reason — hyperscalers like Microsoft Azure offer elastic capacity, global reach, and a pace of innovation most on-prem teams can't match. But here's the uncomfortable truth I've learned: a blanket "cloud-only" strategy is often the wrong answer for Malaysian enterprises, especially those in regulated industries like finance, healthcare, and government.Enter Azure Local — Microsoft's rebranded and significantly evolved hybrid cloud platform, formerly known as Azure Stack HCI.This isn't just a name change. The rebrand to Azure Local reflects a fundamental shift in how Microsoft wants you to think about hybrid: your on-premises infrastructure should feel like an Azure region. Not a separate island that happens to have some Azure integration bolted on, but a genuine extension of the Azure control plane running in your own data centre.In this article, I'll walk through why Azure Local matters for Malaysian enterprises: the decision framework, reference architectures, real cost considerations, and the pitfalls to avoid.The Problem with Cloud-Only in MalaysiaLet me start by acknowledging what the cloud does well. I've helped Malaysian enterprises spin up globally distributed applications in hours, leverage Azure AI services without buying GPUs, and eliminate the CAPEX cycle of hardware refresh. These are real wins.But cloud-only has blind spots that hit Malaysian enterprises particularly hard:Data Residency and SovereigntyThe Personal Data Protection Act (PDPA) 2010 doesn't explicitly mandate local data storage, but sectoral regulators do. Bank Negara Malaysia's RMiT (Risk Management in Technology) framework requires financial institutions to maintain certain data within Malaysia's borders. The same applies to healthcare data under the Ministry of Health's guidelines.Yes, Azure has a Malaysia region: Malaysia West. Microsoft has also announced additional in-country cloud capacity for Malaysia, but architects should verify current regional availability and service support before treating any future local region as a design dependency. The point remains: "the region exists" is not the same as "all my workloads belong there." Some legacy systems cannot tolerate the latency, operational dependency, or egress cost of a cloud round-trip, even to a local Azure region.Latency-Sensitive WorkloadsManufacturing is Malaysia's fourth-largest economic sector. An IoT line controller on a factory floor in Penang cannot wait 15-30ms for a cloud round-trip to decide whether to stop a press. That decision needs to happen in under 5ms, locally. Similarly, POS systems in retail, trading platforms in financial services, and real-time analytics in logistics — these workloads demand on-prem latency with cloud-like management.Regulatory Audit and ComplianceThe compliance burden doesn't end with storing data locally. Malaysian regulators increasingly expect auditable, immutable logs, strict RBAC over physical infrastructure, and demonstrable control over who touches what. While Azure Policy and Defender for Cloud provide excellent tooling, some compliance officers (rightly or wrongly) feel more comfortable when the actual hardware sits in a cage they can walk through.The Connectivity RealityMalaysia has undersea cables — SEA-ME-WE 5, SEA-ME-WE 6 coming online — but the tail circuit from your Shah Alam office to the nearest Azure edge still goes through last-mile providers with variable reliability. When that link fails during a 2 AM batch window, "cloud-native" offers little comfort.Azure Local: What Actually ChangedBefore we dive into architecture, let me clarify what the Azure Local rebrand actually means — because the naming confusion is real. Old Name New Name Azure Stack HCI Azure Local Azure Stack Edge Azure Local Edge Azure Stack Hub Separate product under Microsoft lifecycle; evaluate new deployments carefully and prefer Azure Local or native Azure where they fit the requirement The key insight: Azure Local is no longer positioned as "the on-prem version of Azure" but as an Azure region that happens to live in your data centre. The control plane, management APIs, billing, monitoring — everything flows through Azure Resource Manager. When you deploy a VM on Azure Local, you do it through the same Azure portal, CLI, or ARM templates you'd use for a VM in Malaysia West or Southeast Asia.This is a genuinely different architectural philosophy from the old Azure Stack HCI, which many Malaysian IT teams found clunky and hard to manage.What Runs on Azure LocalAzure Local supports a broad range of workloads:Virtual Machines: Windows Server and Linux VMs managed through Azure ArcAzure Kubernetes Service (AKS) on Azure Local: Run containerised workloads with a consistent Kubernetes experience across cloud and on-premAzure App Service on Azure Local: For modern web appsAzure Storage services: Blob storage, Files shares, tiered storage with NVMe cacheAzure AI inferencing: Run ONNX models locally without cloud dependencySQL Server and Azure SQL Managed Instance: Hybrid database scenariosDecision Framework: When Should Malaysian Enterprises Go Hybrid?After years of helping Malaysian organisations evaluate this, I've developed a simple three-factor decision framework. You don't need a 200-page TCO analysis to start — answer these three questions.Factor 1: Latency Sensitivity Latency Requirement Recommendation < 5ms required On-prem or Azure Local 5-20ms acceptable Azure Local or Azure near-region > 20ms acceptable Cloud-only is fine Real example: A Malaysian palm oil plantation group needed sub-10ms response for their weighbridge and grading systems across 12 mills. Each mill had intermittent internet. Cloud-only was a non-starter. Azure Local at each mill with central management from HQ in KL was the right call.Factor 2: Regulatory Data BoundaryAsk yourself: "If a regulator audits me today, can I prove where every byte lives and who can access it?"Tier 1 data (must stay in Malaysia, on-prem acceptable): Azure LocalTier 2 data (must stay in Malaysia, cloud acceptable): Azure Malaysia regionsTier 3 data (no restriction): Any Azure regionI recommend classifying your workloads into these three tiers before making any infrastructure decision. Most Malaysian enterprises I work with end up with roughly 30% Tier 1, 40% Tier 2, and 30% Tier 3.Factor 3: Connectivity ProfileReliable connectivity (>99.9% uptime, <10ms to nearest Azure region): Cloud or Azure Local in hybrid modeIntermittent connectivity (frequent disconnections, high latency): Azure Local with disconnected mode (you can run for up to 30 days without Azure connectivity)No connectivity (remote sites, offshore platforms): Azure Local Edge with local managementReference Architecture: Three Patterns for Malaysian EnterprisesLet me give you three architecture patterns I've seen work well in the Malaysian context.Pattern 1: Single-Site Hybrid (Mid-Size Enterprise)Use case: A Malaysian fintech startup needs to run core transaction processing on-prem for RMiT compliance while using Azure for analytics and DR.┌─────────────────┐ ┌──────────────────────┐ │ Azure Local │◄────────►│ Azure Cloud │ │ (On-Prem) │ VPN or │ (Malaysia West) │ │ │ Express │ │ │ ┌─────────────┐ │ Route │ ┌──────────────────┐ │ │ │ Core Tx VMs │ │ │ │ Power BI │ │ │ │ SQL Server │ │ │ │ Azure ML │ │ │ │ AKS Pods │ │ │ │ Azure SQL MI DR │ │ │ └─────────────┘ │ │ └──────────────────┘ │ │ Azure Arc │ │ Azure Policy │ │ Governance │ │ Sentinel SIEM │ └─────────────────┘ └──────────────────────┘ Key configuration using Azure CLI:# Register your Azure Local instance az connectedmachine create \ --resource-group rg-malaysia-finance \ --name arc-finance-hci \ --location eastasia # Deploy a workload VM with Azure governance az vm create \ --resource-group rg-malaysia-finance \ --name core-tx-vm-01 \ --location eastasia \ --custom-data cloud-init.txt \ --enable-agent true Note that the VM location shows eastasia — because Azure Local resources are managed through the Azure control plane. The VM itself runs on your hardware in Malaysia, but you manage it through Azure Resource Manager just like any cloud resource.Pattern 2: Distributed Retail / Branch OfficeUse case: A Malaysian retail chain with 50+ outlets across Peninsular Malaysia needs local POS processing with centralised management.Each outlet gets a compact 2-node Azure Local cluster. Central IT in KL manages everything through Azure Arc and Azure Policy.# PowerShell example: Deploy Azure Local at a branch $clusterParams = @{ Name = "store-kulai-001" ResourceGroupName = "rg-retail-malaysia" Location = "eastasia" NodeCount = 2 StorageSizeGB = 4000 NetworkVLAN = 10 } New-AzStackHciCluster @clusterParams # Apply a Policy that enforces all VMs must use managed disks $policy = Get-AzPolicyDefinition -Name "Audit VMs without managed disks" New-AzPolicyAssignment -Name "enforce-managed-disks-kulai" ` -Scope "/subscriptions/.../resourceGroups/rg-retail-malaysia/providers/Microsoft.AzureStackHCI/clusters/store-kulai-001" ` -PolicyDefinition $policy Pattern 3: Hybrid Kubernetes for ManufacturingUse case: A Malaysian semiconductor manufacturer runs edge inferencing for wafer inspection on-prem, with model training in Azure.# AKS on Azure Local deployment manifest apiVersion: v1 kind: Namespace metadata: name: wafer-inspection --- apiVersion: apps/v1 kind: Deployment metadata: name: defect-detector namespace: wafer-inspection spec: replicas: 3 selector: matchLabels: app: defect-detector template: metadata: labels: app: defect-detector spec: containers: - name: inference image: myregistry.azurecr.io/wafer-defect-model:latest resources: limits: cpu: "4" memory: "8Gi" nvidia.com/gpu: "1" volumeMounts: - name: model-cache mountPath: /models volumes: - name: model-cache persistentVolumeClaim: claimName: model-pvc The beauty of this pattern: the same kubectl apply -f deployment.yaml works against AKS on Azure Local and AKS in the cloud. Your DevOps team doesn't need two deployment workflows.Cost Comparison: Azure Local vs. Cloud-OnlyLet me address the elephant in the room. Everyone asks: "Is it cheaper?"The honest answer: It depends on your workload profile. But here's a framework that helps.The Three Cost DimensionsCompute utilisation breakeven: If your VMs run 24/7 at >40% average utilisation, Azure Local generally wins on compute. Below that, cloud's elasticity pays off.Data egress: This is the hidden killer. A data-heavy application that moves terabytes between on-prem and cloud can incur significant egress charges ($0.05-0.12/GB depending on region). For Malaysian enterprises with large on-prem datasets, egress costs alone can justify Azure Local.Compliance overhead: If your compliance team needs 5 additional tools, 3 extra audits, and 2 dedicated security engineers because you're in the cloud, those hidden costs add up. Azure Local with Azure Policy and Defender for Cloud can reduce this burden.Rough Cost Comparison (Monthly, Malaysian Ringgit) Scenario Cloud-Only (Azure) Azure Local On-Prem (Traditional) 10 VMs (4 vCPU, 16GB each) 24/7 RM 8,000 - 12,000 RM 5,000 - 7,000 RM 6,000 - 9,000 50 VMs with 5TB storage RM 35,000 - 50,000 RM 22,000 - 30,000 RM 25,000 - 35,000 Burstable workloads (auto-scale) RM 4,000 - 8,000 RM 8,000 - 12,000 RM 10,000 - 15,000 Compliance-heavy (finance sector) RM 15,000 - 25,000 RM 10,000 - 15,000 RM 12,000 - 18,000 Note: These are rough estimates based on published Azure pricing and hardware TCO models as of Q1 2026. Your mileage will vary significantly based on reserved instances, EA agreements, and hardware procurement.The key takeaway: Azure Local is typically 20-40% cheaper than cloud-only for steady-state, always-on workloads. For bursty or experimental workloads, cloud-only still wins on agility.Pitfalls to AvoidI've seen Malaysian enterprises stumble on the same few issues repeatedly. Learn from their mistakes.Pitfall 1: Underestimating Azure Local's Connectivity RequirementsAzure Local needs to check in with Azure every 30 days minimum. If it loses connectivity longer than that, you can't deploy new VMs, apply updates, or make configuration changes. I've seen sites in rural Sabah and Sarawak struggle with this.Fix: Deploy a redundant internet connection (e.g., Starlink as backup) for remote sites. Azure Local Edge is better for truly disconnected scenarios.Pitfall 2: Treating Azure Local Like Traditional On-PremAzure Local is licensed per core, billed monthly through your Azure subscription. If you treat it like a CAPEX purchase and forget the OPEX billing, your finance team will have an unpleasant surprise.Fix: Set up Azure Cost Management alerts specific to your Azure Local clusters from day one.# Create a budget alert for your Azure Local cluster az consumption budget create \ --budget-name azure-local-factory-01 \ --amount 15000 \ --time-grain monthly \ --time-period-start-date 2026-05-01 \ --time-period-end-date 2027-04-30 \ --category cost \ --notifications '{ "threshold-gte-80-pct": { "enabled": true, "operator": "GreaterThanOrEqualTo", "threshold": 80, "contact-emails": ["finance@company.com.my", "itops@company.com.my"] } }' Pitfall 3: Overlooking Azure Local's Hardware CertificationYou can't run Azure Local on any old server. Microsoft maintains a strict Azure Local Hardware Certification (formerly Azure Stack HCI Integrated System) catalog. Malaysian resellers sometimes push uncertified hardware — don't take the bait.Fix: Always verify against the official Azure Local hardware catalog. Stick with validated nodes from Dell, HPE, Lenovo, or Supermicro through authorised Malaysian partners.Pitfall 4: Ignoring the Skills GapYour on-prem Windows Server administrators may know Hyper-V, but Azure Local management through ARM templates, Azure Policy, and Infrastructure-as-Code is a different skillset. I've seen projects stall because the team couldn't bridge this gap.Fix: Budget for training. Microsoft Learn has a dedicated Azure Local learning path. Consider a pilot deployment before production.ConclusionAzure Local answers a question Malaysian enterprises have wrestled with: get cloud benefits without sacrificing control, performance, or compliance. It's not a cloud replacement or a return to traditional on-prem — it's a third path that treats your data centre as a first-class Azure citizen.For Malaysian enterprises navigating data residency, latency-sensitive operations, and regulatory scrutiny, Azure Local is increasingly the architecture that makes everything else work.Key TakeawaysUse the three-factor framework — latency sensitivity, regulatory data boundary, and connectivity profile — to determine when hybrid beats cloud-only for each workload.Azure Local reduces management complexity by unifying on-prem and cloud under a single Azure control plane, but it introduces new operational disciplines (consistent connectivity, OPEX billing, Infrastructure-as-Code).Cost-wise, Azure Local wins for steady-state workloads (20-40% cheaper than cloud-only) but lags behind for bursty or experimental workloads — choose based on workload profile, not ideology.Hardware certification matters. Always deploy validated Azure Local nodes and verify against the official hardware catalog to avoid compatibility and support issues.Invest in skills and pilot before scaling. The shift from traditional on-prem management to Azure Local's cloud-native operations model is the most common failure point I see — budget for training and run a proper pilot before committing to a full rollout.Law Wen Feng is a Principal Solution Architect based in Malaysia, specialising in cloud infrastructure, hybrid architecture, and digital transformation for ASEAN enterprises. The views expressed are his own.Have questions about Azure Local for your organisation? Reach out to discuss your specific architecture. --- ## Production Multi-Agent Systems: Observability, Error Handling, and Cost Control with Hermes Agent and OpenClaw URL: https://wenfeng.my/production-multi-agent-systems-observability-error-handling-cost-control/ Author: admin Published: 2026-05-04 00:47:31 Modified: 2026-10-04 15:31:26 ⚡ TL;DRMost AI agent demos stop at the happy path: one prompt, one tool call, one clean answer.The real production problem1. Observability: every agent action needs a trace2. Separate planning state from execution state3. Error handling: fail closed, not silently4. Cost control: budget the workflow, not just the modelMost AI agent demos stop at the happy path: one prompt, one tool call, one clean answer.Production is different.Once an AI agent has access to real tools, customer systems, cloud environments, cron jobs, browsers, files, APIs, and messaging channels, the problem is no longer just “which model is best?” The harder question is whether the system can be observed, controlled, recovered, and operated when something goes wrong.This is especially important for multi-agent systems. A single assistant failure is already inconvenient. A chain of agents failing silently can create duplicate work, publish stale content, overwrite files, leak cost, or leave a business process half-completed.I have been running both Hermes Agent and OpenClaw-style agent workflows in practical environments. The main lesson is simple: multi-agent systems need production engineering, not just prompt engineering.The real production problemA multi-agent system usually has several moving parts:A primary orchestrator that receives the user requestSpecialist agents for research, writing, coding, QA, infrastructure, or operationsTool integrations such as terminal, browser, file system, APIs, messaging, cron, and cloud CLIsPersistent memory and reusable skillsScheduled jobs that run without a human presentExternal systems such as Ghost, Buffer, GitHub, Azure, AWS, Google Search Console, or Microsoft 365The risk is not only that an answer may be wrong. The risk is that the wrong action may be taken automatically.Examples:A social-posting agent creates posts for a blog article that was never published.A cron job marks a task as complete even though the API call failed.A coding agent edits files but never runs tests.A fallback model answers with different assumptions from the primary model.A background task continues after the user thinks it has stopped.A tool call succeeds technically but produces unusable business output.This is why production agent systems need observability, error handling, cost control, and rollback patterns from day one.1. Observability: every agent action needs a traceIn a traditional application, we log requests, errors, latency, and system health. Agent systems need the same discipline, but with additional context.At minimum, every agent workflow should record:User request or job triggerAgent or subagent identityModel/provider usedTool calls executedInput files and output files touchedExternal API endpoints calledFinal resultFailure reason, if anyWhether the action was verifiedFor scheduled jobs, the logs must also include the schedule, target date, and delivery destination. This matters because many pipeline failures are not immediate errors. They are state-alignment errors.For example, a blog pipeline may have these stages:Topic bank → editorial calendar → draft → SEO review → publish → social calendar → Buffer postingIf the social calendar says “posted” but the blog URL does not exist, the system has a broken state. Without traceability, the team will not know which step created the mismatch.2. Separate planning state from execution stateA common mistake is mixing intent with completion.For agent pipelines, I recommend maintaining explicit states:`pending` — work has been identified but not started`drafted` — a draft artifact exists`assigned` — scheduled for a publishing slot`published` — live and verified`scheduled` — queued for external posting`posted` — external platform accepted the post`failed` — action attempted but did not completeThe important point is that a state should only be advanced after verification.For a blog post, “published” should mean the public URL returns a valid page. For a social post, “posted” should mean the Buffer or platform API returned a post ID. For a GitHub change, “complete” should mean the commit exists and tests passed, or the exception is clearly documented.This sounds basic, but many agent workflows skip this and rely on the model’s self-report. That is not good enough for production.3. Error handling: fail closed, not silentlyAgent systems should avoid silent failure.A production-grade workflow should handle at least these failure modes:Tool unavailableAPI authentication failureRate limit or quota exhaustionInvalid JSON or schema mismatchMissing filesNetwork timeoutPartial writeExternal system accepted the request but did not apply the changeModel fallback changed behaviorContext compression removed important stateThe safest default is to fail closed:Do not mark a post as published unless the public URL works.Do not mark a social item as posted unless Buffer returns an ID.Do not delete or overwrite existing generated content unless there is a backup.Do not continue downstream jobs when upstream state is incomplete.For autonomous agents, the failure message should be operationally useful. “Something went wrong” is not enough. The system should say which file, API, status code, record ID, or validation step failed.4. Cost control: budget the workflow, not just the modelMulti-agent systems can become expensive because cost multiplies across agents, retries, tools, and context size.The cost drivers are usually:Number of agents spawnedContext size per agentLong-running background tasksRepeated web searchesLarge file readsFailed retriesHigh-context premium models used for simple workScheduled jobs running even when there is no new workA practical model is to classify agents by workload:Cheap model for routine checks, cleanup, and status validationStrong model for architecture, complex writing, code review, and root-cause analysisVision model only when screenshots or image analysis are requiredDeterministic scripts for repeatable operationsFor example, a content pipeline should not ask a premium model to regenerate all social posts every day if the social calendar is already complete. It should first run a deterministic coverage check, then only ask the model to generate missing content.5. Fallback chains must be visibleFallback models are useful, but they create operational risk.If the primary model fails and the fallback model completes the task, the user may assume the same reasoning quality, tool behavior, or context handling was used. That may not be true.A production system should record:Primary provider/modelFallback provider/modelWhether fallback was usedAny difference in context window or tool behaviorWhether the output requires additional reviewThis is especially important for long-context work. A model with a smaller effective context window may compress or omit details that another model would retain.6. Human approval should be based on impactNot every action needs human approval. But the approval boundary should be explicit.I recommend requiring confirmation for:Publishing public contentSending messages externallyDeleting files or recordsRotating credentialsInfrastructure changes with downtime riskProduction database changesLarge cost-impacting cloud changesFor lower-risk tasks, the system can act autonomously if it has validation and rollback.The goal is not to slow everything down. The goal is to prevent agents from performing irreversible or externally visible actions without the right guardrails.7. Use deterministic validation wherever possibleThe model should not be the only judge of success.Use scripts and checks such as:JSON schema validationLink checksHTTP status checksUnit testsGit diffsCloud resource queriesDatabase queriesAPI response verificationCalendar coverage checksFor example, a social media pipeline can validate:Every published blog post has X, LinkedIn, and Facebook entriesEvery social item has a parent editorial IDEvery scheduled social post has non-empty textEvery published blog URL returns HTTP 200X posts are under 280 charactersThese checks are not glamorous, but they are what make the agent reliable.Reference architecture patternA reliable multi-agent workflow should look more like this:User / Cron Trigger ↓ Orchestrator ↓ State discovery and validation ↓ Specialist agents only where needed ↓ Deterministic execution scripts ↓ Verification checks ↓ State update ↓ Notification / reportThe key is the order. Do not update state before verification. Do not run downstream work if upstream artifacts do not exist. Do not rely on the model’s self-report when a tool can verify the result.Hermes Agent and OpenClaw lessonsHermes Agent is strong when you want persistent skills, memory, cron jobs, messaging integration, and direct operational tool access. It is useful as a long-running personal or team automation layer.OpenClaw-style workflows are useful for structured coding and agent execution patterns, especially where you want isolated agent sessions and strong developer ergonomics.The production lesson is that the framework matters, but the operating model matters more.A good agent framework still needs:Clear state filesRepeatable scriptsLog retentionRecovery stepsTool permission boundariesModel routing rulesCost controlsHuman approval gatesVerification before completionWithout those, a multi-agent setup becomes an impressive demo that is hard to trust.Practical recommendationsIf you are building production multi-agent systems, start with these principles:1. Design the state machine before designing the prompts.2. Keep planning state separate from execution state.3. Log every external action with IDs and timestamps.4. Verify public outputs with tools, not model confidence.5. Make fallback model usage visible.6. Use cheaper models for routine checks and stronger models for high-value reasoning.7. Keep destructive and external actions behind approval or strict validation.8. Build small recovery scripts for common failures.9. Treat cron jobs as production services, not background experiments.10. Review agent outputs the same way you would review code or infrastructure changes.Final thoughtThe future of AI agents is not only smarter models. It is better operational discipline around those models.Multi-agent systems will become normal in software delivery, cloud operations, content production, and business automation. The winners will not be the teams with the flashiest demos. They will be the teams that can run agents safely, observe them clearly, recover from failures quickly, and control cost while still moving fast.That is where the real engineering work begins. --- ## Azure Landing Zones for AI Workloads URL: https://wenfeng.my/azure-landing-zones-ai-workloads/ Author: admin Published: 2026-05-02 19:16:39 Modified: 2026-10-04 15:30:53 ⚡ TL;DRIf your AI project is stuck waiting for GPU quota approvals while your data scientists spin up shadow IT resources, you do not have an AI problem — you have a landing zone problem.Why Standard Landing Zones Fall Short for AIThe Architecture I Would Build TodayGPU Quota Strategy: The Part Nobody Talks AboutModel Registry PatternNetworking: Private Link EverywhereIf your AI project is stuck waiting for GPU quota approvals while your data scientists spin up shadow IT resources, you do not have an AI problem — you have a landing zone problem.Azure Landing Zones are well-documented for traditional enterprise workloads. But AI workloads break the standard patterns: they need GPUs, large ephemeral storage, model registries, feature stores, and data pipelines that move petabytes. The default Enterprise-Scale Landing Zone assumes steady-state compute. AI is anything but steady-state.This post covers the practical decisions I have made when designing Azure infrastructure for AI workloads — GPU quota strategy, model lifecycle management, and the data plumbing that actually matters.Why Standard Landing Zones Fall Short for AIThe Azure Landing Zone Accelerator is built around Hub-Spoke networking, centralized identity, and predictable resource topologies. That works for web apps and APIs. AI workloads introduce three problems the standard pattern does not address well:1. GPU Quota Is a Bottleneck, Not a Line ItemAzure GPU quotas (NC, ND, NV series) are per-subscription and per-region. If your landing zone uses a single subscription for AI compute, you will hit quota limits fast — and quota increase requests take days. The fix: dedicate a subscription specifically for AI compute, separate from general workloads. This gives you a clean quota ceiling and prevents GPU requests from competing with standard VM reservations.resource "azurerm_subscription" "ai_compute" { alias = "ai_compute" subscription_id = var.ai_subscription_id display_name = "AI Compute Sub" billing_account_id = var.billing_account_id }2. Model Registries Need Their Own LifecycleAzure Machine Learning has a built-in model registry, but it sits inside the ML workspace — which means it shares lifecycle with experiments, endpoints, and data assets. For production, I treat the model registry as infrastructure:Azure ML Model Registry for versioned model artifacts with lineageAzure Container Registry (ACR) for inference container imagesBlob Storage for raw model weights (especially for large open-source models)The registry is the contract between training and inference. If you do not version your models, you cannot roll back a bad deployment. Period.3. Data Pipelines Are the Hidden CostAI workloads consume data at scale — training data, feature stores, embedding caches, evaluation datasets. The standard landing zone provides shared services networking, but AI needs high-throughput data movement:Azure Data Lake Storage Gen2 for raw and processed dataAzure Data Factory or Apache Spark on Fabric for ETLPrivate Link endpoints for data access without crossing the public internetThe Architecture I Would Build TodayHere is the hub-spoke topology I would design for an organization deploying AI workloads at scale on Azure.Hub Subscription: Shared ServicesFirewall (Azure Firewall Premium for TLS inspection)Bastion host for secure VM accessDNS Private ZonesAzure Monitor and Log Analytics workspaceKey Vault (HSM-backed) for secrets and model API keysSpoke 1: AI Training ComputeDedicated subscription with GPU quota (NC A100 v4, ND H100 v5)Azure ML workspace with managed VNetCycleCloud or AML Compute Clusters for auto-scaling GPU poolsPrivate endpoints to ADLS Gen2 and ACRSpoke 2: AI Inference and ServingAzure Container Apps or AKS for model serving endpointsAKS with GPU node pools for large model inferenceApplication Gateway with WAF for public-facing inference APIsCosmos DB for request/response logging and feature cacheSpoke 3: Data PlatformAzure Data Lake Storage Gen2 (Bronze, Silver, Gold layers)Microsoft Fabric for analytics and data science notebooksAzure Event Hubs for real-time feature streamingPrivate Link to all storage accountsGPU Quota Strategy: The Part Nobody Talks AboutGPU quota is the single biggest operational blocker for AI teams on Azure. Here is what I have learned:Request early. GPU quota increases take 2-5 business days. Do not wait until you need them.Use multiple subscriptions. Each subscription has its own quota ceiling. Spread training and inference across subscriptions if you need more aggregate GPU capacity.Reservation vs. on-demand. Azure Reservations can lock in GPU capacity for predictable workloads, but they are 1-3 year commitments. Use them for production inference, not experimentation.Spot VMs for training. NCasT4 v3 and NDm A100 v4 spot instances can cut training costs by 60-80%. Use AML managed spot clusters with automatic checkpointing.Model Registry PatternI use a three-tier registry approach that keeps things clean: TierServicePurposeArtifactAzure ML Model RegistryVersioned models with lineage, metrics, and tagsContainerAzure Container RegistryInference container images with dependencies baked inRaw WeightsBlob Storage (ADLS)Large model weights (LLaMA, Mistral, etc.) — too big for ML registry The deployment pipeline promotes models through these tiers: train, register in ML, build container, push to ACR, deploy to AKS or Container Apps. If a model fails validation, it never reaches ACR. That is the quality gate.Networking: Private Link EverywhereAI workloads move large datasets. The default Azure networking model (public endpoints with NSGs) works but introduces latency and security risk. For production AI, I enforce:Private Link for all storage accounts — ADLS, ACR, Key Vault, Cosmos DBService endpoints for Azure ML — keeps training traffic off the public internetAzure Firewall rules — explicit egress control for PyPI, Hugging Face, and model download endpointsHub-Spoke VNet peering — each spoke connects to hub, never directly to each otherPrivate Link is not optional for production AI. It is the difference between we hope nothing leaks and nothing can leak.Cost Controls That Actually WorkAI infrastructure costs can spiral. Here are the controls I have seen make the biggest difference:Dedicated cost allocation tags: Tag every resource with cost-center, project, and environment. Azure Cost Management can then attribute GPU spend to specific teams.Auto-shutdown for training: AML Compute Clusters auto-scale to zero. Do not leave GPU VMs running 24/7 for weekly training jobs.Budget alerts: Set Azure Budgets with 80% and 100% thresholds on the AI subscription. Alert the team, not just finance.Spot for training, reserved for inference: Training is interruptible; inference is not. Match the purchase model to the workload.Common MistakesThree patterns I see repeated across organizations:Mistake 1: Putting everything in one subscription. AI compute, data, and serving in the same subscription means quota contention, billing confusion, and blast radius problems. Split them.Mistake 2: Skipping Private Link. We will add it later always means we will never add it. Design for private endpoints from day one.Mistake 3: No model versioning. If your team cannot tell you which model version is running in production right now, you have a problem. The ML Model Registry is free to use — there is no excuse.Disaster Recovery for AI WorkloadsAI infrastructure has unique DR considerations. Your training data in ADLS Gen2 needs geo-redundancy (GRS or ZRS). Your model registry in Azure ML is region-locked, so back up model artifacts to a secondary region via ACR geo-replication. For inference endpoints, deploy to a secondary region with traffic manager failover. The key insight: DR for AI is not just about VMs and databases — it is about preserving the entire training pipeline state, including data lineage, model versions, and container images.Putting It All TogetherThe landing zone is not a one-time project. It evolves as your AI workloads mature. Start with the hub-spoke topology and GPU-dedicated subscription. Add the model registry and data platform spokes as your team scales. Enforce Private Link from day one. Tag everything for cost attribution. And most importantly — treat your AI infrastructure with the same discipline as your production application infrastructure. The GPU does not get a pass on operational excellence.Wrapping UpAzure Landing Zones for AI are not a different framework — they are the same hub-spoke topology with GPU-specific considerations, dedicated subscriptions for quota management, a model registry lifecycle, and private networking for data movement. The fundamentals do not change; the operational details do.If you are starting an AI initiative on Azure, invest the time in the landing zone first. It is cheaper than fixing it after your data scientists have already built a shadow IT empire. --- ## 2026 Is the Year of Multi-Agent Systems - Here Is What I Have Learned Building One URL: https://wenfeng.my/multi-agent-systems-2026-lessons/ Author: admin Published: 2026-05-01 19:18:47 Modified: 2026-05-01 19:18:47 ⚡ TL;DREveryone is declaring 2026 the year of multi-agent systems. Deloitte, industry analysts, and half the LinkedIn posts in my feed are all converging on the same thesis. But here is the thing — most people are still stuck on single-agent prototypes.The Multi-Agent Moment Is HereWhy Single Agents Hit a WallWhat Actually Changes with Multi-Agent SystemsThe Architecture That WorksPitfalls I Hit Along the WayThe Multi-Agent Moment Is HereEveryone is declaring 2026 the year of multi-agent systems. Deloitte, industry analysts, and half the LinkedIn posts in my feed are all converging on the same thesis. But here is the thing — most people are still stuck on single-agent prototypes. They built a chatbot that can fetch a web page and called it AI.I have spent the last six months building real multi-agent orchestration with Hermes Agent and OpenClaw. Not demos. Not prototypes. Systems that run on schedules, coordinate across tools, and actually deliver value without me babysitting them. The gap between I built an agent and I run a system of agents is enormous, and most content skips right over it.This is what I have learned.Why Single Agents Hit a WallA single AI agent — even a good one — has a fundamental limitation: it can only do one thing well at a time. You can give it tools, chain prompts, add RAG, but it is still one brain juggling one context window. The moment you need it to research and write and optimize SEO and post to social media, you are asking one entity to context-switch between fundamentally different skills.That is not how good teams work. And it is not how good AI systems should work either.The real unlock in 2026 is not bigger models or longer context windows. It is delegation. Letting specialized agents handle specialized tasks, coordinated by an orchestrator that knows when to hand off, when to wait, and when to escalate.What Actually Changes with Multi-Agent Systems1. Specialization Over GeneralizationInstead of one model trying to be good at everything, you build agents with focused mandates. My Research Scout does not write content — it researches and reports. My Content Strategist does not search the web — it plans. My Technical Writer does not research trends — it writes articles. Each agent has a narrow scope, and that constraint makes it better at its job.This is the same principle behind why DevOps teams split responsibilities: a CI/CD pipeline agent does not try to also be a monitoring agent. Specialization is not a limitation — it is a feature.2. Orchestration Is the Real SkillThe hard part is not building individual agents. It is building the orchestration layer that coordinates them. With Hermes Agent, I use scheduled cron jobs to create a pipeline: Research Scout runs daily, Content Strategist runs weekly, and on-demand agents (Technical Writer, Content Writer) are triggered by the strategists output.The key patterns I have found:Structured handoffs — Agent A produces output in a known format; Agent B consumes it. No freeform text passing between agents.Scheduled triggers — Cron-based execution with fixed schedules. Research daily, strategy weekly, writing on-demand. Predictable cadence beats reactive chaos.Human-in-the-loop gates — Agents draft and recommend. Nothing publishes without review. This is not about removing humans — it is about removing the busywork that prevents humans from focusing on quality decisions.3. Cost Optimization Through Model SelectionNot every agent needs the most expensive model. My Research Scout runs on mimo-v2.5 — it is cost-optimized for search-and-summarize tasks. My Technical Writer uses a more capable model for nuanced technical prose. This is how you run a multi-agent system without burning through your budget.The math is simple: a daily cron job using a cheap model costs pennies. A weekly strategy job using a mid-tier model costs less than a cup of coffee. You are paying for orchestration, not for every agent to be GPT-4 level.The Architecture That WorksHere is the actual pipeline I have built and refined over the last few months:Research Scout (daily, 12:15 AM MYT) — Searches for trending topics in Azure, AI agents, DevOps, and cloud infrastructure. Produces a structured report with 5 topic suggestions, each with content angles, platform recommendations, and taxonomy tags.Content Strategist (weekly, Sunday) — Takes the weeks research findings and creates a content calendar. Blog topic with outline, X thread structure, LinkedIn post angles, Instagram carousel plan.Technical Writer (on-demand) — Triggered by the strategist. Drafts long-form blog articles (1,200-2,000 words) and posts them as drafts to Ghost via the Admin API. Never publishes — always draft, always human review.Content Writer (on-demand) — Triggered by published blog posts. Creates platform-specific social media content: X threads, LinkedIn posts, Facebook shares, Instagram carousel scripts.SEO and Analytics Agent (weekly) — Monitors GA4, Clarity, and Google Search Console. Pre-publish SEO audits, post-publish performance tracking, competitive analysis.The critical design decision: agents draft, humans publish. Every output is a draft. The orchestrator delivers recommendations to me for review. I approve, edit, or reject. This keeps quality high while eliminating the research-to-draft busywork that used to eat hours.Pitfalls I Hit Along the WayContext window pollution. If your orchestrator accumulates too much state, agents start hallucinating context from earlier runs. Keep agent state minimal. Save detailed configs to files, not memory.Timeout cascades. If one agent in a chain times out, downstream agents get garbage input. Set explicit timeouts for each agent and implement graceful degradation — a failed Research Scout should not kill the Content Strategists week.Cron timezone confusion. UTC is default. If you are in Malaysia (MYT = UTC+8), you need to convert explicitly. I schedule everything in UTC and annotate with MYT equivalents in comments. One wrong conversion and your Research Scout runs at noon instead of midnight.Model quality vs cost. Cheap models are great for search-and-summarize but terrible for technical writing. I tested several combinations and found that Research Scout on a cost-optimized model plus Technical Writer on a capable model hits the right balance. Do not optimize for cost everywhere — optimize where it matters.The Real Value PropositionHere is what this system actually gives me:Research time: ~0 minutes/day. The Research Scout handles daily topic discovery. I review a 5-item report in 2 minutes.Content planning: ~5 minutes/week. The Content Strategist produces a full weekly plan. I adjust priorities, not start from scratch.Draft writing: ~0 minutes for first drafts. The Technical Writer produces structured, on-brand articles. I edit and refine, which is where the real value is.Social media prep: ~5 minutes per post. The Content Writer produces platform-specific drafts. I tweak tone and post.Total time savings: roughly 10-15 hours per week compared to doing everything manually. And the consistency is better — I never miss a daily post because the pipeline runs regardless of how busy I am.Should You Build This?If you are a technical professional trying to build a personal brand, the answer is yes — but start small. Do not build five agents on day one. Start with one: a research agent that delivers daily topic suggestions. Once that is running reliably, add a content planner. Then a writer. Then a social media agent. Iterative complexity beats big-bang architecture.The tools are here. Hermes Agent, OpenClaw, cron-based orchestration, cheap LLM APIs. The barrier to entry is no longer technology — it is willingness to build the pipeline and commit to the cadence.2026 is the year of multi-agent systems, but only if you actually build them. Not just talk about them.What is NextI will be sharing more detailed architecture breakdowns, including the actual cron configurations, agent prompts, and deployment setup. If you are building something similar, I would love to hear about it — what patterns are working for you, what is still broken? --- ## Hermes Agent vs OpenClaw: What I Learned Running Both URL: https://wenfeng.my/hermes-agent-vs-openclaw/ Author: admin Published: 2026-05-01 07:43:26 Modified: 2026-05-01 07:43:26 ⚡ TL;DRI have been running Hermes Agent and OpenClaw side by side as part of my own AI operating environment. Both are useful, but they are not solving the same problem.The short versionDecision matrixWhere Hermes works wellWhere OpenClaw works wellThe architecture lessonI have been running Hermes Agent and OpenClaw side by side as part of my own AI operating environment. Both are useful, but they are not solving the same problem.This is not a benchmark. It is a field note from wiring agents into infrastructure, messaging platforms, content workflows, analytics, and day-to-day technical operations.The short versionHermes Agent is strongest as the operator cockpit. It gives me one place to use tools, memory, files, browser automation, scheduled jobs, and direct system execution.OpenClaw is strongest as a gateway and multi-agent runtime pattern. It works well when the workflow involves Discord, WhatsApp, delegated workers, and routing work between specialized agents.I do not see them as mutually exclusive. Hermes is where I operate. OpenClaw is where I experiment with distributed agent surfaces and gateway patterns.Decision matrixPersonal operations: Hermes AgentDirect system actions: Hermes AgentScheduled automation: Hermes AgentMessaging gateway experiments: OpenClawMulti-agent routing: OpenClawOperational visibility: Hermes currently feels easier to inspect end-to-endWhere Hermes works wellHermes is effective when a task needs tool access and continuity. It can inspect files, run commands, create cron jobs, use browser automation, generate documents, and keep enough memory to avoid re-explaining the environment every time.That makes it suitable for personal operations: maintaining this blog, checking analytics, creating proposals, updating infrastructure, writing automation, and coordinating small technical workflows.Where OpenClaw works wellOpenClaw is useful when the problem involves multiple agents and external messaging surfaces. The gateway model works well for routing work between Discord, WhatsApp, and background workers.The trade-off is observability. Once work is delegated agent-to-agent, I need a reliable way to see progress, failures, and intermediate state. Without that, the system can feel like a black box.The architecture lessonThe important question is not which agent framework is better. The better question is: where should orchestration live, and how much operational visibility do you need?For production-style agent systems, I care about identity, audit trail, permissions, secrets handling, retries, observability, and rollback. Fancy model output is secondary if the operating model is weak.My current viewI would use Hermes as the principal operator interface for personal and technical workflows. I would use OpenClaw where messaging gateways and multi-agent delegation are the core requirement.The long-term direction is a hub-and-spoke model: one orchestrator, specialized workers, clear task ownership, and all results flowing back through the orchestrator.That is the pattern I am building toward: practical AI agents that help with real cloud and architecture work, not just demos. ---