Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
Azure Databricks on Azure Government is in Beta. To join, register a new Azure Government subscription for the Azure Databricks New GovCloud environment preview feature. See Get started with the Beta.
This article describes Azure Databricks on Microsoft Azure Government, including compliance controls, how to join the Beta of the new Azure Government environment, supported regions and URLs, data residency, and supported system tables.
Azure Government overview
Azure Government is a dedicated Microsoft Azure cloud for United States government agencies and their partners. It is physically isolated from the global Azure cloud and operated by screened US persons to meet the compliance requirements of US federal, state, local, and tribal governments. For details, see Microsoft Azure Government.
Azure Databricks on Azure Government provides the Azure Databricks platform deployed in Azure Government with compliance and security controls. It is available in the Azure Government usgovva and usgovaz regions.
Every workspace has the compliance security profile and enhanced security monitoring turned on by default, requires Azure Private Link for front-end and back-end connections, and stores and processes customer content only in US government cloud regions.
Beta workspaces run in a new environment that is separate from existing Azure Databricks workspaces in Azure Government. Existing workspaces aren't migrated. Processing Controlled Unclassified Information (CUI) or other regulated data isn't allowed.
Requirements
- An Azure Government subscription that has never contained a Azure Databricks workspace. Don't register a subscription that contains a Azure Databricks workspace, previously contained a Azure Databricks workspace that was later deleted, or is associated with the existing Azure Databricks environment in Azure Government.
- Permission to register preview features on the subscription.
Important
You are solely responsible for verifying that sensitive information is never entered in customer-defined input fields, such as workspace names, compute resource names, tags, job names, job run names, network names, credential names, storage account names, and Git repository IDs or URLs. These fields might be stored, processed, or accessed outside the compliance boundary.
Compliance
Azure Databricks on Microsoft Azure Government is in Beta. Azure Databricks is working to obtain authorization for the compliance regimes. Until authorization is complete, do not process Controlled Unclassified Information (CUI) or any other data regulated by these standards.
Note
Azure Private Link is required for both back-end and front-end connections. For details, see Private Link concepts.
Get started with the Beta
Before you register
Use a dedicated Azure Government subscription that has never been used to create an Azure Databricks workspace. Do not register a subscription that:
- Contains an existing Azure Databricks workspace.
- Previously contained an Azure Databricks workspace that was later deleted.
- Is already associated with the existing Azure Databricks environment in Azure Government.
The Beta uses a subscription-level feature registration to determine which environment receives workspace operations. After you register the new-environment feature, new Azure Databricks workspaces that you create in that subscription are routed to the new environment. To register the feature, you need permission to register preview features on the subscription. For details, see Set up preview features in Azure subscription.
Important
You must register the Azure Databricks New GovCloud environment feature before you create any Azure Databricks workspaces in the subscription. Workspaces created before the feature is registered are not created in the new environment.
Register the preview feature
- Sign in to the Azure Government portal.
- Open Subscriptions.
- Select the Azure Government subscription that you want to use for the Beta.
- Under Settings, select Preview features.
- Search for the Azure Databricks New GovCloud environment feature.
- Select the feature, and then select Register.
- Wait until the registration state shows Registered.
- Create a new Azure Databricks workspace by using the standard Azure Government workspace-creation experience. See Deploy a workspace using the Azure Portal.
Note
A subscription can be associated with only one Azure Databricks environment in Azure Government: the existing environment or the new environment. A subscription becomes associated with the existing environment when it is used to create an Azure Databricks workspace, or when the existing-environment feature is registered on it. Azure Resource Manager uses this association to decide where to route workspace operations, so it needs a single, unambiguous destination. If you register the Azure Databricks New GovCloud environment feature on a subscription that is already associated with the existing environment, workspace creation and other workspace operations are blocked to protect your existing resources.
Resolve a registration conflict
If workspace creation or other workspace operations fail in a subscription associated with the existing Azure Databricks environment in Azure Government, the subscription might also be registered for the Azure Databricks New GovCloud environment feature. A subscription can't be associated with both environments. Azure Resource Manager requires a single destination for workspace operations, so it blocks them to protect existing resources.
Don't retry workspace creation. Remove the conflicting registration instead.
Warning
Don't unregister the feature from a subscription that has active Beta workspaces unless Microsoft Support instructs you to.
In the Azure Government portal, go to Subscriptions > your subscription > Settings > Preview features, and check whether Azure Databricks New GovCloud environment is registered. See List preview features.
Set the Azure CLI to Azure Government and sign in:
az cloud set --name AzureUSGovernment az loginUnregister the feature. Replace
<subscription-id>with your subscription ID:az feature unregister --namespace Microsoft.Databricks --name UseEverGreenControlPlane --subscription <subscription-id>To use the portal instead, see Unregister preview feature.
Confirm that the feature is no longer registered:
az feature show --namespace Microsoft.Databricks --name UseEverGreenControlPlane --subscription <subscription-id>Retry the workspace operation.
If you can't determine whether the subscription was previously used with Azure Databricks, or the conflict persists, contact support as described in Get support.
To join the Beta, register a new Azure Government subscription that has never contained an Azure Databricks workspace.
Beta limitations
During the Beta:
- Access is enabled at the subscription level.
- The subscription must not have current or prior Azure Databricks workspace usage.
- New workspaces in the registered subscription use the new environment.
- Existing workspaces are not migrated.
- Production and Controlled Unclassified Information (CUI) workloads are not supported.
- Feature availability differs from the generally available Azure Databricks service. For example, only a subset of system tables is available. See Supported system tables.
- Each customer is responsible for its own internal authorization and Authority to Operate (ATO) requirements before production use.
Existing workspaces
Existing workspaces remain associated with the environment in which they were created. Beta registration does not migrate existing workspaces, notebooks, data, configurations, or other workspace resources. Registering the new-environment feature is not a migration method: do not enable it on a subscription that contains existing or previously deleted Azure Databricks workspaces.
Get support
Use the standard Azure Government support process. In the support request, state that the affected subscription or workspace is participating in the Beta of the new Azure Databricks on Microsoft Azure Government environment. Include the subscription ID, the workspace name, ID and URL, the region, the feature-registration state, the error message, and the operation you were attempting.
Compliance security profile
The compliance security profile is enabled on all Azure Databricks workspaces on Azure Government by default. The compliance security profile has additional monitoring, enforced instance types for inter-node encryption, a hardened compute image, and other features that help meet compliance requirements. Automatic cluster update and enhanced security monitoring are also automatically enabled.
The compliance security profile enforces the use of VM instance types that support Azure Virtual Network encryption, which provides hardware-implemented network encryption between cluster nodes. For requirements, see Compliance security profile requirements.
Azure Databricks on Azure Government regions and URLs
Azure Databricks on Azure Government is available in the usgovva and usgovaz regions. See Azure Databricks regions.
Azure Databricks on Azure Government URLs differ from Azure Databricks URLs on the commercial offering. Use the following URLs for Azure Databricks on Azure Government:
- Account console URL and base URL for REST APIs:
https://accounts.usgov.databricks.azure.us/ - Workspace URL and base URL for workspace-level REST APIs:
https://adb-<workspace-id>.<random-number>.usgov.databricks.azure.us/
After you create the workspace, you can confirm its URL on the workspace Overview page in the Azure Government portal.
Use the account console URL to establish your first account admin. For steps, see Establish your first account admin. To complete these steps, you must be a Microsoft Entra ID Global Administrator with elevated access. For details, see Elevate access for a Global Administrator.
Data residency
Data residency is always enforced for Azure Databricks on Azure Government workspaces. Customer content is stored and processed only in US government cloud regions. The cross-Geo processing setting does not apply to Azure Government workspaces and cannot be used to process data outside of US government cloud regions.
Supported system tables
The following system tables are available in the new Azure Databricks on Azure Government environment (Beta) for monitoring, governance, and cost management:
- system.access schema:
assistant_events,audit,inbound_network,outbound_network,workspaces_latest - system.billing schema:
list_prices,usage - system.compute schema:
clusters,node_timeline,node_types,warehouse_events,warehouses - system.lakeflow schema:
job_run_timeline,job_task_run_timeline,job_tasks,jobs,pipeline_update_timeline,pipelines - system.mlflow schema:
experiments_latest,run_metrics_history,runs_latest - system.query schema:
history - system.storage schema:
predictive_optimization_operations_history,table_auto_upgrade_operations_history