Below is one recommended solution:

1. Convert your notebook job into an Azure Machine Learning Pipeline or experiment run. You can do this either by (a) refactoring the notebook into a Python script (or using an “Execute Notebook” pipeline step) or (b) wrapping the logic in a pipeline step that can run on your AML compute cluster.

2. From your Databricks notebook, authenticate to your Azure ML workspace and then trigger the run. You can do this in one of two ways:

• Using the Azure ML SDK—for example:

---

Make sure you have installed the Azure ML SDK in your Databricks cluster:

%pip install azureml-core

from azureml.core import Workspace, Experiment
from azureml.pipeline.core import PublishedPipeline

Connect to your Azure ML Workspace (adjust parameters)

ws = Workspace(subscription_id='your_subscription_id',
resource_group='your_resource_group',
workspace_name='your_workspace_name')

Option A: If you have an Azure ML experiment set up that runs a script (or notebook) on your compute cluster:

experiment = Experiment(workspace=ws, name='triggered_experiment')

Assuming you have a run configuration (for example, a ScriptRunConfig for a Python script version of your notebook)

from azureml.core import ScriptRunConfig

run_config = ScriptRunConfig(source_directory='./',
script='your_script.py',
compute_target='your_aml_compute_cluster')

run = experiment.submit(run_config)
print("Run ID: ", run.id)

Option B: If you have converted your notebook to an Azure ML Pipeline and published it

published_pipeline = PublishedPipeline.get(ws, id='your_pipeline_id')
pipeline_run = published_pipeline.submit(ws, experiment_name='triggered_pipeline_experiment')
print("Pipeline Run ID:", pipeline_run.id)

• Or, using the AML REST API:

– First, deploy your AML Pipeline as a REST endpoint.
– Then, from your Databricks notebook, use a simple HTTP call (using Python’s requests library) to trigger an execution. For example:

---

import requests
import json

Set up your endpoint URL, access token, etc.

(You might obtain an access token using a service principal, managed identity, or interactive login)

endpoint_url = "https://.api.azureml.ms/pipelines/v1.0/subscriptions/your_subscription/resourceGroups/your_resource_group/workspaces/your_workspace_name/pipelines/your_pipeline_id/createJob?api-version=2022-02-01-preview"
access_token = "YOUR_ACCESS_TOKEN" # Get this via Azure AD

Define your job parameters if needed:

job_payload = {
"experiment_name": "triggered_pipeline_experiment",

add additional parameters if your pipeline expects them

}

headers = {
"Authorization": "Bearer " + access_token,
"Content-Type": "application/json"
}

response = requests.post(endpoint_url, headers=headers, data=json.dumps(job_payload))
if response.ok:
job_response = response.json()
print("Pipeline triggered successfully. Job details:", job_response)
else:
print("Failed to trigger the pipeline:", response.text)

3. Choose the approach that best fits your workflow. If you prefer to stay entirely within Python, using the AML SDK from Databricks is straightforward. If you need to trigger the job from any REST-capable environment or want it decoupled from your Databricks cluster’s Python environment, the REST API approach is best.

Key things to note:

• Make sure that your Databricks cluster has network access to your Azure ML workspace endpoints.
• Ensure that authentication is correctly set up, whether through an interactive login, service principal, or managed identity.
• If you originally developed long-form notebooks in AML Studio, consider refactoring them into pipeline steps or Python scripts to easily integrate with the programmatic job submission options.

This is the recommended approach to triggering an Azure ML Studio notebook (or job that replicates its functionality) from a Databricks notebook using available Azure APIs and services.