Databricks Integration

Connect Qualia to your Databricks workspace so it can reach your lakehouse data over two paths: Genie, which answers natural-language questions over a curated Genie Space, and the SQL warehouse, which runs queries directly. Both the desktop app and Qualia Cloud connect the same way, from the same settings page. Where they differ — who configures the workspace, and where credentials are stored — is called out below. Under OAuth U2M — the default — Qualia queries Databricks as you, not as a shared service account, so Unity Catalog enforces your own permissions on both paths. The two paths differ in where the query runs: a SQL warehouse query runs on the warehouse you select, so you need access to that warehouse as well as to the data; a Genie question runs on the compute embedded in the Genie Space, so you need to be able to run the Space itself, and Unity Catalog still decides which rows come back. Under OAuth M2M or token federation a service principal’s permissions apply instead of yours.

Qualia uses the connected user’s or service principal’s Databricks permissions. Grant read-only access to the tables you want to explore if writes must be prevented.

Prerequisites

Before connecting, your workspace needs:

Unity Catalog enabled, with at least one catalog you want Qualia to read.
A SQL warehouse. Serverless is recommended — it starts in seconds, where a classic warehouse can take several minutes on the first query.
A Genie Space covering the tables Qualia should answer questions about. Selected during setup and checked by the access preflight.
Partner-powered AI features enabled in workspace admin settings. Genie does not answer without it.
An OAuth application registered in your Databricks account console. Databricks does not support automatic app registration, so this step cannot be skipped.

Required privileges

Registering the OAuth application needs someone with access to the Databricks account console. On Qualia Cloud, saving the workspace configuration also needs an organization admin in Qualia; on desktop there is one install and one user, so you configure it yourself. Each person who then signs in needs the following in Databricks:

Privilege	On	Why
`databricks-sql-access` entitlement	The workspace	Query a SQL warehouse
`CAN USE`	The SQL warehouse	Run queries
`CAN RUN`	The Genie Space	Ask Genie questions
`USE CATALOG`	Each catalog to read	Resolve three-level names
`USE SCHEMA`	Each schema to read	Resolve three-level names
`SELECT`	Each table to read	Read data

These are deliberately non-admin privileges. Nobody needs workspace-admin rights to use the integration.

Registering the OAuth application

In the Databricks account console, create an OAuth application:

Note the client ID. Create a client secret as well if you want a confidential client; a public client works without one.
Add the redirect URL shown in Qualia’s Databricks settings. It must match exactly. If the desktop callback port is already in use, set DATABRICKS_CALLBACK_PORT and register the updated URL.
Grant the scopes sql, genie, unity-catalog, offline_access, openid, email, and profile.

Connecting

Setup is two steps, and the settings page labels them as such. Step 1, Workspace points Qualia at a Databricks workspace and its OAuth application. Step 2, Your access is your personal sign-in, and stays disabled until step 1 is saved.

Go to Settings → Integrations → Databricks.
Enter your workspace URL (for example dbc-a1b2c3d4-e5f6.cloud.databricks.com) and pick the cloud it runs on.
Paste the OAuth client ID, and the client secret if you created one.
Confirm the redirect URL matches what you registered.
Click Save and continue. Qualia verifies the workspace responds before storing anything.
Under Your access, click Sign in with Databricks and authorize.

After signing in, Qualia runs a set of access checks and lists the Genie Spaces and SQL warehouses you can reach. Pick one of each. Any failed check comes with the specific grant needed to fix it. You can also pick a default catalog and schema. Qualia lists the catalogs you can see, then the schemas inside the one you choose. These set what an unqualified table name resolves against; generated SQL still uses full catalog.schema.table names, so the default is a convenience rather than something queries depend on. On Qualia Cloud, every other member of your organization then signs in individually from the same settings page: the workspace configuration is shared, the authorization is not. The warehouse, Genie Space, catalog, and schema are part of that shared configuration, so only an organization admin can change them — everyone else signs in and uses what the admin picked. On desktop the configuration and the authorization are both yours alone.

Authentication methods

Method	Used for	Notes
OAuth U2M (authorization code + PKCE)	Interactive use	The default, and the only method the desktop app supports. Each person authorizes as themselves.
OAuth M2M (client credentials)	Automation, Qualia Cloud only	Uses a service principal. Its permissions apply instead of a person’s.
Token federation (workload identity)	Automation, Qualia Cloud only	Reaches a service principal with no stored secret. Chosen in settings under Query as, and preferred over M2M where both are configured.

Use one of the OAuth methods above; personal access tokens are not supported for queries.

Setting up a service principal (M2M)

Use this when work runs without a person present to authorize it. The service principal’s Unity Catalog grants apply instead of the requesting user’s, so grant it only what the automation needs.

In your Databricks account console, go to User management → Service principals and create one.
On the service principal, generate an OAuth secret. Record the client ID and secret; the secret is shown once.
Grant it the same privileges a person needs — CAN USE on the SQL warehouse, USE CATALOG and USE SCHEMA on the parents, and SELECT on the tables.
Provide the client ID and secret through the Databricks Partner Connect tile. Nothing in Qualia’s settings page accepts an M2M secret, so a workspace you connected by hand should use token federation below instead — it needs no secret at all.

Setting up token federation

Token federation lets Qualia Cloud query as a service principal without a saved client secret. Configure a federation policy in Databricks to authorize it.

Create a service principal as above — steps 1 and 3, skipping the OAuth secret.
In Qualia, open Settings → Integrations → Databricks. Under Authentication method, set Query as to Service principal, then enter the service principal’s application ID. Leave Federation audience blank unless your policy needs a specific one. Save. Note what this changes: every query then runs as that one service principal, so its Unity Catalog grants apply to everyone in your organization and no answer is attributable to an individual. Leave Query as on Each person if you want per-user governance.
A Federation policy section appears with the exact values Qualia will sign. They are specific to your organization, and each one has a copy button.
In your Databricks account console, open the service principal and add a federation policy:

Field	Value
Issuer URL	Exactly `https://api.quadrillion.ai`, with no trailing slash
Subject	`qualia:org:<organization>:databricks:<connection>`, exactly as Qualia shows it
Audiences	`databricks`, unless you set a different audience in Qualia

Leave token signature validation empty. Databricks then fetches the key from https://api.quadrillion.ai/.well-known/openid-configuration, which is what lets Qualia rotate signing keys without you touching the policy. Copy the issuer, subject, and audience exactly from Federation policy, including any trailing slashes. Nobody signs in individually under Service principal, so step 2 offers no Sign in with Databricks button. The warehouse and Genie Space pickers appear as soon as the connection is saved, and what they list is what the service principal can reach.

Partner Connect

If you reach Qualia through the Databricks Partner Connect tile, Databricks provisions the workspace details for you and sends you to Qualia to finish. An organization admin lands on the settings page with the workspace already filled in and only needs to confirm it, then pick a warehouse and Genie Space. Re-running the tile for a workspace that is already connected updates it in place rather than creating a second connection, and re-enables one that a trial expiry had disabled. If your trial lapses, the connection is disabled rather than deleted: existing authorizations stop working immediately, and converting restores the setup instead of making you redo it.

Querying from a notebook

Use the Databricks helper from Python, R, or Julia notebooks to query with your saved connection. In Python the helper needs the databricks-sql-connector[pyarrow] package. Qualia Cloud kernels already have it. On desktop, the first query reports that it is missing and the agent offers to install it for you — a one-time step per kernel environment, not something you repeat.

import qualia_databricks

df = qualia_databricks.query("SELECT * FROM main.sales.orders LIMIT 100")

df <- databricks_query("SELECT * FROM main.sales.orders LIMIT 100")

using DataFrames
df = DataFrame(Qualia.Databricks.query("SELECT * FROM main.sales.orders LIMIT 100"))

All three return a table — a pandas DataFrame, an R data.frame, and a column table Julia’s DataFrame accepts directly — and all three take catalog and schema to override the defaults saved on the connection. workspace_info() reports which workspace, warehouse, catalog, and schema a notebook is pointed at, without exposing the token. Pass values as parameters rather than building them into the SQL string. The warehouse then parses them as data and never as SQL:

qualia_databricks.query(
    "SELECT * FROM main.sales.orders WHERE total > :floor",
    {"floor": 100},
)

databricks_query(
  "SELECT * FROM main.sales.orders WHERE total > :floor",
  list(floor = 100)
)

Qualia.Databricks.query(
    "SELECT * FROM main.sales.orders WHERE total > :floor";
    parameters = Dict("floor" => 100),
)

In Python, query opens its own connection. For several statements, use connect instead, which returns a DB-API connection you can reuse:

with qualia_databricks.connect() as connection:
    with connection.cursor() as cursor:
        cursor.execute("SELECT count(*) FROM main.sales.orders")
        total = cursor.fetchone()[0]

Always use the three-level catalog.schema.table name. A bare table name resolves against whatever default the connection carries, which is the most common cause of an unexpected “table not found”.

The R and Julia clients read results over the Databricks SQL Statement Execution API, which caps a single result at 25 MiB. Aggregate or LIMIT in SQL rather than pulling a whole table into the kernel. Python has no such cap.

Asking Genie

With a Genie Space selected, the agent can put a question to it in plain language and show you the answer with a link back to the Space. Genie knows the joins and business definitions your admin curated there, so it is often a better first stop than SQL written against tables nobody has inspected yet. Genie returns an answer to read, not data in your notebook. Ask Genie to orient, then query through the client above when you need values you can compute on — a Genie answer cannot be captured as evidence for a finding.

Reconnecting

If Settings shows reconnect required, click Re-authorize and complete sign-in.

Rotating the client secret

Enter the new value in the OAuth client secret field and save. Leaving it blank keeps the stored secret, so an unrelated edit never overwrites it. Existing user authorizations survive a rotation.

Disconnecting

Disconnect removes your own authorization and discards your stored tokens immediately. Any open Genie session is closed at the same time. On Qualia Cloud, other members are unaffected. Remove deletes the whole workspace configuration along with every authorization it covers. On Qualia Cloud only an organization admin can do this.

Where credentials are stored

Credentials are encrypted in Qualia Cloud and protected by your operating system’s keychain on desktop. See API key storage for systems without a keychain. If the keychain is locked, unlock it and retry. If a locked keychain makes Databricks appear disconnected, unlock it and refresh, or sign in again. If Qualia specifically asks you to unlock the keychain for a saved client secret, unlock it before retrying. Query results remain in your notebook and Genie answers remain in chat after you disconnect.

Troubleshooting

Symptom	Fix
”Could not reach Databricks at …”	Check the workspace URL. It should be the hostname only, with no path.
Sign-in returns to a failure page	The redirect URL registered in Databricks does not exactly match the one in Qualia settings.
No Genie Space listed	Either none exists, or you lack `CAN RUN` on it. Genie Spaces are created in Databricks.
No SQL warehouse listed	You lack `CAN USE` on any warehouse.
Genie returns nothing useful	The Space may not cover the tables in question. Genie only sees what its Space includes.
Queries are slow to start	A classic warehouse is starting. Switch to serverless if that matters.
Cloud: “This Databricks connection is disabled”	The connection was retired, usually when a Partner Connect trial ended. Re-run the Qualia tile in Partner Connect, or remove the connection and set it up again. Partner Connect provisions Qualia Cloud only, so a desktop connection is never disabled this way.
”reconnect required” keeps returning	The OAuth application may be missing `offline_access`. Without it, Databricks issues no refresh token.
”Could not exchange the Databricks workload identity assertion”	The federation policy does not match. Compare its issuer, subject, and audience against the values Qualia shows, character for character, and confirm it is on the same service principal as the client ID you entered.
Desktop: “another process is already using the callback port”	Free that port, or set `DATABRICKS_CALLBACK_PORT`, save, and register the new URL in Databricks. Either way the sign-in still finishes: paste the callback URL from the browser’s address bar into the field Qualia shows.
Desktop: sign-in never returns to Qualia	Use the “Didn’t return to Qualia?” link under the sign-in button and paste the callback URL from the browser’s address bar.
Desktop over SSH / remote backend: browser shows connection refused after authorizing	The callback lands on the machine running the browser, not the remote backend. Copy the full callback URL from that browser’s address bar and paste it into the sign-in form Qualia shows.
Desktop: “unlock your keychain and try again”	The OS keychain holds your saved OAuth client secret and refused to hand it over. Unlock it and retry; the connection is intact.
Desktop: shows disconnected right after the keychain locked	Expected. Unlock the keychain and refresh, or sign in again; either way nothing is lost.
Desktop: the settings page says connected right after signing in with the keychain locked	Expected. The tokens are in Qualia’s restricted file and move into the keychain the next time it answers.
`databricks-sql-connector is not installed`	A desktop or custom kernel — cloud kernels ship it. Ask the agent to install it; it stays installed for that environment.
Desktop: the connector goes missing again	Rebuilding a managed environment recreates its virtualenv from scratch. Add `databricks-sql-connector[pyarrow]` to that environment’s package list so it is reinstalled every time.

Attribution

Qualia identifies itself to Databricks on every request so your workspace admins can attribute usage in system.access.audit. Genie answers show Powered by Genie and link to their source Space.

Welcome

Core Features

Research Workflow

Tools

Customization & Settings

Interface

Databricks Integration

Prerequisites

Required privileges

Registering the OAuth application

Connecting

Authentication methods

Setting up a service principal (M2M)

Setting up token federation

Partner Connect

Querying from a notebook

Asking Genie

Reconnecting

Rotating the client secret

Disconnecting

Where credentials are stored

Troubleshooting

Attribution

​Prerequisites

​Required privileges

​Registering the OAuth application

​Connecting

​Authentication methods

​Setting up a service principal (M2M)

​Setting up token federation

​Partner Connect

​Querying from a notebook

​Asking Genie

​Reconnecting

​Rotating the client secret

​Disconnecting

​Where credentials are stored

​Troubleshooting

​Attribution

Prerequisites

Required privileges

Registering the OAuth application

Connecting

Authentication methods

Setting up a service principal (M2M)

Setting up token federation

Partner Connect

Querying from a notebook

Asking Genie

Reconnecting

Rotating the client secret

Disconnecting

Where credentials are stored

Troubleshooting

Attribution