> ## Documentation Index
> Fetch the complete documentation index at: https://docs.quadrillion.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Databricks Integration
> Connect Qualia to your Databricks lakehouse, on desktop or in Qualia Cloud.
Connect Qualia to your Databricks workspace so it can reach your lakehouse data over two paths: **Genie**, which answers natural-language questions over a curated Genie Space, and the **SQL warehouse**, which runs queries directly. This page covers establishing that connection; the features built on it are still landing.
Both the desktop app and Qualia Cloud connect the same way, from the same settings page. Where they differ — who configures the workspace, and where credentials are stored — is called out below.
Under OAuth U2M — the default — Qualia queries Databricks as you, not as a shared service account, so Unity Catalog enforces your own permissions on both paths. The two paths differ in where the query runs: a SQL warehouse query runs on the warehouse you select, so you need access to that warehouse as well as to the data; a Genie question runs on the compute embedded in the Genie Space, so you need to be able to run the Space itself, and Unity Catalog still decides which rows come back. Under OAuth M2M or token federation a service principal's permissions apply instead of yours.
The agent is instructed to use this integration for reads only. That instruction is not enforcement: the SQL clients send the statement they are given, as you, so a connecting user who can write can write. Grant `SELECT` and nothing more if you want writes to be impossible rather than merely unasked for.
## Prerequisites
Before connecting, your workspace needs:
* **Unity Catalog enabled**, with at least one catalog you want Qualia to read.
* **A SQL warehouse**. Serverless is recommended — it starts in seconds, where a classic warehouse can take several minutes on the first query.
* **A Genie Space** covering the tables Qualia should answer questions about. Selected during setup and checked by the access preflight.
* **Partner-powered AI features enabled** in workspace admin settings. Genie does not answer without it.
* **An OAuth application** registered in your Databricks account console. Databricks does not support automatic app registration, so this step cannot be skipped.
## Required privileges
Registering the OAuth application needs someone with access to the **Databricks account console**. On Qualia Cloud, saving the workspace configuration also needs an **organization admin in Qualia**; on desktop there is one install and one user, so you configure it yourself.
Each person who then signs in needs the following in Databricks:
| Privilege | On | Why |
| ----------------------------------- | -------------------- | ------------------------- |
| `databricks-sql-access` entitlement | The workspace | Query a SQL warehouse |
| `CAN USE` | The SQL warehouse | Run queries |
| `CAN RUN` | The Genie Space | Ask Genie questions |
| `USE CATALOG` | Each catalog to read | Resolve three-level names |
| `USE SCHEMA` | Each schema to read | Resolve three-level names |
| `SELECT` | Each table to read | Read data |
These are deliberately non-admin privileges. Nobody needs workspace-admin rights to use the integration.
## Registering the OAuth application
In the Databricks **account console**, create an OAuth application:
1. Note the **client ID**. Create a **client secret** as well if you want a confidential client; a public client works without one.
2. Add the **redirect URL** shown in Qualia's Databricks settings. It must match exactly, including the path and, on desktop, the port. Databricks does not accept the arbitrary loopback port that [RFC 8252](https://datatracker.ietf.org/doc/html/rfc8252#section-7.3) allows, which is why the desktop URL names a fixed one — `http://127.0.0.1:8020/api/databricks/callback` by default. Set `DATABRICKS_CALLBACK_PORT` if something else on your machine already holds that port, then register the URL Qualia shows afterward.
3. Grant the scopes `sql`, `genie`, `unity-catalog`, `offline_access`, `openid`, `email`, and `profile`. `unity-catalog` is what lets the access preflight list your catalogs; `offline_access` lets Qualia refresh access without sending you back through the browser every hour; and `openid email profile` identify who authorized, so Qualia can bind the authorization to your Qualia account.
## Connecting
Setup is two steps, and the settings page labels them as such. **Step 1, Workspace** points Qualia at a Databricks workspace and its OAuth application. **Step 2, Your access** is your personal sign-in, and stays disabled until step 1 is saved.
1. Go to **Settings → Integrations → Databricks**.
2. Enter your **workspace URL** (for example `dbc-a1b2c3d4-e5f6.cloud.databricks.com`) and pick the **cloud** it runs on.
3. Paste the **OAuth client ID**, and the client secret if you created one.
4. Confirm the **redirect URL** matches what you registered.
5. Click **Save and continue**. Qualia verifies the workspace responds before storing anything.
6. Under **Your access**, click **Sign in with Databricks** and authorize.
After signing in, Qualia runs a set of access checks and lists the Genie Spaces and SQL warehouses you can reach. Pick one of each. Any failed check comes with the specific grant needed to fix it.
You can also pick a default catalog and schema. Qualia lists the catalogs you can see, then the schemas inside the one you choose. These set what an unqualified table name resolves against; generated SQL still uses full `catalog.schema.table` names, so the default is a convenience rather than something queries depend on.
On Qualia Cloud, every other member of your organization then signs in individually from the same settings page: the workspace configuration is shared, the authorization is not. The warehouse, Genie Space, catalog, and schema are part of that shared configuration, so only an organization admin can change them — everyone else signs in and uses what the admin picked. On desktop the configuration and the authorization are both yours alone.
## Authentication methods
| Method | Used for | Notes |
| ----------------------------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| **OAuth U2M** (authorization code + PKCE) | Interactive use | The default, and the only method the desktop app supports. Each person authorizes as themselves. |
| **OAuth M2M** (client credentials) | Automation, Qualia Cloud only | Uses a service principal. Its permissions apply instead of a person's. |
| **Token federation** (workload identity) | Automation, Qualia Cloud only | Reaches a service principal with no stored secret. Chosen in settings under **Query as**, and preferred over M2M where both are configured. |
Personal access tokens are not supported for any path Qualia actually queries with. They do not expire on their own and cannot be attributed to a person, which makes them a poor fit for governed data access. One narrow exception exists for compatibility: some versions of the Databricks Partner Connect contract still send a token when provisioning, and Qualia stores what it is sent. Nothing reads it — every credential Qualia queries with is an OAuth token — so a workspace provisioned that way has no working path until an admin registers an OAuth application or configures a service principal, and re-provisioning with OAuth credentials removes the stored token.
### Setting up a service principal (M2M)
Use this when work runs without a person present to authorize it. The service principal's Unity Catalog grants apply instead of the requesting user's, so grant it only what the automation needs.
1. In your Databricks **account console**, go to **User management → Service principals** and create one.
2. On the service principal, generate an **OAuth secret**. Record the client ID and secret; the secret is shown once.
3. Grant it the same privileges a person needs — `CAN USE` on the SQL warehouse, `USE CATALOG` and `USE SCHEMA` on the parents, and `SELECT` on the tables.
4. Provide the client ID and secret through the Databricks **Partner Connect** tile. Nothing in Qualia's settings page accepts an M2M secret, so a workspace you connected by hand should use token federation below instead — it needs no secret at all.
Qualia never stores an M2M access token. One is minted per call from the client credentials and discarded.
### Setting up token federation
Token federation reaches the same service principal without a secret stored anywhere. Qualia signs a short-lived assertion, a policy you control decides whether to trust it, and Databricks returns a token. You have no secret to rotate or leak; Qualia rotates its own signing keys, and the policy discovers them. Where both are configured, Qualia prefers federation over M2M.
Qualia Cloud only. The desktop app has no public address to publish verification keys from.
1. Create a service principal as above — steps 1 and 3, skipping the OAuth secret.
2. In Qualia, open **Settings → Integrations → Databricks**. Under **Authentication method**, set **Query as** to **Service principal**, then enter the service principal's **application ID**. Leave **Federation audience** blank unless your policy needs a specific one. Save.
Note what this changes: every query then runs as that one service principal, so its Unity Catalog grants apply to everyone in your organization and no answer is attributable to an individual. Leave **Query as** on **Each person** if you want per-user governance.
3. A **Federation policy** section appears with the exact values Qualia will sign. They are specific to your organization, and each one has a copy button.
4. In your Databricks **account console**, open the service principal and add a **federation policy**:
| Field | Value |
| ---------- | ------------------------------------------------------------------------------- |
| Issuer URL | Qualia's public API origin, for example `https://api.quadrillion.io` |
| Subject | `qualia:org::databricks:`, exactly as Qualia shows it |
| Audiences | `databricks`, unless you set a different audience in Qualia |
Leave **token signature validation** empty. Databricks then fetches the key from `https://api.quadrillion.io/.well-known/openid-configuration`, which is what lets Qualia rotate signing keys without you touching the policy.
The subject names your organization, so a policy pinned to it cannot be satisfied by an assertion minted for any other Qualia tenant.
Nobody signs in individually under **Service principal**, so step 2 offers no **Sign in with Databricks** button. The warehouse and Genie Space pickers appear as soon as the connection is saved, and what they list is what the service principal can reach.
## Partner Connect
If you reach Qualia through the Databricks **Partner Connect** tile, Databricks provisions the workspace details for you and sends you to Qualia to finish. An organization admin lands on the settings page with the workspace already filled in and only needs to confirm it, then pick a warehouse and Genie Space.
Re-running the tile for a workspace that is already connected updates it in place rather than creating a second connection, and re-enables one that a trial expiry had disabled. If your trial lapses, the connection is disabled rather than deleted: existing authorizations stop working immediately, and converting restores the setup instead of making you redo it.
## Querying from a notebook
A Databricks helper is available in every Python, R, and Julia kernel. It fetches a credential on your behalf and renews it as it ages, so nothing expires mid-session and no token is written into the kernel environment.
In Python the helper needs the `databricks-sql-connector[pyarrow]` package. Qualia Cloud kernels already have it. On desktop, the first query reports that it is missing and the agent offers to install it for you — a one-time step per kernel environment, not something you repeat.
```python Python theme={null}
import qualia_databricks
df = qualia_databricks.query("SELECT * FROM main.sales.orders LIMIT 100")
```
```r R theme={null}
df <- databricks_query("SELECT * FROM main.sales.orders LIMIT 100")
```
```julia Julia theme={null}
using DataFrames
df = DataFrame(Qualia.Databricks.query("SELECT * FROM main.sales.orders LIMIT 100"))
```
All three return a table — a pandas DataFrame, an R `data.frame`, and a column table Julia's `DataFrame` accepts directly — and all three take `catalog` and `schema` to override the defaults saved on the connection. `workspace_info()` reports which workspace, warehouse, catalog, and schema a notebook is pointed at, without exposing the token.
Pass values as parameters rather than building them into the SQL string. The warehouse then parses them as data and never as SQL:
```python Python theme={null}
qualia_databricks.query(
"SELECT * FROM main.sales.orders WHERE total > :floor",
{"floor": 100},
)
```
```r R theme={null}
databricks_query(
"SELECT * FROM main.sales.orders WHERE total > :floor",
list(floor = 100)
)
```
```julia Julia theme={null}
Qualia.Databricks.query(
"SELECT * FROM main.sales.orders WHERE total > :floor";
parameters = Dict("floor" => 100),
)
```
In Python, `query` opens its own connection. For several statements, use `connect` instead, which returns a DB-API connection you can reuse:
```python theme={null}
with qualia_databricks.connect() as connection:
with connection.cursor() as cursor:
cursor.execute("SELECT count(*) FROM main.sales.orders")
total = cursor.fetchone()[0]
```
Always use the three-level `catalog.schema.table` name. A bare table name resolves against whatever default the connection carries, which is the most common cause of an unexpected "table not found".
The R and Julia clients read results over the Databricks SQL Statement Execution API, which caps a single result at 25 MiB. Aggregate or `LIMIT` in SQL rather than pulling a whole table into the kernel. Python has no such cap.
## Asking Genie
With a Genie Space selected, the agent can put a question to it in plain language and show you the answer with a link back to the Space. Genie knows the joins and business definitions your admin curated there, so it is often a better first stop than SQL written against tables nobody has inspected yet.
Genie returns an answer to read, not data in your notebook. Ask Genie to orient, then query through the client above when you need values you can compute on — a Genie answer cannot be captured as evidence for a finding.
## Reconnecting
Access tokens expire roughly hourly and Qualia refreshes them silently. If a refresh fails permanently — your Databricks session ended, an admin revoked the app, or the OAuth application changed — the settings page shows **reconnect required**. Click **Re-authorize** to fix it. Nothing else is affected in the meantime.
## Rotating the client secret
Enter the new value in the **OAuth client secret** field and save. Leaving it blank keeps the stored secret, so an unrelated edit never overwrites it. Existing user authorizations survive a rotation.
## Disconnecting
**Disconnect** removes your own authorization and discards your stored tokens immediately. Any open Genie session is closed at the same time. On Qualia Cloud, other members are unaffected.
**Remove** deletes the whole workspace configuration along with every authorization it covers. On Qualia Cloud only an organization admin can do this.
## Where credentials are stored
Qualia stores your workspace configuration, the OAuth client ID and secret, and access and refresh tokens. Tokens and secrets never appear in logs, tool results, notebook environments, or API responses.
A federated connection stores no credential at all beyond the service principal's client ID, which is an identifier rather than a secret. The assertion is signed on demand and the token Databricks returns is held in memory until it expires.
On Qualia Cloud they are encrypted at rest in the database, one authorization per member. On desktop the configuration goes to your `settings.json` and the tokens to your operating system keychain — Keychain on macOS, Credential Manager on Windows, Secret Service or KWallet on Linux. If your machine has no keychain, Qualia falls back to a `0600` file under `~/.quadrillion/databricks/`, which is weaker: anything running as your user can read it.
If the keychain is locked when Qualia needs a token, the settings page reports that rather than showing you as disconnected. Unlock it and retry.
Qualia does not copy your lakehouse data. Query results live in the notebook you ran them in, and Genie answers live in the conversation. Disconnecting discards the stored tokens; removing the connection discards the configuration too.
## Troubleshooting
| Symptom | Fix |
| --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| "Could not reach Databricks at ..." | Check the workspace URL. It should be the hostname only, with no path. |
| Sign-in returns to a failure page | The redirect URL registered in Databricks does not exactly match the one in Qualia settings. |
| No Genie Space listed | Either none exists, or you lack `CAN RUN` on it. Genie Spaces are created in Databricks. |
| No SQL warehouse listed | You lack `CAN USE` on any warehouse. |
| Genie returns nothing useful | The Space may not cover the tables in question. Genie only sees what its Space includes. |
| Queries are slow to start | A classic warehouse is starting. Switch to serverless if that matters. |
| Cloud: "This Databricks connection is disabled" | The connection was retired, usually when a Partner Connect trial ended. Re-run the Qualia tile in Partner Connect, or remove the connection and set it up again. Partner Connect provisions Qualia Cloud only, so a desktop connection is never disabled this way. |
| "reconnect required" keeps returning | The OAuth application may be missing `offline_access`. Without it, Databricks issues no refresh token. |
| "Could not exchange the Databricks workload identity assertion" | The federation policy does not match. Compare its issuer, subject, and audience against the values Qualia shows, character for character, and confirm it is on the same service principal as the client ID you entered. |
| Desktop: sign-in never returns to Qualia | Something else holds the callback port. Set `DATABRICKS_CALLBACK_PORT`, save, and register the new URL in Databricks. |
| Desktop: "unlock your keychain and try again" | The OS keychain refused to hand over a stored credential. Unlock it and retry; the connection is intact. |
| `databricks-sql-connector is not installed` | A desktop or custom kernel — cloud kernels ship it. Ask the agent to install it; it stays installed for that environment. |
| Desktop: the connector goes missing again | Rebuilding a managed environment recreates its virtualenv from scratch. Add `databricks-sql-connector[pyarrow]` to that environment's package list so it is reinstalled every time. |
## Attribution
Qualia identifies itself to Databricks on every request so your workspace admins can attribute usage in `system.access.audit`.
The Databricks Partner Well Architected Framework requires attribution wherever Genie-generated content appears, and in any product reference to the integration. Qualia shows a **Powered by Genie** label on every surface that references the integration.
Each Genie answer displayed in chat also:
* Renders Genie's content blocks unmodified rather than summarized by the model.
* Keeps every URI Genie returns a working link.
* Cites the Genie Space that produced it.