Repository navigation
Add mssql-python as an optional DBAPI driver for MsSqlHook - #74539
RehanAhmad25 wants to merge 5 commits into
Conversation
bce8d0d to
dc2a3b0
Compare
| "host": "server.com", | ||
| "port": 1433, | ||
| "schema": "database_name", | ||
| "extra": {"dbapi_driver": "mssql_python", "TrustServerCertificate": "yes"} |
There was a problem hiding this comment.
Can you look at other db providers and see if there is a convention for this?
There was a problem hiding this comment.
Thanks for pointing this out. I went through the other DB providers to see what they do.
For picking a client library through an extra, MySQL is the only precedent, and it uses client (mysqlclient by default, or mysql-connector-python). JDBC uses driver_path and driver_class, and ODBC's driver is the ODBC driver name, so I went with dbapi_driver to avoid mixing it up with that. If you'd rather have client for consistency with MySQL, I'm happy to rename it.
On the docs, the other providers lead with a URI example and show the JSON form with extras nested under extra. I'll add a URI example next to the JSON one.
While checking, I also saw that the other hooks raise AirflowOptionalProviderFeatureException when an optional package is missing, so I'll switch the mssql-python import error to that.
There was a problem hiding this comment.
Added a URI example next to the JSON one, matching the other providers. I kept dbapi_driver as the key name, but I'm happy to rename it to client to match MySQL if you prefer.
dabla
left a comment
There was a problem hiding this comment.
Addressed
- @SameerMesiah97 docs Notes block: replaced with the suggested text (resolved, visible in
mssql.rst). - @SameerMesiah97
dbapi_drivervalidation: suggested form adopted (hooks/mssql.pyL87-100). - @SameerMesiah97 invalid driver cases: parametrized over
"",False,0, list and dict. - @SameerMesiah97
autocommit=False: test parametrized over both values. - @ashb
AirflowOptionalProviderFeatureException: used in_get_mssql_python_conn, test updated (thread still open on GitHub; the code matches the ask).
Open
- @ashb on the docs (convention across DB providers): URI example added, but the question of
dbapi_driverversus MySQL'sclientis still waiting for ashb's answer. - Inline [1] (URI keeps the hook's extras, which breaks the SQLAlchemy route with mssql-python), [2] (non-row handler results get converted), [3] (driver branching, nit).
- Out of scope / not flagged: the pymssql path still passes
placeholderand the otherDbApiHookextras topymssql.connect, as it did before this PR.
Pass: head d49fcae457, diff-based review with full reads of the hook, its dialect, DbApiHook (run, placeholder, autocommit, inspector) and the installed sources of mssql-python 1.15.0 and SQLAlchemy 2.1.5 (mssqlpython dialect). Both findings come from reading those primitives; no test run against a real SQL Server. CI is green.
d49fcae to
e40aec6
Compare
|
Pushed one small follow up: CI's MyPy providers check failed on a method-assign error in the test helper. I fixed it with the same inline type ignore other provider tests use. Nothing else changed. Could a committer approve the workflow runs when you get a chance? Thanks! |
|
@ashb Quick bump on this one: CI is green, @dabla 's comments are addressed, and the key name is the only open question on my side. It's just |
|
Quickest fix: git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-leaseAutomated nudge — ignore if you're not ready to rebase. This comment is updated in place on future |
…ssing and add URI docs example
094100e to
2fc17bc
Compare
|
Rebased onto latest main to clear the conflict in mssql.py (an import line next to the new cached_property change). uv.lock merged cleanly and still checks out. No other changes. The mssql tests pass in Breeze. Could a committer approve the workflow runs when you get a chance? Thanks! |
|
Hi @eladkal @kaxil, could you please take a look and approve the workflow run whenever you have time? The review has gone through a few rounds and is ready on my end:
Thanks! |
| return engine.connect(**(connect_kwargs or {})) | ||
|
|
||
| def get_conn(self) -> PymssqlConnection: | ||
| def get_conn(self) -> Any: |
There was a problem hiding this comment.
Switching a connection to mssql_python also switches MSSQLToGCSOperator and MsSqlToHiveOperator, which call get_conn() and map cursor.description[i][1] through pymssql's integer type codes. mssql-python puts Python types there (int, decimal.Decimal, datetime.datetime), so the Hive operator creates a table with every column as STRING, and a parquet export from the GCS operator fails on the first numeric column. Can both operators handle the mssql-python types here, or should the docs say they need pymssql for now?
| r[0] = self.sqlalchemy_scheme | ||
| # remove query string 'sqlalchemy_scheme' like parameters: | ||
| # remove query string parameters that are meant for the hook, not for the sqlalchemy dialect: | ||
| hook_only_extras = ( |
There was a problem hiding this comment.
With {"TrustServerCertificate": true} in the extras, Connection.get_uri() can't round-trip the bool, so it packs everything into one __extra__=<json> param and this filter never sees dbapi_driver or anything else. The mssqlpython dialect copies that param into the connection string and mssql-python rejects __extra__ as an unknown keyword, while get_conn() takes the same extras without complaint. Leftover pymssql keys like timeout hit the same rejection here, where connect() only warns about them. Could the mssql-python branch build sqlalchemy_url with URL.create(...) from stringified extras, dropping what connect() would ignore?
| * ``dbapi_driver`` accepts ``pymssql`` (the default) or ``mssql_python`` and is case-insensitive. | ||
| * Additional driver options, such as ``Encrypt``, ``TrustServerCertificate`` and | ||
| ``Authentication``, can be supplied through connection extras. | ||
| * The default SQLAlchemy scheme is ``mssql+mssqlpython``, which requires SQLAlchemy 2.1 |
There was a problem hiding this comment.
constraints-main pins SQLAlchemy 2.0.57 (3.2.0 pins 2.0.48), and Airflow 2.11, which this provider still supports, caps it below 2.0. So on a constrained install the default scheme makes get_sqlalchemy_engine() fail with NoSuchModuleError. Can this note say that get_conn() works there but the SQLAlchemy paths need an upgrade past the constraints?
| Choosing the DBAPI driver | ||
| ------------------------- | ||
|
|
||
| By default the hook connects with `pymssql <https://pypi.org/project/pymssql/>`__. To use Microsoft's |
There was a problem hiding this comment.
This section explains how to switch but not why someone would. Could it open with what mssql-python gives you over pymssql (Entra ID auth, Microsoft's bundled driver instead of FreeTDS) and what changes when you switch, like %s SQL needing a rewrite?
|
|
||
| def _get_pymssql_conn(self) -> PymssqlConnection: | ||
| """Return ``pymssql`` connection object.""" | ||
| import pymssql |
There was a problem hiding this comment.
pymssql is still a required dependency, so I don't think this needs to move into the method. Can it stay at module level?
| assert hook.get_connection().extra | ||
|
|
||
|
|
||
| def _mssql_hook(extra: str = "{}") -> MsSqlHook: |
There was a problem hiding this comment.
MsSqlHook(connection=conn) works here (DbApiHook pops connection from kwargs), which drops the unspecced Mock and the type: ignore. Could the helper also take port, so one test covers a connection without one? Nothing pins the if server and conn.port guard today, and without it the driver gets Server=localhost,None.
kaxil
left a comment
There was a problem hiding this comment.
Approving but please do address the comments
closes: #74432
What
MsSqlHookcan now connect with Microsoft'smssql-pythondriver instead ofpymssql. The driver is chosen with a newdbapi_driverconnection extra (pymssqlormssql_python, case-insensitive).pymssqlstays the default, so existing connections behave exactly as before. An unsupported value raises aValueError.How it works
With
dbapi_driverset tomssql_python:get_conn()builds the connection withmssql_python.connect(...). The host and port are sent asServer=host,port, and the schema, login and password map toDatabase,UIDandPWD.Encrypt,TrustServerCertificate,Authentication) are passed to the driver as keyword arguments. The hook's own extras (dbapi_driver,sqlalchemy_scheme,placeholderand the otherDbApiHookformat extras) are not passed on, and empty values are dropped.mssql+mssqlpython.sqlalchemy_schemestill overrides it.get_uri()stripsdbapi_driverfrom the query string.?, because mssql-python does not accept a plain%s. Theplaceholderextra still overrides it.Rowobjects.autocommitproperty.mssql-pythonextra.pymssqlis now imported inside the pymssql code path, not at module level. It is still a required dependency of the provider. Making it optional would be a separate change.Where
providers/microsoft/mssql/src/airflow/providers/microsoft/mssql/hooks/mssql.py: the hook changesproviders/microsoft/mssql/tests/unit/microsoft/mssql/hooks/test_mssql.py: new testsproviders/microsoft/mssql/docs/connections/mssql.rst: new "Choosing the DBAPI driver" sectionproviders/microsoft/mssql/pyproject.toml: new optional extramssql-python(mssql-python>=1.15.0)providers/microsoft/mssql/docs/index.rst: extras table, updated by the repo's hookuv.lock: regenerated withuv 0.13.0for the new extra (addsmssql-pythonandmssql-python-odbc)Notes for reviewers
>=1.15.0as the floor because it is the version I checked the driver's behaviour against. Happy to loosen it if you prefer.Testing
I added 14 new test cases for driver selection and validation, scheme and placeholder defaults,
get_uri, the arguments passed toconnect, the missing-package error, autocommit and row conversion. The existing pymssql tests pass unchanged (35 passed in Breeze). The prek hooks, including ruff, mypy for providers and the provider.yaml check, pass locally.Both drivers are mocked in the tests, so I have not run this against a real SQL Server yet. The Microsoft Entra ID
Authenticationoptions in particular are untested, so a check on a real instance would be welcome.Related: #72000
Was generative AI tooling used to co-author this PR?
Generated-by: Claude (Anthropic) following the guidelines