akan mengemas kini bahagian ini secara automatik.
Subseksyen [tool.poetry.dev-dependencies] ialah tempat anda mengisytiharkan kebergantungan pembangunan sahaja, seperti rangka kerja ujian, linters, dsb.
Bahagian [bina-sistem] digunakan untuk menyimpan data berkaitan binaan. Dalam kes ini, ia menentukan binaan belakang sebagai "poetry.core.masonry.api". Dalam erti kata yang sempit, tanggungjawab teras a
build-backend ialah membina roda dan sdist.
Repositori juga termasuk fail poetry.lock, komponen khusus Puisi yang dijana dengan menjalankan pemasangan puisi atau kemas kini puisi. Fail kunci ini menentukan versi tepat semua kebergantungan dan sub-bergantungan untuk projek anda, memastikan pemasangan boleh dihasilkan semula merentas persekitaran yang berbeza.
Adalah penting untuk mengelakkan pengeditan manual pada fail poetry.lock, kerana ini boleh menyebabkan ketidakkonsistenan dan isu pemasangan. Sebaliknya, buat perubahan pada fail pyproject.toml anda dan benarkan Puisi mengemas kini fail kunci secara automatik dengan menjalankan kunci puisi.
Mendapat Puisi
Per amaran pemasangan Puisi
::: {.amaran}
Puisi hendaklah sentiasa dipasang dalam persekitaran maya khusus untuk mengasingkannya daripada seluruh sistem anda. Ia tidak sepatutnya dipasang dalam persekitaran projek yang akan diuruskan oleh Puisi.
:::
Di sini kami akan menganggap anda mempunyai akses kepada Puisi dengan menjalankan pipx install poetry
Membangunkan dalam persekitaran maya
Dengan struktur fail kami dijelaskan, mari kita mulakan proses pembangunan dengan menyediakan persekitaran kita. Memandangkan projek kami sudah termasuk fail pyproject.toml dan poetry.lock, kami boleh memulakan persekitaran kami menggunakan arahan shell puisi.
Arahan ini mengaktifkan persekitaran maya yang dipautkan kepada projek Puisi semasa, memastikan semua operasi seterusnya berlaku dalam konteks pergantungan projek. Jika tiada persekitaran maya wujud, cangkerang puisi mencipta dan mengaktifkannya secara automatik.
kulit puisi mengesan cangkang semasa anda dan melancarkan contoh baharu dalam persekitaran maya. Memandangkan Puisi memusatkan persekitaran maya secara lalai, arahan ini menghapuskan keperluan untuk mencari atau mengingat semula laluan khusus ke skrip aktifkan.
Untuk mengesahkan persekitaran Python yang sedang digunakan dengan Puisi, anda boleh menggunakan arahan berikut:
poetry env list --full-path
Salin selepas log masuk
Ini akan menunjukkan semua persekitaran maya yang dikaitkan dengan projek anda dan menunjukkan yang mana yang sedang aktif.
Sebagai alternatif, anda boleh mendapatkan laluan penuh hanya persekitaran semasa:
poetry env info -p
Salin selepas log masuk
Dengan persekitaran diaktifkan, gunakan pemasangan puisi untuk memasang kebergantungan yang diperlukan. Perintah berfungsi seperti berikut
- Jika fail poetry.lock ada, pemasangan puisi akan menggunakan versi tepat yang dinyatakan dalam fail itu dan bukannya menyelesaikan kebergantungan secara dinamik. Ini memastikan pemasangan yang konsisten dan berulang merentasi persekitaran yang berbeza.
i. Jika anda menjalankan pemasangan puisi dan ia nampaknya tidak berjalan lancar, anda mungkin perlu menjalankan eksport PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring dalam shell yang anda pasangkan
- Jika tidak, ia membaca fail pyproject.toml dalam projek semasa, menyelesaikan kebergantungan yang disenaraikan di sana dan memasangnya.
- Jika tiada fail poetry.lock wujud, pemasangan puisi akan mencipta satu selepas menyelesaikan kebergantungan, jika tidak, ia akan mengemas kini yang sedia ada.
Untuk melengkapkan persediaan persekitaran, kami perlu menambah perpustakaan gabungan data pada kebergantungan kami. Jalankan arahan berikut:
poetry add datafusion
Salin selepas log masuk
Arahan ini mengemas kini fail pyproject.toml anda dengan pakej datafusion dan memasangnya. Jika anda tidak menentukan versi, Puisi akan memilih versi yang sesuai secara automatik berdasarkan versi pakej yang tersedia.
Implementing the Interfaces
To create a Harlequin Adapter, you need to implement three interfaces defined as abstract classes in the harlequin.adapter module.
The first one is the HarlequinAdapter.
#| eval: false
#| code-fold: false
#| code-summary: implementation of HarlequinAdapter
class DataFusionAdapter(HarlequinAdapter):
def __init__(self, conn_str: Sequence[str], **options: Any) -> None:
self.conn_str = conn_str
self.options = options
def connect(self) -> DataFusionConnection:
conn = DataFusionConnection(self.conn_str, self.options)
return conn
Salin selepas log masuk
The second one is the HarlequinConnection, particularly the methods execute and get_catalog.
#| eval: false
#| code-fold: false
#| code-summary: implementation of execution of HarlequinConnection
def execute(self, query: str) -> HarlequinCursor | None:
try:
cur = self.conn.sql(query) # type: ignore
if str(cur.logical_plan()) == "EmptyRelation":
return None
except Exception as e:
raise HarlequinQueryError(
msg=str(e),
title="Harlequin encountered an error while executing your query.",
) from e
else:
if cur is not None:
return DataFusionCursor(cur)
else:
return None
Salin selepas log masuk
For brevity, we've omitted the implementation of the get_catalog function. You can find the full code in the adapter.py file within our GitHub repository.
Finally, a HarlequinCursor implementation must be provided as well:
#| eval: false
#| code-fold: false
#| code-summary: implementation of HarlequinCursor
class DataFusionCursor(HarlequinCursor):
def __init__(self, *args: Any, **kwargs: Any) -> None:
self.cur = args[0]
self._limit: int | None = None
def columns(self) -> list[tuple[str, str]]:
return [
(field.name, _mapping.get(field.type, "?")) for field in self.cur.schema()
]
def set_limit(self, limit: int) -> DataFusionCursor:
self._limit = limit
return self
def fetchall(self) -> AutoBackendType:
try:
if self._limit is None:
return self.cur.to_arrow_table()
else:
return self.cur.limit(self._limit).to_arrow_table()
except Exception as e:
raise HarlequinQueryError(
msg=str(e),
title="Harlequin encountered an error while executing your query.",
) from e
Salin selepas log masuk
Making the plugin discoverable
Your adapter must register an entry point in the harlequin.adapters group using the packaging software you use to build your project.
If you use Poetry, you can define the entry point in your pyproject.toml file:
[tool.poetry.plugins."harlequin.adapter"]
datafusion = "harlequin_datafusion:DataFusionAdapter"
Salin selepas log masuk
An entry point is a mechanism for code to advertise components it provides to be discovered and used by other code.
Notice that registering a plugin with Poetry is equivalent to the following pyproject.toml specification for entry points:
[project.entry-points."harlequin.adapter"]
datafusion = "harlequin_datafusion:DataFusionAdapter"
Salin selepas log masuk
Testing
The template provides a set of pre-configured tests, some of which are applicable to DataFusion while others may not be relevant. One test that's pretty cool checks if the plugin can be discovered, which is crucial for ensuring proper integration:
#| eval: false
#| code-fold: false
if sys.version_info < (3, 10):
from importlib_metadata import entry_points
else:
from importlib.metadata import entry_points
def test_plugin_discovery() -> None:
PLUGIN_NAME = "datafusion"
eps = entry_points(group="harlequin.adapter")
assert eps[PLUGIN_NAME]
adapter_cls = eps[PLUGIN_NAME].load()
assert issubclass(adapter_cls, HarlequinAdapter)
assert adapter_cls == DataFusionAdapter
Salin selepas log masuk
To make sure the tests are passing, run:
poetry run pytest
Salin selepas log masuk
The run command executes the given command inside the project’s virtualenv.
Building and Publishing to PyPI
With the tests passing, we're nearly ready to publish our project. Let's enhance our pyproject.toml file to make our package more discoverable and appealing on PyPI. We'll add key metadata including:
- A link to the GitHub repository
- A path to the README file
- A list of relevant classifiers
These additions will help potential users find and understand our package more easily.
classifiers = [
"Development Status :: 3 - Alpha",
"Intended Audience :: Developers",
"Topic :: Software Development :: User Interfaces",
"Topic :: Database :: Database Engines/Servers",
"License :: OSI Approved :: MIT License",
"Programming Language :: Python :: Implementation :: CPython"
]
readme = "README.md"
repository = "https://github.com/mesejo/datafusion-adapter"
Salin selepas log masuk
For reference:
- The complete list of classifiers is available on PyPI's website.
- For a detailed guide on writing pyproject.toml, check out this resource.
- The formal, technical specification for pyproject.toml can be found on packaging.python.org.
Building
We're now ready to build our library and verify its functionality by installing it in a clean virtual environment. Let's start with the build process:
poetry build
Salin selepas log masuk
This command will create distribution packages (both source and wheel) in the dist directory.
The wheel file should have a name like harlequin_datafusion-0.1.1-py3-none-any.whl. This follows the standard naming convention:
-
harlequin_datafusion is the package (or distribution) name
-
0.1.1 is the version number
-
py3 indicates it's compatible with Python 3
-
none compatible with any CPU architecture
-
any with no ABI (pure Python)
To test the installation, create a new directory called test_install. Then, set up a fresh virtual environment with the following command:
python -m venv .venv
Salin selepas log masuk
To activate the virtual environment on MacOS or Linux:
source .venv/bin/activate
Salin selepas log masuk
After running this command, you should see the name of your virtual environment (.venv) prepended to your command prompt, indicating that the virtual environment is now active.
To install the wheel file we just built, use pip as follows:
pip install /path/to/harlequin_datafusion-0.1.1-py3-none-any.whl
Salin selepas log masuk
Replace /path/to/harlequin_datafusion-0.1.1-py3-none-any.whl with the actual path to the wheel file you want to install.
If everything works fined, you should see some dependencies installed, and you should be able to do:
harlequin -a datafusion
Salin selepas log masuk
Congrats! You have built a Python library. Now it is time to share it with the world.
Publishing to PyPI
The best practice before publishing to PyPI is to actually publish to the Test Python Package Index (TestPyPI)
To publish a package to TestPyPI using Poetry, follow these steps:
Create an account at TestPyPI if you haven't already.
Generate an API token on your TestPyPI account page.
-
Register the TestPyPI repository with Poetry by running:
poetry config repositories.test-pypi https://test.pypi.org/legacy/
Salin selepas log masuk
-
To publish your package, run:
poetry publish -r testpypi --username __token__ --password <token>
Salin selepas log masuk
Replace with the actual token value you generated in step 2. To verify the publishing process, use the following command:
python -m pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple <package-name>
Salin selepas log masuk
This command uses two key arguments:
-
--index-url: Directs pip to find your package on TestPyPI.
-
--extra-index-url: Allows pip to fetch any dependencies from the main PyPI repository.
Replace with your specific package name (e.g., harlequin-datafusion if following this post). For additional details, consult the information provided in this post.
To publish to the actual Python Package Index (PyPI) instead:
Create an account at https://pypi.org/ if you haven't already.
Generate an API token on your PyPI account page.
-
Run:
poetry publish --username __token__ --password <token>
Salin selepas log masuk
The default repository is PyPI, so there's no need to specify it.
Is worth noting that Poetry only supports the Legacy Upload API when publishing your project.
Automated Publishing on GitHub release
Manually publishing each time is repetitive and error-prone, so to fix this problem, let us create a GitHub Action to
publish each time we create a release.
Here are the key steps to publish a Python package to PyPI using GitHub Actions and Poetry:
Set up PyPI authentication: You must provide your PyPI credentials (the API token) as GitHub secrets so the GitHub Actions workflow can access them. Name these secrets something like PYPI_TOKEN.
Create a GitHub Actions workflow file: In your project's .github/workflows directory, create a new file like publish.yml with the following content:
name: Build and publish python package
on:
release:
types: [ published ]
jobs:
publish-package:
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: '3.10'
- name: Install Poetry
uses: snok/install-poetry@v1
- run: poetry config pypi-token.pypi "${{ secrets.PYPI_TOKEN }}"
- name: Publish package
run: poetry publish --build --username __token__
Salin selepas log masuk
The key is to leverage GitHub Actions to automate the publishing process and use Poetry to manage your package's dependencies and metadata.
Conclusion
Poetry is a user-friendly Python package management tool that simplifies project setup and publication. Its intuitive command-line interface streamlines environment management and dependency installation. It supports plugin development, integrates with other tools, and emphasizes testing for robust code. With straightforward commands for building and publishing packages, Poetry makes it easier for developers to share their work with the Python community.
At LETSQL, we're committed to contributing to the developer community. We hope this blog post serves as a straightforward guide to developing and publishing Python packages, emphasizing best practices and providing valuable resources.
To subscribe to our newsletter, visit letsql.com.
Future Work
As we continue to refine the adapter, we would like to provide better autocompletion and direct reading from files (parquet, csv) as in the DataFusion-cli. This requires a tighter integration with the Rust library without going through the Python bindings.
Your thoughts and feedback are invaluable as we navigate this journey. Share your experiences, questions, or suggestions in the comments below or on our community forum. Let's redefine the boundaries of data science and machine learning integration.
Acknowledgements
Thanks to Dan Lovell and Hussain Sultan for the comments and the thorough review.
![[Web front-end] Permulaan pantas Node.js](https://img.php.cn/upload/course/000/000/067/662b5d34ba7c0227.png)
