Skip to content

CrateDB: Add database destination adapter #237

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

amotl
Copy link
Contributor

@amotl amotl commented Jun 9, 2025

About

The patch unlocks CrateDB as a database destination. Thanks!

Preview

Using this command, you can run the corresponding example without much ado.

alias ingestr='uvx --with="ingestr @ git+https://github.com/crate-workbench/ingestr@cratedb-destination" ingestr'

Details

CrateDB's destination adapter is based on the dlt postgres adapter, so it is also using the psycopg2 package.

References

dlt==1.10.0
dlt @ git+https://github.com/crate-workbench/dlt@cratedb
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will need to see the patch landing and a subsequent release of the dlt package.

A corresponding notice on the dlt repository does not raise many expectations, it looks like CrateDB might be too late to the party.

📣 New destinations are unlikely to be merged due to high maintenance cost.

@amotl amotl force-pushed the cratedb-destination branch from f86cf43 to 085f6a0 Compare June 9, 2025 14:09
@amotl amotl force-pushed the cratedb-destination branch from 085f6a0 to 15cd568 Compare June 9, 2025 15:09
Comment on lines +61 to +113
> [!WARNING]
> CrateDB supports the `replace` incremental materialization strategy, but
> currently does not support the `delete+insert`, `merge`, or `scd2` strategies.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hlcianfagna: I've blatantly copied this from the Clickhouse documentation, without knowing too much about corresponding details. As I think you have expertise in this area, could you please validate this statement in the context of ingestr and CrateDB, possibly including some hands-on validations? 🙏

At least I can tell from my basic orientation flights that I have been able to use Incremental Strategy: replace successfully.

image

This is probably not a big surprise, as it doesn't need any database support at all?

Copy link
Contributor Author

@amotl amotl Jun 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because this patch does not include much code at all, the topic is possibly a concern of the underlying dlt adapter.

However, we can also discuss relevant matters here, at your disposal, and also to create less noise on the dlt project.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume the ingestr incremental loading strategy is the same parameter that gets mapped to the dlt write disposition? In this case, the documentation about PostgreSQL's destination adapter says:

All write dispositions are supported.

So, after possibly validating all of them with CrateDB, we may want to adjust the documentation here.

Comment on lines +178 to +180
[tool.hatch.metadata]
allow-direct-references = true
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's just temporary in order to support installing from a Git branch.

@amotl amotl force-pushed the cratedb-destination branch 5 times, most recently from 71eff57 to e8abfcd Compare June 10, 2025 14:22
@amotl amotl force-pushed the cratedb-destination branch from e8abfcd to e2b9ecf Compare June 21, 2025 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant