pearcy
Explore posts from serversCCConvex Community
•Created by pearcy on 12/27/2024 in #support-community
Airbyte Convex Destination Connector - 404 Error
Thank you @Tom, that worked. That is a relief!
7 replies
CCConvex Community
•Created by pearcy on 12/27/2024 in #support-community
Airbyte Convex Destination Connector - 404 Error
If it makes a difference - I'm using the Convex starter plan and a self-hosted, open-source Airbyte instance, which is running locally via Docker. My React Native app works locally with the Convex development database. The goal is to upload JSON files via Airbyte, enabling independent deployments.
7 replies
CCConvex Community
•Created by pearcy on 12/27/2024 in #support-community
External Data Import to Convex with Dagster
10 replies
CCConvex Community
•Created by pearcy on 12/27/2024 in #support-community
External Data Import to Convex with Dagster
@ballingt ok, I just posted my 404 error. I will keep an eye out on this other post since it is specific to this error.
10 replies
CCConvex Community
•Created by pearcy on 12/27/2024 in #support-community
External Data Import to Convex with Dagster
Thanks Tom, I started with Airbyte but got errors so tried connecting with Dagster directly. This helps. I will go back and try get it to work with Airbyte. Feel free to close this out.
10 replies
CCConvex Community
•Created by pearcy on 12/27/2024 in #support-community
External Data Import to Convex with Dagster
Questions:
a. Is Airbyte a hard dependency for using the /api/streaming_import/import_airbyte_records endpoint, or can clients like Dagster directly connect and utilize this API?
b. It's peculiar that the article records are successfully upserted into the articles table using the same API endpoint, but the associated tools, articleTools, materials, and articleMaterials records fail with an index error. Is there a different behavior/configuration expected for related tables?
c. Is it necessary to send all records in a single request? Could the issue be related to the fact that I'm performing a staged import:
1. First import articles.
2. Then import tools with article references.
3. Finally import materials with article references.
Is there correct way to handle related data/foreign keys with streaming import?
I initially attempted a single-stage import but encountered errors, which led me to break the process into the current staged approach. And I can successfully upsert this same JSON data using my React Native/Convex application, but I am aiming for a more scalable, robust solution using the streaming import API within my data pipeline, orchestrated by Dagster.
Any suggestions would be greatly appreciated.
10 replies
CCConvex Community
•Created by pearcy on 12/27/2024 in #support-community
External Data Import to Convex with Dagster
4. Current status
The base article record is being created correctly using the /api/streaming_import/import_airbyte_records endpoint.
We have confirmed that the BadJsonBody errors are no longer an issue because we are sending the correct payload.
We are receiving IndexNotFoundError: Index tools._by_airbyte_primary_key not found. which indicates the streaming import endpoint is expecting a specific index _by_airbyte_primary_key.
10 replies