zen - Hi team, I am fetching the technical meta...
Hi team, I am fetching the technical metadata from all data objects of thoughtspot and creating lineage in informatica IDMC using a custom resource creation tool in the Data governance catalog. To create a lineage, I would need mappings and transformations details about all objects. And till now I have identified below APIs would provide me with required information-
1. metadata/details
2. metadata/list
3. All dependency APIs
Can you help me with exactly which APIs out of these would give me required details (mappings and transformations) about all data objects (data source, tables, views, models, answers, liveboards) ?
4 Replies
Could you please elaborate on what exactly you mean by mapping and transformation details?
There is no single API which can help you here in your case. You will have to take help from a combination of the APIs you listed.
I want to fetch the details about how columns are mapped to one another in different data objects (I'm guessing dependency APIs would help for this)and also how metadata is being transformed from one data object to another (similar to ETL process) . And yes I know it will take combinations of these but I would like to know specifically about which APIs would provide me with these details. Any other APIs than those which I've listed?
1. Dependency APIs (https://developers.thoughtspot.com/docs/dependent-objects-api)
GET /tspublic/v1/dependency/logicalcolumn: This API returns a list of objects that depend on a specific logical column, helping you trace how a column is referenced or used in other objects.
GET /tspublic/v1/dependency/logicaltable: This shows which objects depend on a given table, helping you map relationships at the table level.
GET /tspublic/v1/dependency/logicalrelationship: This is useful for understanding how columns are joined between tables, which is a key part of mapping and transformation logic.
GET /tspublic/v1/dependency/physicalcolumn and GET /tspublic/v1/dependency/physicaltable: These endpoints help you trace dependencies at the physical data layer, which is important for understanding the lineage from source to logical objects.
GET /tspublic/v1/dependency/pinboard: if you need to see how visualizations or dashboards depend on underlying data objects.
2. Metadata APIs (https://developers.thoughtspot.com/docs/metadata-api)
/tspublic/v1/metadata/details and /tspublic/v1/metadata/listobjectheaders. These APIs let you fetch detailed metadata about tables, columns, worksheets, views, and their relationships. You can use these to get GUIDs and further details required for dependency queries.
• Subtype filtering:
For columns, you can filter by subtypes such as FORMULA to specifically find columns that are derived or transformed via formulas, which is directly related to ETL-like transformations.
3. Security APIs (https://developers.thoughtspot.com/docs/security-api):
While not directly about mapping or transformation, the security APIs can show you permission inheritance and sharing, which may be relevant if you need to understand access control as part of your data flow
Dependent objects APIs
The dependency API endpoints allow you to get details of the dependents for a metadata object
Metadata APIs
Metadata API
you can also try out the following:
1. metadata search api - (/api/rest/2.0/metadata/search) with include_dependent_objects = true in payload
2. search connection api - (/api/rest/2.0/connection/search) with include_details = true and data_warehouse_object_type = COLUMN
https://developers.thoughtspot.com/docs/rest-apiv2-metadata-search
https://developers.thoughtspot.com/docs/rest-apiv2-reference#_connections
Search metadata
Many use cases are possible with the very V2.0 metadata/search endpoint
REST API v2.0 Reference
REST API Reference