Data and Report APIs
Hey all, have some questions around requests to the Data API endpoints. We are trying to use runtime filters the user saves using a custom action hit the endpoint and write rows of data (just an ID field from the filtered data) to our data lake. We are having no problems with smaller sets, but for larger sets we end up getting a 500 error on the third or fourth request.
1. Would you recommend using the Search Data endpoint or the Fetch Liveboard Data endpoint? We are currently using Fetch Liveboard Data on a fairly large table contained on a Liveboard.
2. Any pagination best practices that could help allieviate the issue?
3. If the above aren't applicable, we would like to increase our max to 100K as mentioned here: https://developers.thoughtspot.com/docs/fetch-data-and-report-apis#_pagination_settings_for_data_and_report_apis.
Data and Report APIs
Data and Report APIs
3 Replies
@gabe_f A couple questions:
1. Are you using batching or just trying to get all of the records in one go?
2. Are you seeing errors details from the 500 error?
Specific answers: 1. I personally like the SearchData API, but it shouldn't matter in this scenrio. I like to say that SearchData is like doing SQL queries and Liveboard/AnswerData endpoints are like calling a stored proc.
2. I would use pagination for large data sets. You may have to experiment with the size. For the cstools code, as an example, we default to pagination to avoid timeouts. 3. I don't think increasing the max would matter beyond the size of data you get. But knowing the error would help a lot.
Specific answers: 1. I personally like the SearchData API, but it shouldn't matter in this scenrio. I like to say that SearchData is like doing SQL queries and Liveboard/AnswerData endpoints are like calling a stored proc.
2. I would use pagination for large data sets. You may have to experiment with the size. For the cstools code, as an example, we default to pagination to avoid timeouts. 3. I don't think increasing the max would matter beyond the size of data you get. But knowing the error would help a lot.
1. Ideally one go
2. It was an error relating to pagination. I was essentially unable to grab anything beyond the first 1000 rows on the liveboard. I am able to access the correct number of rows using SearchData API so I think that solves the majority of the issues.
Great to hear!