So I have an API with Cursors to page through data (100 items at a time) which I can automate through Parabola.
The issue is that for each request there could be dozens of pages of older data that I don’t really want. While Parabola does let me limit the number of pages pulled from an API request, its a fixed number.
Would it be possible to specify a condition in which the number of pages pulled varies?
Hey @Fernando_Campos1,
Setting the Max pages to fetch as a fixed number is the only way to change the number of pages to be returned.
Does your API have date parameters that you can pass into the URL? If so, you can add dynamic date values by using the Insert date & time step and passing that column through an Enrich with an API endpoint.
You will still have to adjust your pagination settings to ensure your response includes all results within that timeframe.
Feel free to add dynamic pagination settings as a #feature-requests so other people can chime in.
Thank you Daniel,
Indeed the issue is that this specific Endpoint does not accept date parameters (weirdly all other endpoints for this company do). So that’s I was trying to figure out a way to stop fetching when a certain datestamp was reached.
That makes sense. It is unusual that this specific endpoint doesn’t accept date parameters.
Have you tried making a request using date parameters that are formatted similarly to the other endpoints? Curious to know if that works.
Otherwise, the best option is to increase the maximum pages to fetch to ensure the timeframe you’re looking for is returned. You can then filter out rows that are outside of the timeframe.