Hi Parabola team, first off, thank you for building such an amazing product. I have what may be a silly noob question:
I am trying to pull from the WordPress REST API (V2). The API gives the total number of pages as a header response using the Key X-WP-TotalPages. I am using the page and key method to paginate through but that obviously isn’t scalable as I’d have to increment the max pages field under rate limits (WordPress throws a 404 error if I give a random number higher than the X-WP-TotalPages).
I am not sure how to go about automating the process where the pagination goes to till the total pages count every day (I have the flow on a recurring schedule).
Good question! Unfortunately, there isn’t a way to retrieve response headers from the WordPress API and load them into your Parabola results. We’ve had many requests to support dynamic headers with our pagination, so it’s on our radar!
Instead of iterating through pages with one request, try making a CSV file with a column listing the URL to each page. Be sure to add a buffer to easily load more pages in the future.
Next, use the Enrich with an API step to make a new request for each row in your CSV file by passing the column name as the API Endpoint URL using curly brackets.
This method will make a new request for each row, eradicating the need to configure your pagination settings.
If you only have 7 pages (but enter 9 pages of URLs), the API will still error out. To fix this, head to the Error Handling section and configure it to stop the step by setting the row failure value to a high number.
The final dataset should include all 7 pages with 2 rows showing the error messages you received. Go ahead and filter those out and you should be good to go.
It’s not perfect but should help with scaling your requests in the interim. Let me know if this helps!
