API Pagination for Phorest

[Tim_Stadtlander]

How do I setup pagination correctly with the following in mind?

I’m trying to get pagination working for Phorest’s API (Phorest API).

I’m getting data back without pagination settings, but am getting a message “There was a calculation error in ‘Get Staffs from Phorest’” when I try to use paging.

I can get paging to work properly in Postman like so:
{{third-party-api}}/api/business/{{business}}/branch/{{branch}}/staff?page=0&size=2

Here’s example data:

{
    "_embedded": {
        "staffs": [
            {
                "staffId": "abc123",
                "staffCategoryId": "xyz789",
                "userId": "something",
                "staffCategoryName": "Doctor",
                "firstName": "Becca",
                "lastName": "Hiya",
                "selfEmployed": false,
                "mobile": "5553214567",
                "email": "anemail@example.com",
                "hideFromOnlineBookings": true,
                "hideFromAppointmentScreen": false,
                "disqualifiedServices": [
                    "no5s4UTbky6UmJQx-WnQkw",
                    "hd__0qiZO1CfGs2FWONIew",
                    "CyOCMbFe8o28ijcMncgLGw",
                ]
            },
            {
                "staffId": "123abc",
                "staffCategoryId": "xyz789",
                "userId": "somethingElse",
                "staffCategoryName": "Doctor",
                "firstName": "Billy",
                "lastName": "Bob",
                "selfEmployed": false,
                "mobile": "12345678",
                "email": "bill@example.com",
                "hideFromOnlineBookings": true,
                "hideFromAppointmentScreen": true,
                "disqualifiedServices": [
                    "jACUHMcczpl3aLvDl1x_Pw",
                    "tmgjzYQcdZAcYGO01AklpA"
                ]
            }
        ]
    },
    "page": {
        "size": 2,
        "totalElements": 43,
        "totalPages": 22,
        "number": 2
    }
}

This is my setting in Parabola. Not sure if this matters, but note that I pull the Branches first and then use Enrich with an API to get all staff from all branches.

[Adam_R]

Hey @Tim_Stadtlander

Appreciate the detailed explanation. Have seen this strange behavior/ vague error message in the past when pulling from Phorest – if this is the same issue, we display “There was a calculation error in ‘Get Staffs from Phorest’” when we try to pull blank pages (we are pulling blank pages because “Max pages to fetch” is larger than the amount of existing pages).

Couple of recommendations here:

1. In your Pagination settings, change Increment each page by to 1. If it is set to 2, that means we are pulling page 0, 2, 4, etc. and skipping over pages.

2. Phorest has a high rate limit – Max requests per minute can safely be raised to 500

3. Let’s try reducing Max pages to fetch to 1 to determine how many records are being pulled on a given page. Since you’re pulling staff members, I imagine you have a ballpark estimate for the number of expected results. You’ll want to play some guess-and-check here – for instance, if changing Max pages to fetch to 3 still throws the calculation error, try reducing the value to 2. We’re looking for the largest number that does not throw the error.

4. Exclude page and size parameters from the API Endpoint

Realize that this isn’t the cleanest/ most scalable approach – for whatever reason, Phorest is throwing us an error message which we are translating and showing as the calculation error.

That said, if this is similar to previous issues, the above guidance should do the trick. Let me know how it goes!

[Tim_Stadtlander]

Hi @Adam_R ,

Thanks for the quick and helpful response. The settings you mentioned fixed the issue if I’m calling a single branch with the specific number of pages. Is there a way that I can get the page number of staff members from each branch and then loop through that?

This is the data I get back that I’m using the Enrich step against. I would just make a request for all staff, but Phorest’s API requires that a branch be in the API endpoint.

Thanks,
Tim

[Adam_R]

Glad to hear you were able to resolve the first part here.

Couple modifications we can make to pull in all pages for these datasets:

1. For this new example, let’s actually add the size=100 parameter; that way, no row will have a totalPages value greater than 2
2. Leaving the data un-expanded, add 2 filter steps to create 2 branches: 1 where api.page.totalPages = 1 , and the other where totalPages = 2
3. Where pages = 1, use an “Expand JSON” step to unpack the data
4. Where pages = 2, let’s re-enrich the data while pulling both pages. I’d recommend adding a “Select columns” step before the duplicate “Enrich with an API” step to remove any columns returned by the previous enrichment step to avoid duplicate column names and any confusion.
5. For rows where page = 2, pass the data back through the exact same enrichment step, but this time add the following settings:

6. Expand the JSON and re-combine the data with a “Stack Tables” step

If you’re worries about additional records, you could replicate this approach to capture 3 and 4 pages as well.

Quite the workaround I know! Appreciate your understanding here. Let us know if you have any other questions.