Not quite sure how to get started with Parabola - I have two main source csv files which get generated daily and uploaded to an AWS bucket. Each file can have. thousands of rows.
I want to build a flow that will grab the new files each day, check if anything has changed in either of them vs the previous day’s files, and if so then grab the newest file and run a moderately complicated series of transformations on the date.
I’ve already figured out how to tackle the transformations, but I’m not that sure how to manage automated daily file upload and check for changes.
Would appreciate a bit of help getting pointed in the right direction!
Hey Richard - Here’s a post asking a similar question.
Managing automated daily file uploads - Unfortunately, this isn’t possible on Parabola at this time. Can you post this as a feature request here?
What service is that CSV file from? Do they have an API that Parabola can connect to?
Checking for changes - This is possible to do in Parabola. Essentially, we would have an Import pulling the most recent data and then another Import pulling the data from yesterday. We would most likely use the List Contains step to check for changes and only pass through new/updated rows from yesterday’s data to be transformed by the flow.