JSON file URL import

Ask a question
#1

Hi there,

I’m using Phantom Buster to extract hashtagged tweets. It outputs to a JSON file that is available as a URL. What do I need to pull that data from the file? I can see it’s possible to get it from a JSON file local but not via a URL?

Is this possible or is there another way?

Cheers

#2

Hey Jason! You can put that URL into the API Import to get the data!

#3

Hey Brian, that’s what I thought but it failed. I ran it again and realized there was an error in the data. All working now. Thanks

2 Likes
#4

Wouldnt it be great if there was a feature that handled the authentication process for sites so that we could get json responses from XHR calls behind a login? I need invoices, product catalogs, pricing, etc.

#5

This is what I am trying to figure out. I know I can input the JSON URL and just use that but I would rather have parabola run phantombuster for me and get the json afterwards. Is there a way to do that?

I have figured out how to have parabola run the phantombuster query but I have not figured out how to export the JSON.

#6

Phantom buster appears to require a manual login before it can get the data, not very helpful if you want to automate the capture of info behind a login.

I have had some success, and some failure, using Linux bash with curl to login and capture the appropriate bearer token… Then make the calls directly for the json data and pass them through to Parabola using php. In essence, creating an API for the data that Parabola can use.

One problem with this approach is that it doesn’t support Javascript and therefore fails on one of my data sources. I can manually grab the token and semi automate the rest of the process, but… that’s not the point of automation.

As for your question, if all you need is an authentication cookie, thats easy to grab and with help you may be able to use Google App Engine to host a php script that does it for you. That way you can make the json requests directly instead of parsing through html.

2 Likes
#7

Hey Henry,

Do you know if PhantomBuster can export to a Google Sheet? I did some reseach on my end, and may have found something that worked.

Taking a look at this video, it seems like you can use a simple function on your Google Sheet that will auto-populate every time your Phantom launches and collects new data.

You can then import that data as a source step using the Pull from Google Sheets step. Let me know if that works.

#8

This looks like an interesting solution. Thanks. I will try that.

1 Like