Using after parameter with get user followers

API
#1

To be frank - i have never used pagination/after parameters before so this is a new one for me.

Code:

Ask for Username

echo "What’s the username: "
read name

Curl for token

curl -X POST ‘https://id.twitch.tv/oauth2/token?client_id=’$client_id’&client_secret=’$client_secret’&grant_type=client_credentials&scope=user:read:email’ >> token.txt && awk -F ‘"’ ‘{print $4}’ token.txt >> token_only.txt
token=$(cat token_only.txt)

Get User ID

curl -H 'Client-ID: '$client_id -H 'Authorization: Bearer '$token 'https://api.twitch.tv/helix/users?login=’$name >> user_id.txt && awk -F ‘"’ ‘{print $6}’ user_id.txt >> uid_only.txt
user_id=$(cat uid_only.txt)

Get follow list for user

curl -H 'Client-ID: '$client_id -H 'Authorization: Bearer '$token ‘https://api.twitch.tv/helix/users/follows?to_id=’$user_id’&first=100’ | json_pp >> fl_list.txt
cat fl_list.txt | grep ‘“cursor”’ | awk -F ‘:’ ‘{print $2}’ | sed ‘s/"//g’ >> cursor.txt
cursor=$(cat cursor.txt)
curl -H 'Client-ID: '$client_id -H 'Authorization: Bearer '$token 'https://api.twitch.tv/helix/users/follows?to_id=’$user_id’&after=’$cursor -v | json_pp >> fl_list.txt
cat fl_list.txt | grep ‘“from_id”’ | awk -F ‘:’ ‘{print $2}’ | sed ‘s/"//g’ | sed ‘s/,//g’ >> clean_fl_list.txt
wc clean_fl_list.txt

Remove Token for next go

rm token* uid* user* fl_list* clean_fl* cursor*

in the curl output - it doesnt seem like its passing the data after the “after” parameter.

GET /helix/users/follows?to_id=<user_id>&after= HTTP/2
Host: api.twitch.tv
user-agent: curl/7.74.0
accept: /

any ideas anyone?

#2

Wrap the whole URL in “” in case the & in the address is being misintrepreted by your interpreter

Also please can you wrap your code in code blocks either three ` at start/end or start each line with fourc spaces, or select all the code and hit the “preformatted text button”

It’ll help for clairty understanding your scripts

#3

so i was able to manage to get it working - but the curl output still says it cant resolve it - even though its getting all the users - well at least second iteration of querying.

#4

ok so i didnt - i just realize i was duplicating the results…
but this is what i have:
i wrapped the url in “” and still the same thing…

# Curl for token
curl -X POST 'https://id.twitch.tv/oauth2/token?client_id='$client_id'&client_secret='$client_secret'&grant_type=client_credentials&scope=user:read:email' >> token.txt && awk -F '"' '{print $4}' token.txt >> token_only.txt
token=$(cat token_only.txt)

# Get User ID
curl -H 'Client-ID: '$client_id -H 'Authorization: Bearer '$token 'https://api.twitch.tv/helix/users?login='$name >> user_id.txt && awk -F '"' '{print $6}' user_id.txt >> uid_only.txt
user_id=$(cat uid_only.txt)

# Get follow list for user
curl -H 'Client-ID: '$client_id -H 'Authorization: Bearer '$token 'https://api.twitch.tv/helix/users/follows?to_id='$user_id'&first=100' | json_pp >> fl_list.txt
cat fl_list.txt | grep '"cursor"' | awk -F ':' '{print $2}' | sed 's/"//g' >> cursor.txt
cursor=$(cat cursor.txt)
curl -H 'Client-ID: '$client_id -H 'Authorization: Bearer '$token "https://api.twitch.tv/helix/users/follows?to_id="$user_id"&after="$cursor | json_pp >> fl_list.txt
cat fl_list.txt | grep '"from_id"' | awk -F ':' '{print $2}' | sed 's/"//g' | sed 's/,//g' >> clean_fl_list.txt
wc clean_fl_list.txt

# See if follower is live
for followers in `cat clean_fl_list.txt`
do
    curl -H 'Client-ID: '$client_id -H 'Authorization: Bearer '$token 'https://api.twitch.tv/helix/streams?user_id='$followers | json_pp >> live_list.txt
done

cat live_list.txt | grep "user_name" && cat live_list.txt | grep "user_id" && cat live_list.txt | grep "type"

# Remove Token for next go
rm token* uid* user* fl_list* clean_fl* cursor* live_list*
#5

Make sure you keep using first=100 when paginating, otherwise you’ll get 100 results on the first page, then each subsequent page will have the default number of results per page.

#6

ill add it back it - but cURL isnt recongnizing the parameter data and just duplicates the data

#7

If you’re getting the same data, you’re making the same request.

If you want to debug the best thing to do is log the raw request you’re actually making so you can see what is actually happening and ensure the values being passed to the the request are the ones you intend.

#8

the only bit of data that isnt being pass is the pagination cursor data
below is the verbose output of curl with the combine error below that

GET /helix/users/follows?to_id=506991194&first=100&after= HTTP/2
Host: api.twitch.tv
user-agent: curl/7.74.0
accept: /
client-id:
curl: (6) Could not resolve host: <cursor_info>

#9

Your problem seems to be one of bash string concatenation

The frist request ran and the second request made a call to <cursor_info> and of course <cursor_info> is not a valid URL so the host look up failed.

curl -H "Client-ID: $client_id" -H "Authorization: Bearer $token" "https://api.twitch.tv/helix/users/follows?to_id=$user_id&after=$cursor" | json_pp >> fl_list.txt

may work, I’ve not tested but matches my quick remind google

I did a quick google to remind myself of bash and referred to https://stackoverflow.com/questions/50613030/scripted-concatenation-of-strings-in-curl-command-payload

#10

So i found out the issue was with the grep, sed, and awk converstions.
there was a space in front of the <cursor_info> that was breaking the curl.

jsut need to figure out how to loop it till no more pagination populates.
any ideas? lol

#11

Normally I could the records to see if it matches my first
And/or if the page is empty
And/or no further cursor is presented