I am trying to extract the URL of specific articles from NYT API.
This is my code:
for i in range(0,100):
r = requests.get("http://api.nytimes.com/svc/search/v2/articlesearch.json?begin_date=20100101&q=terrorist+attack&page="+page+"&api-key=***")
data = r.json()
article = data['response']['docs']
for url in article:
After printing the first 20 URL it gives me this error
however by checking random pages the key 'response' is present in any of them. What can I do to print all the URLs from the next 88 pages?
You are assuming that there are at least 101 pages to make requests (0 to 100).
If you make a request to page 100, do you still get the same JSON structure with a
What you should instead use is a while loop that breaks when you get a
Yo I ran into a similar problem. You might be requesting faster than the allowed limit of 5 per second. In that case, the NYT server is going to hit you with an error message, so there will be no 'response' key. I would suggest printing out the keys from every GET request using something like:
If you keep seeing 'message' as one of your keys, then you know that you're probably requesting too fast. So just put in a time.sleep(0.5) to slow things down and you should be good.