Symphony.

I am currently working on a personal site using symphony 2 and I am running into some odd behavior involving pulling the Twitter RSS feed in as a dynamic XML data-source.

As mentioned in the post title I am hitting the 150 requests per hour API limit, but this is the limit for a particular IP address, my server, not my user based request limit, I can still pull feeds from my desktop, update from wherever, post it to Facebook etc.

What I find odd about this is that I have the data-source set to refresh only every 20 minutes, so even if that’s 3 API calls per request that should be only 9 an hour not 150. If it makes any difference I have been looking over and debugging the site at times, but not to the extent that would cause that many API calls. For example I was greeted with the ‘Rate Limit Exceeded’ in the feed and I haven’t debugged the site for hours.

Anyone else run into anything like this?

Does your website have a unique IP address? Perhaps there are multiple (can be hundreds) websites hosted on your IP.

I have the feeling that the Dynamic XML DS caching does not really work. There was a bugreport flying around somewhere… Don’t know what happened with it.

As far as I remember, Dynamic XML is not cached in Symphony, because you can’t rely on external server headers. So it’s hard to decide when to reload the XML.

What about importing the feed? Nick wrote a great tutorial!

So it’s hard to decide when to reload the XML.

When it says “30 Minutes” reload it after 30 Minutes have passed. Can’t be that hard…

Anyone else run into anything like this?

I have had the same problem even with the article that michael-e links to above. I put it down to faults with the Twitter API.

As phoque confirms, your interpretation of Dynamic XML DS caching is correct. If you have a single DS pointing to Twitter with a timeout of 20 minutes then this should only actually fetch the feed a maximum of 3 times per hour.

This could well be attributed to the bug phoque mentions above whereby a Dynamic XML DS isn’t properly cached and always fetches the feed regardless.

When it says “30 Minutes” reload it after 30 Minutes have passed. Can’t be that hard…

You are right, of course. I only babbled what I thought to remember from a very old discussion, some months ago. I am rather sure that we were discussing this issue…

Well, it’s a bug anyway, isn’t it?.

@phogue, would you please post this on the bugtracker?

I went through Twitter’s API docs and didn’t find anything definitive in regards to whether or not they check IPs or domains for API calls. I am on a shared host so this might be an issue. For whatever reason I can’t open a new ticket right now for help, so I posted a question on their GetSatisfaction issue board, just to cover all bases.

Bugreport filed: Issue 150.

Okay, we’ve found the issue.

It was the left out URL scheme (the http://-thingy) in cURL calls. It slipped through because only a few cURL versions don’t tolerate that. It will be fixed in the next release. Until then change Line 55 of class.gateway.php to

$this->_host = $url_parsed['scheme'] . "://" . $url_parsed['host'];

Now that I think about it again… It has kept Symphony from loading anything at all, not just “not cache it”.

Maybe the Dynamic XML DS could be expanded to keep a log of the last x refreshes (of course the log will be propagated to the XML as well).

For what use case would this be of benefit?

Due to its nature the Dynamic XML DS is very prone to errors. The log could be helpful to diagnose possible issues (rate limits, broken cache, estimate next update etc.).

I consistently run in to rate limiting errors using either a Dynamic XML DS or the Remote DS extension for my Twitter timeline, even if I bump up the cache time to e.g. 600 minutes. Did anything more come of this?