Everyone knows that GitHub is awesome and so is their API. However, there are limits to the amount of unauthenticated request you can make to the API from a single IP address. I am using their API for automatic updates for all of my GitHub hosted plugins with the Git Update plugin and it hit the limits really quick.
I decided to create a reverse proxy with generous level of caching to enable almost unlimited API requests where all request to:
http://github.kaspars.net/kasparsd
(where kasparsd is my GitHub username) are being proxied to
https://api.github.com/repos/kasparsd
using the following Nginx config rules:
proxy_cache_path /var/www/cache-github levels=1:2 keys_zone=github:10m;
server {
server_name github.kaspars.net;
location ~ ^/kasparsd/(.*) {
resolver 8.8.8.8;
proxy_pass https://api.github.com/repos/kasparsd/$1;
proxy_cache github;
proxy_cache_valid 200 302 1h;
proxy_ignore_headers Expires Cache-Control Set-Cookie X-Accel-Redirect X-Accel-Expires;
proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504;
#proxy_set_header User-Agent YOURGITHUBUSERNAME;
#proxy_set_header Authorization "Basic ...";
}
}Where resolver is the DNS server for resolving proxy_pass resource and all 200 and 302 responses are cached for an hour while forcing Nginx to ignore Expires and Cache-Control response headers to avoid skipping cache. GitHub also suggests sending optional User-Agent and Authorization headers.
There’s a free service which does this now too :)
http://rawgit.com/
Nice! Looks like there is demand for this kind of stuff. Great to have both self-hosted and hosted options available.
Hi Kaspars,
I’ve tried to set this up and it’s currently not working for me. The rate limit still decreases.
I do get a HTTP/1.1 304 Not Modified when using the etag but even then the rate limit still decreases.
Is this still a valid solution to cache api.github.com?
I can post up my nginx config without the api token etc. If that would help?
Here is a working example on my instance. Please share a link to your config and I’ll have a look.
The config you linked me to is in a single line, I’m not sure I understand what that is?
My config is:
proxy_cache_path /var/lib/nginx/cache-github levels=1:2 keys_zone=github:1024m; server { server_name api.github-proxy.corp.mydomain.com; location / { resolver 8.8.8.8; proxy_pass https://api.github.com/; proxy_set_header Authorization "token MyGeneratedToken"; proxy_cache github; proxy_cache_valid 200 302 1h; proxy_ignore_headers Expires Cache-Control Set-Cookie X-Accel-Redirect X-Accel-Expires; proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504; } }Nimish, could you check if Nginx has the permissions to write to the cache folder
/var/lib/nginx/cache-github? Secondly, you probably need to send theUser-Agentheader with your username.Here are the permissions on the directories
/var/lib/nginx
drwxrwxrwx. 4 nginx nginx 35 Jul 15 15:28 nginx
/var/lib/nginx/cache-github
drwxrwxrwx. 2 nginx nginx 6 Jul 15 15:28 cache-github
I’ve set the following as well but still get the X-RateLimit-Remaining decreasing on each request.
It decreases even if I use the ETag header in my curl request and get HTTP/1.1 304 Not Modified reply.
-H ‘If-None-Match: “ETag_Number”‘
proxy_set_header User-Agent GitHubUserName;
proxy_set_header Authorization “token …”;
This gives me the correct info via curl and my web browser but it doesn’t seem to be caching anything and the X-RateLimit-Remaining counter decreases with every subsequent request.