Use Nginx Reverse Proxy to Mirror GitHub API

Everyone knows that GitHub is awesome and so is their API. However, there are limits to the amount of unauthenticated request you can make to the API from a single IP address. I am using their API for automatic updates for all of my GitHub hosted plugins with the Git Update plugin and it hit the limits really quick.

I decided to create a reverse proxy with generous level of caching to enable almost unlimited API requests where all request to:

http://github.kaspars.net/kasparsd

(where kasparsd is my GitHub username) are being proxied to

https://api.github.com/repos/kasparsd

using the following Nginx config rules:

proxy_cache_path /var/www/cache-github  levels=1:2 keys_zone=github:10m;

server {
        server_name github.kaspars.net;

        location ~ ^/kasparsd/(.*) {
                resolver 8.8.8.8;
                proxy_pass  https://api.github.com/repos/kasparsd/$1;
                proxy_cache  github;
                proxy_cache_valid  200 302 1h;
                proxy_ignore_headers  Expires Cache-Control Set-Cookie X-Accel-Redirect X-Accel-Expires;
                proxy_cache_use_stale  error timeout invalid_header updating http_500 http_502 http_503 http_504;

                #proxy_set_header  User-Agent YOURGITHUBUSERNAME;
                #proxy_set_header  Authorization "Basic ...";
        }
}

Where resolver is the DNS server for resolving proxy_pass resource and all 200 and 302 responses are cached for an hour while forcing Nginx to ignore Expires and Cache-Control response headers to avoid skipping cache. GitHub also suggests sending optional User-Agent and Authorization headers.

8 Comments

  1. Ryan Hellyer says:

    There’s a free service which does this now too :)

    http://rawgit.com/

  2. Nimish says:

    Hi Kaspars,

    I’ve tried to set this up and it’s currently not working for me. The rate limit still decreases.

    I do get a HTTP/1.1 304 Not Modified when using the etag but even then the rate limit still decreases.

    Is this still a valid solution to cache api.github.com?

    I can post up my nginx config without the api token etc. If that would help?

  3. Nimish says:

    The config you linked me to is in a single line, I’m not sure I understand what that is?

    My config is:

    proxy_cache_path /var/lib/nginx/cache-github  levels=1:2 keys_zone=github:1024m;
    
    server {
            server_name api.github-proxy.corp.mydomain.com;
    
            location / {
                    resolver 8.8.8.8;
                    proxy_pass  https://api.github.com/;
                    proxy_set_header Authorization "token MyGeneratedToken";
                    proxy_cache  github;
                    proxy_cache_valid  200 302 1h;
                    proxy_ignore_headers  Expires Cache-Control Set-Cookie X-Accel-Redirect X-Accel-Expires;
                    proxy_cache_use_stale  error timeout invalid_header updating http_500 http_502 http_503 http_504;
    
            }
    }
    • Kaspars says:

      Nimish, could you check if Nginx has the permissions to write to the cache folder /var/lib/nginx/cache-github? Secondly, you probably need to send the User-Agent header with your username.

      • Nimish says:

        Here are the permissions on the directories

        /var/lib/nginx
        drwxrwxrwx. 4 nginx nginx 35 Jul 15 15:28 nginx

        /var/lib/nginx/cache-github

        drwxrwxrwx. 2 nginx nginx 6 Jul 15 15:28 cache-github

        I’ve set the following as well but still get the X-RateLimit-Remaining decreasing on each request.

        It decreases even if I use the ETag header in my curl request and get HTTP/1.1 304 Not Modified reply.

        -H ‘If-None-Match: “ETag_Number”‘

        proxy_set_header User-Agent GitHubUserName;
        proxy_set_header Authorization “token …”;

  4. Nimish says:

    This gives me the correct info via curl and my web browser but it doesn’t seem to be caching anything and the X-RateLimit-Remaining counter decreases with every subsequent request.

Leave a Reply to Nimish Cancel reply