How do I receive Github Webhooks in Python

Question

Github offers to send Post-receive hooks to an URL of your choice when there's activity on your repo. I want to write a small Python command-line/background (i.e. no GUI or webapp) application running on my computer (later on a NAS), which continually listens for those incoming POST requests, and once a POST is received from Github, it processes the JSON information contained within. Processing the json as soon as I have it is no problem. The POST can come from a small number of IPs given by github; I plan/hope to specify a port on my computer where it should get sent.

The problem is, I don't know enough about web technologies to deal with the vast number of options you find when searching.. do I use Django, Requests, sockets,Flask, microframeworks...? I don't know what most of the terms involved mean, and most sound like they offer too much/are too big to solve my problem - I'm simply overwhelmed and don't know where to start.

Most tutorials about POST/GET I could find seem to be concerned with either sending or directly requesting data from a website, but not with continually listening for it.

I feel the problem is not really a difficult one, and will boil down to a couple of lines, once I know where to go/how to do it. Can anybody offer pointers/tutorials/examples/sample code?

Panchito

web.py is where I would start. – Knowling

The thing is, I don't want/need to serve webpages. – Panchito

Then use plain werkzeug (which is what flask uses as its backend) – Rightness

I'm doing this thing myself. I'm probably going to use Tornado to do it, mostly because Tornado is already installed on the system that needs to listen. Any micro HTTP server framework should be fine. A friend of mine swears by webobj. – Wampumpeag

How would i get this post data from bash? – Tenpins

Answer 1

Here's a basic web.py example for receiving data via POST and doing something with it (in this case, just printing it to stdout):

import web

urls = ('/.*', 'hooks')

app = web.application(urls, globals())

class hooks:
    def POST(self):
        data = web.data()
        print
        print 'DATA RECEIVED:'
        print data
        print
        return 'OK'

if __name__ == '__main__':
    app.run()

I POSTed some data to it using hurl.it (after forwarding 8080 on my router), and saw the following output:

$ python hooks.py 
http://0.0.0.0:8080/

DATA RECEIVED: 
test=thisisatest&test2=25

50.19.170.198:33407 - - [27/Jan/2013 10:18:37] "HTTP/1.1 POST /hooks" - 200 OK

You should be able to swap out the print statements for your JSON processing.

To specify the port number, call the script with an extra argument:

$ python hooks.py 1234 

Knowling

OK, I accepted this answer since it's what I tried first, and it works satisfactorily for a first implementation. Later I'll maybe switch it out for plain Werkzeug as recommended in a question comment. – Panchito

I am also new to webhooks and python both, I want to understand , do I need to use URLs given be the webhook provider , are these URLs are different from normal APIs? What is the calling mechanism for continuous listening after hosting this application? – Windbreak

Answer 2

First thing is, web is request-response based. So something will request your link, and you will respond accordingly. Your server application will be continuously listening on a port; that you don't have to worry about.

Here is the similar version in Flask (my micro framework of choice):

from flask import Flask, request
import json

app = Flask(__name__)

@app.route('/',methods=['POST'])
def foo():
   data = json.loads(request.data)
   print "New commit by: {}".format(data['commits'][0]['author']['name'])
   return "OK"

if __name__ == '__main__':
   app.run()

Here is a sample run, using the example from github:

Running the server (the above code is saved in sample.py):

burhan@lenux:~$ python sample.py 
 * Running on http://127.0.0.1:5000/

Here is a request to the server, basically what github will do:

burhan@lenux:~$ http POST http://127.0.0.1:5000 < sample.json
HTTP/1.0 200 OK
Content-Length: 2
Content-Type: text/html; charset=utf-8
Date: Sun, 27 Jan 2013 19:07:56 GMT
Server: Werkzeug/0.8.3 Python/2.7.3

OK # <-- this is the response the client gets

Here is the output at the server:

New commit by: Chris Wanstrath
127.0.0.1 - - [27/Jan/2013 22:07:56] "POST / HTTP/1.1" 200 -

Clapboard

I think the link to the github example is incorrect. – Lagrange

It was correct at the time of writing (more than 3 years ago). – Clapboard

Answer 3

I would use:

https://github.com/carlos-jenkins/python-github-webhooks

You can configure a web server to use it, or if you just need a process running there without a web server you can launch the integrated server:

python webhooks.py

This will allow you to do everything you said you need. It, nevertheless, requires a bit of setup in your repository and in your hooks.

Late to the party and shameless autopromotion, sorry.

Trover

I'm trying to use this example to deploy a web app that tests github commits, specifically to run a Bamboo build plan and report back the status. Any suggestions/advice? I appreciate any help you can offer. – Lagrange

if we are promoting our own stuff here is my github -> puppet adapter that uses AWS Lambda. I also figured out how to process the HMAC headers github.com/vorsprung/github-to-aws-puppet-webhook – Rattigan

Answer 4

If you are using Flask, here's a very minimal code to listen for webhooks:

from flask import Flask, request, Response

app = Flask(__name__)

@app.route('/webhook', methods=['POST'])
def respond():
    print(request.json) # Handle webhook request here
    return Response(status=200)

And the same example using Django:

from django.http import HttpResponse
from django.views.decorators.http import require_POST

@require_POST
def example(request):
    print(request.json) # Handle webhook request here
    return HttpResponse('Hello, world. This is the webhook response.')

If you need more information, here's a great tutorial on how to listen for webhooks with Python.

Pattani

Answer 5

If you're looking to watch for changes in any repo...

1. If you own the repo that you want to watch

• In your repo page, Go to settings
• click webhooks, new webhook (top right)
• give it your ip/endpoint and setup everything to your liking
• use any server to get notified

2. Not your Repo

• take the url you want i.e https://github.com/fire17/gd-xo/

• add /commits/master.atom to the end such as:

  https://github.com/fire17/gd-xo/commits/master.atom

• Use any library you want to get that page's content, like:

• filter out the keys you want, for example the element

response = requests.get("https://github.com/fire17/gd-xo/commits/master.atom").text
response.split("<updated>")[1].split("</updated>")[0]
'2021-08-06T19:01:53Z'

• make a loop that checks this every so often and if this string has changed, then you can initiate a clone/pull request or do whatever you like

Abruption