Importing python libraries from Github

Question

I have written some libraries in Python for use in my project. I have stored them locally on my system and also remotely on Github. Now every time I write some code I use sys.path.append() in the beginning to help import my libraries from the directory in my system. I was wondering that if there is anyway to import these files directly from my Github repository

The link to my repo is this - Quacpy

Staats

If they have a setup.py you can pip install ... direct from a GitHub repo – Trusty

Answer 1

This feels a bit off the wall but might work for you (if any of your libraries depend on each other, you'll have to change those imports to githubimports too!?):

import requests
def githubimport(user, repo, module):
   d = {}
   url = 'https://raw.githubusercontent.com/{}/{}/master/{}.py'.format(user, repo, module)
   r = requests.get(url).text
   exec(r, d)
   return d

qoperator = githubimport('biryani', 'Quacpy', 'qoperator')

Mayhap

Answer 2

If you want to use a repo which has to be installed, I'm not sure how you would want to automate installation inside another python script (also what to do if the installation fails).

However, if you just want to use some methods from another file, you could download that file and then import it:

import urllib2

def download(url):
    filename = url.split('/')[-1]
    print 'Downloading', filename
    f = urllib2.urlopen(url)
    data = f.read()
    f.close()
    with open(filename, 'w') as myfile:
        myfile.write(data)

# get repository
download('https://raw.githubusercontent.com/biryani/Quacpy/master/auxfun.py')

# try to import something from it
from auxfun import qregnorm
q = qregnorm([0, 1, 2])
print 'Success! q =', q

Maybe you could even download the whole zip, unzip it and then import the files.

Eggett

Thanks! This is exactly what I was looking for. Turns out it is not so simple. – Staats

If an answer was exactly what you were looking for, you should consider making it the accepted answer ;-) – Eggett

Answer 3

Assuming you have a valid setup.py file, pip supports git-based installation. See https://pip.pypa.io/en/latest/reference/pip_install.html#git for details

Spoiler: Because you don't have a setup.py file, you'll see the following error if you try using pip currently:

pip install -e git+https://github.com/biryani/Quacpy.git#egg=quacpy
Obtaining quacpy from git+https://github.com/biryani/Quacpy.git#egg=quacpy
  Cloning https://github.com/biryani/Quacpy.git to /.../quacpy
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 18, in <module>
    IOError: [Errno 2] No such file or directory: '/.../quacpy/setup.py'

    ----------------------------------------
    Command "python setup.py egg_info" failed with error code 1 in /.../quacpy

Conquest

Thanks for the answer. But I am not looking to install the module, rather importing some files from it remotely every time I run a code. Your method is the sensible way to do it though. – Staats

Answer 4

This imports the whole repository as a module, Python 3:

import sys
import urllib.request # python 3
import zipfile
import os

REPOSITORY_ZIP_URL = 'https://github.com/biryani/Quacpy/archive/master.zip'

filename, headers = urllib.request.urlretrieve(REPOSITORY_ZIP_URL)

zip = zipfile.ZipFile(filename)

directory = filename + '_dir'

zip.extractall(directory)

module_directory_from_zip = os.listdir(directory)[0]
module_directory = 'Quacpy'
os.rename(os.path.join(directory, module_directory_from_zip),
          os.path.join(directory, module_directory))

sys.path.append(directory)

import Quacpy

Cartier

Answer 5

One workaround I use in Colab is copying the file to Colab's folder and then importing it in Python's interpreter. With your library this would be:

!wget -q https://github.com/biryani/Quacpy/__init__.py -O __init__.py > txt.log
from __init__ import *

There should be a way to download the whole repo, unzip if necessary, and then import.

Outstanding