How can a Python module be imported from a URL?

Question

As an experiment, I want to see how to import a Python module from a URL. The hypothetical goal here would be to import from a central location which keeps the modules up-to-date. How could this be done?

My attempt is as follows:

>>> import urllib
>>> 
>>> def import_URL(URL):
...     exec urllib.urlopen(URL) in globals()
... 
>>> import_URL("https://cdn.rawgit.com/wdbm/shijian/master/shijian.py")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in import_URL
TypeError: exec: arg 1 must be a string, file, or code object

---

EDIT: Martijn Pieters identified a fix for the example code that results in the string representation of the remote module. The resulting code is as follows:

import urllib
def import_URL(URL):
    exec urllib.urlopen(URL).read() in globals()

Foggy

You really shouldn't. Don't load code over the internet and run it, not unless you want to be hacked. That said, your only error is not calling .read() on the urlopen() result. – Hookworm

Rather than load from a URL, use a Revision Control system (git, mercurial, etc.) to keep code up-to-date. – Hookworm

Martijn Pieters Thanks for your comments there and for spotting the read() problem. That results in a string that can be executed. I'm aware of the security problems; this is just for experimental purposes. Do you know if there is any effort to have a more secure approach to this idea? – Foggy

Not really. Python is notoriously hard to lock down. The only watertight approach involves virtual machines and shutting those down after a timeout. – Hookworm

It looks like you are customizing the "import", while, you need to look into imp , docs.python.org/2/library/imp.html, it already provides some method into the "import", such as get the magic number (imp.get_magic()) which is useful to check if the file/module has changed. – Coeternity

BTW, if you detect the module has been uploaded, you need to reload the module to get the newest code. – Coeternity

Answer 1

Basically there is a module exactly for this purpose called httpimport. Currently it supports importing from a URL that contains the package/module and also from archives (.tar.*, .zip) that can be found in URLs (this is a way to handle remote dependencies).

It is fully integrated with Python's import system so you don't need to exec anything in globals(). You just:

>>> with httpimport.remote_repo(['package1'], 'http://my-codes.example.com/python_packages'):
...     import package1
...

and then package1 is usable for the rest of the script like it was a local resource.

---

Disclaimer: I'm the author of this module.

---

Edit (31/01/2023): The syntax of most httpimport commands has changed after the 1.0.0 re-write. The new parameters for remote_repo omits the first argument, as below:

>>> with httpimport.remote_repo('http://my-codes.example.com/python_packages'):
...     import package1
...

You might want to look at all usage examples provided in the repository README: https://github.com/operatorequals/httpimport#basic-usage

Thermaesthesia

is there a way to use httpimport to just directly import functions that are defined in a gist? It's not a package or repo, just a single gist file with some functions in it. Can httpimport solve this use case? – Taxable

absolutely! Take this Gist: gist.github.com/operatorequals/… The Raw mode returns this URL: gist.githubusercontent.com/operatorequals/… so use load with this URL: gist.githubusercontent.com/operatorequals/… (without the filename) and load the stealthy_opener "module". – Thermaesthesia

Answer 2

Yes you can.

Just fetch the module with the url and once you have it store it as a string where you can run it using eval()

Using urllib and eval it can be done easily:

import urllib.request
a = urllib.request.urlopen(url)
eval(a.read())

Do note that some modules (such as Pygame and Pydub) require runtimes and they could not be run using eval() because of the missing runtimes.

Good luck with your project, I hope I helped.

Energumen

Isn't this a security risk? And, what if dependent modules should also be loaded from the url? – Machicolate

@Machicolate Yes there is some security risk. But for a lot of purposes it doesn't really matter. Since you are downloading the files temporarily in memory, nothing is actually going online (also if it got hacked ect you could potentially get a virus, but that's the same for pretty much every online application). You could also load runtimes from the url, but that would involve actually downloading everything and writing to file, then deleting it once you are done (good luck if the user decides to force shut your application before it has deleted itself. – Energumen

On windows you might be able to use ctypes, and the winapi to dynamically load runtimes from memory, but I have no idea how that could be done – Energumen

Is the security risk referred to higher than in cloning a git repo? – Allin

I'd say cloning a git repo would be way safer, since it would be developed with security in mind. However that's not to say this couldn't be equally as safe, however it would require someone with the necessary knowledge to ensure that (something that is way out of my depth) so I honestly couldn't give a concrete answer. Also note something like a git repo couldn't be directly put into memory (at least not as easily). but then again it depends what you want to do @RonyArmon – Energumen

Answer 3

Here is an example to copy and paste:

url = "https://gist.githubusercontent.com/operatorequals/ee5049677e7bbc97af2941d1d3f04ace/raw/e55fa867d3fb350f70b2897bb415f410027dd7e4"
with httpimport.remote_repo(["hello"], url):
    import hello
hello.hello()

This answer is inspired by the answer of @operatorequals

Septic