IronPython dependencies for scripts stored as strings

Question

I have a C# application that stores python script files (*.py) as strings. I load them using:

scriptEngine.CreateScriptSourceFromString(code);

But now I have multiple script files with dependencies between them (imports). In order to handle dependencies, I could save all the strings back to files in a folder and load the script that i want to execute using:

scriptEngine.CreateScriptSourceFromFile(filePath);

but this would make all the script files visible. Is there a way to achieve this in a in-memory way, so that the script files are not first saved to disk but loaded from the strings directly?

TL;DR: Example of how this might look:

myutils.py:

def SomeMethod(p):
    print ('SomeMethod(p=%s)' % p)

script1.py:

import myutils;

if __name__ == '__main__':
    myutils.SomeMethod('script1')

script2.py:

import myutils;

if __name__ == '__main__':
    myutils.SomeMethod('script2')

My application has the scripts stored as strings. something like

Dictionary<string, string> filePathToContent = new Dictionary<string, string>();

filePathToContent["myutils.py"] = "..."; // The script file content.
filePathToContent["script1.py"] = "..."; // The script file content.
filePathToContent["script2.py"] = "..."; // The script file content.

I want to call script1.py without having to first save the scripts into a folder. Note: the code is just a simplified example of what I have.

Casia

Why not save the concated script into one temp. file and deleting it when you´ve done? – Podiatry

@HimBromBeere: I would have to handle to order correctly. This can became very complex – Casia

@RobotMess, how are the dependencies declared? Are the strings actually created from (embedded) .py files and contain imports for the other parts? Are all scripts and dependencies known? How? – Playtime

@SimonOpelt: yes the strings are actually embedded py files, and they contain imports. All the script strings are known, but in order to get the dependencies i would have to parse the strings. – Casia

@SimonOpelt: I added an example in my question, I hope this clarifies my situation. PS: I love your work :) – Casia

Thank you ;) Have you seen this answer? I think this would be a reasonable solution assuming you want to handle the custom dependency/file lookup in C#. As an alternative you could implement a custom importer using sys.path_hooks from within ironpython. – Playtime

@SimonOpelt: that is the answer I was looking for. Could you please write the comment as an answer, this way I can pick it as the right answer and award you the bounty – Casia

Answer 1

There are several approaches for custom import handling in IronPython and Python in general. Most concepts are defined in PEP 0302 (New Import Hooks).

Two python mechanisms that can solve the requirements are meta_path and path_hooks. Both can be implemented in Python or (in case of IronPython) C#/.NET as well. Given that the question concerns hosting IronPython from C# implementing the import infrastructure can work either way.

Using meta_path

IronPython ships with ResourceMetaPathImporter which allows you to have a ZIP-archive containing your scripts as an embedded resource. Assuming such an archive is called scripts.zip contained in the currently executing assembly, the required setup could look like:

var engine = Python.CreateEngine();
var sysScope = engine.GetSysModule();
List metaPath = sysScope.GetVariable("meta_path");
var importer = new ResourceMetaPathImporter(Assembly.GetExecutingAssembly(), "scripts.zip");
metaPath.Add(importer);
sysScope.SetVariable("meta_path", metaPath);

This approach works well if the assemblies and scripts are known and ZIP-packaging does not disturb the development process.

Using path_hooks

Path hooks contain a chain of importers that are queried for all items in sys.path in order to determine if they can handle a given path. An importer similar to zipimport.cs but responsible for embedded resources in DLLs/EXEs instead of ZIP archives. This could provide a more general approach that handles additional files by just adding a DLL to the path.

Using PlatformAdaptationLayer

A third approach works by providing a PlatformAdaptationLayer which is part of Microsoft.Scripting/IronPython. This answer shows a full working example of an platform adaption layer resolving embedded resources of a predefined assembly and package namespace.

General note: Related issue/discussion on github.

Playtime

Answer 2

You could create the different scripts as one function each and call those functions based on arguments given

ScriptScope scope = scriptEngine.CreateScope();

scope.SetVariable("language", "en");

scriptEngine.Execute(scope);

and the python (silly example I know):

def doEnStuff():
  print "english"

def doEsStuff():
  print "espagna"

if language == "en"
  doEnStuff()

Temperate

But that is exactly the problem. How do I create the scripts considering that they are in different "strings" and that there might be dependencies between them. On the file system the interpreter would resolve the dependencies and include the right file. How do I do this in memory? – Casia

As I already specified, I am able to create a script without dependencies. – Casia

Answer 3

I developed the following solution, remember to store your script file in your project resources.

using IronPython.Hosting;
using Microsoft.Scripting.Hosting;
using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.Scripting.Hosting.Providers;

namespace JJMasterData.Core.DataDictionary
{

    /// <summary>
    /// Class responsible for handling Python 3.4 scripts
    /// Gustavo Barros 05/01/2022
    /// </summary>
    public class PythonScriptManager
    {
        private ScriptScope Scope;

        private ScriptEngine _engine;
        private ScriptEngine Engine
        {
            get
            {
                if (_engine == null)
                {

                    ///Creating the Python Engine
                    _engine = Python.CreateEngine();

                    ///Setup JJMasterData scripts
                    Scope = _engine.CreateScope();

                    ///Adding IronPython StdLib
                    ICollection<string> searchPaths = _engine.GetSearchPaths();
                    string user = Environment.GetFolderPath(Environment.SpecialFolder.UserProfile);
                    searchPaths.Add(user + $"\\.nuget\\packages\\ironpython.stdlib\\3.4.0-alpha1\\content\\Lib");
                    _engine.SetSearchPaths(searchPaths);

                    ///Importing custom JJMasterData scripts
                    ImportScripts();
                }

                return _engine;
            }

        }

        protected void ImportScripts()
        {
            ///Repeat the same steps to import a new script
            string dataAccess = Encoding.UTF8.GetString(Properties.Resources.data_access);
            Engine.Execute(dataAccess, Scope);
            Scope.SetVariable("data_access", HostingHelpers.GetScope(Scope));
        }

        /// <summary>
        /// Executes a Python 3.4 script from a string.
        /// Gustavo Barros - 05/01/2022
        /// </summary>
        /// <param name="script">Python script to be executed.</param>
        /// <returns>Script dynamic return./returns>
        public object Run(string script) => Engine.CreateScriptSourceFromString(script).Execute(Scope);

    }
}

Horror

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. – Amyl