Custom IronPython import resolution
Asked Answered
M

4

9

I am loading an IronPython script from a database and executing it. This works fine for simple scripts, but imports are a problem. How can I intercept these import calls and then load the appropriate scripts from the database?

EDIT: My main application is written in C# and I'd like to intercept the calls on the C# side without editing the Python scripts.

EDIT: From the research I've done, it looks like creating your own PlatformAdaptationLayer is the way you're supposed to to implement this, but it doesn't work in this case. I've created my own PAL and in my testing, my FileExsists method gets called for every import in the script. But for some reason it never calls any overload of the OpenInputFileStream method. Digging through the IronPython source, once FileExists returns true, it tries to locate the file itself on the path. So this looks like a dead end.

Mccormack answered 5/11, 2010 at 12:25 Comment(0)
M
11

After a great deal of trial and error, I arrived at a solution. I never managed to get the PlatformAdaptationLayer approach to work correctly. It never called back to the PAL when attempting to load the modules.

So what I decided to do was replace the built-in import function by using the SetVariable method as shown below (Engine and Scope are protected members exposing the ScriptEngine and ScriptScope for the parent script):

delegate object ImportDelegate(CodeContext context, string moduleName, PythonDictionary globals, PythonDictionary locals, PythonTuple tuple);

protected void OverrideImport()
{
    ScriptScope scope = IronPython.Hosting.Python.GetBuiltinModule(Engine);
    scope.SetVariable("__import__", new ImportDelegate(DoDatabaseImport));
}

protected object DoDatabaseImport(CodeContext context, string moduleName, PythonDictionary globals, PythonDictionary locals, PythonTuple tuple)
{
    if (ScriptExistsInDb(moduleName))
    {
        string rawScript = GetScriptFromDb(moduleName);
        ScriptSource source = Engine.CreateScriptSourceFromString(rawScript);
        ScriptScope scope = Engine.CreateScope();
        Engine.Execute(rawScript, scope);
        Microsoft.Scripting.Runtime.Scope ret = Microsoft.Scripting.Hosting.Providers.HostingHelpers.GetScope(scope);
        Scope.SetVariable(moduleName, ret);
        return ret;
     }
     else
     {   // fall back on the built-in method
         return IronPython.Modules.Builtin.__import__(context, moduleName);
     }
}

Hope this helps someone!

Mccormack answered 8/11, 2010 at 20:28 Comment(1)
Thank you! I have exactly the same use case as yours.Pheni
P
10

I was just trying to do the same thing, except I wanted to store my scripts as embedded resources. I'm creating a library that is a mixture of C# and IronPython and wanted to distribute it as a single dll. I wrote a PlatformAdaptationLayer that works, it first looks in the resources for the script that's being loaded, but then falls back to the base implementation which looks in the filesystem. Three parts to this:

Part 1, The custom PlatformAdaptationLayer

namespace ZenCoding.Hosting
{
    internal class ResourceAwarePlatformAdaptationLayer : PlatformAdaptationLayer
    {
        private readonly Dictionary<string, string> _resourceFiles = new Dictionary<string, string>();
        private static readonly char Seperator = Path.DirectorySeparatorChar;
        private const string ResourceScriptsPrefix = "ZenCoding.python.";

        public ResourceAwarePlatformAdaptationLayer()
        {
            CreateResourceFileSystemEntries();
        }

        #region Private methods

        private void CreateResourceFileSystemEntries()
        {
            foreach (string name in Assembly.GetExecutingAssembly().GetManifestResourceNames())
            {
                if (!name.EndsWith(".py"))
                {
                    continue;
                }
                string filename = name.Substring(ResourceScriptsPrefix.Length);
                filename = filename.Substring(0, filename.Length - 3); //Remove .py
                filename = filename.Replace('.', Seperator);
                _resourceFiles.Add(filename + ".py", name);
            }
        }

        private Stream OpenResourceInputStream(string path)
        {
            string resourceName;
            if (_resourceFiles.TryGetValue(RemoveCurrentDir(path), out resourceName))
            {
                return Assembly.GetExecutingAssembly().GetManifestResourceStream(resourceName);
            }
            return null;
        }

        private bool ResourceDirectoryExists(string path)
        {
            return _resourceFiles.Keys.Any(f => f.StartsWith(RemoveCurrentDir(path) + Seperator));
        }

        private bool ResourceFileExists(string path)
        {
            return _resourceFiles.ContainsKey(RemoveCurrentDir(path));
        }


        private static string RemoveCurrentDir(string path)
        {
            return path.Replace(Directory.GetCurrentDirectory() + Seperator, "").Replace("." + Seperator, "");
        }

        #endregion

        #region Overrides from PlatformAdaptationLayer

        public override bool FileExists(string path)
        {
            return ResourceFileExists(path) || base.FileExists(path);
        }

        public override string[] GetFileSystemEntries(string path, string searchPattern, bool includeFiles, bool includeDirectories)
        {
            string fullPath = Path.Combine(path, searchPattern);
            if (ResourceFileExists(fullPath) || ResourceDirectoryExists(fullPath))
            {
                return new[] { fullPath };
            }
            if (!ResourceDirectoryExists(path))
            {
                return base.GetFileSystemEntries(path, searchPattern, includeFiles, includeDirectories);
            }
            return new string[0];
        }

        public override bool DirectoryExists(string path)
        {
            return ResourceDirectoryExists(path) || base.DirectoryExists(path);
        }

        public override Stream OpenInputFileStream(string path)
        {
            return OpenResourceInputStream(path) ?? base.OpenInputFileStream(path);
        }

        public override Stream OpenInputFileStream(string path, FileMode mode, FileAccess access, FileShare share)
        {
            return OpenResourceInputStream(path) ?? base.OpenInputFileStream(path, mode, access, share);
        }

        public override Stream OpenInputFileStream(string path, FileMode mode, FileAccess access, FileShare share, int bufferSize)
        {
            return OpenResourceInputStream(path) ?? base.OpenInputFileStream(path, mode, access, share, bufferSize);
        }

        #endregion
    }
}

You would need to change the constant ResourceScriptsPrefix to whatever your base namespace is where you stored the python scripts.

Part 2, The custom ScriptHost

namespace ZenCoding.Hosting
{
    internal class ResourceAwareScriptHost : ScriptHost
    {
        private readonly PlatformAdaptationLayer _layer = new ResourceAwarePlatformAdaptationLayer();
        public override PlatformAdaptationLayer PlatformAdaptationLayer
        {
            get { return _layer; }
        }
    }
}

Part 3, finally, how to get a Python engine using your custom stuff:

namespace ZenCoding.Hosting
{
    internal static class ResourceAwareScriptEngineSetup
    {
        public static ScriptEngine CreateResourceAwareEngine()
        {
            var setup = Python.CreateRuntimeSetup(null);
            setup.HostType = typeof(ResourceAwareScriptHost);
            var runtime = new ScriptRuntime(setup);
            return runtime.GetEngineByTypeName(typeof(PythonContext).AssemblyQualifiedName);
        }
    }
}

It would be easy to change this to load scripts from some other location, like a database. Just change the OpenResourceStream, ResourceFileExists and ResourceDirectoryExists methods.

Hope this helps.

Puss answered 10/1, 2011 at 7:31 Comment(4)
could you add what version of IronPython this was for? I think this was for an older version than IronPython 2.7?Bevins
Yes, this was two years ago, so whichever version was the current one then. Probably 2.6, but I'm not sure.Puss
Is there a better way in 2.7+ ? The above still works btw.Hathaway
@simonjpascoe, I tried to explain two other approaches in this answer.Alix
M
1

You can re-direct all I/O to the database using the PlatformAdaptationLayer. To do this you'll need to implement a ScriptHost which provides the PAL. Then when you create the ScriptRuntime you set the HostType to your host type and it'll be used for the runtime. On the PAL you then override OpenInputFileStream and return a stream object which has the content from the database (you could just use a MemoryStream here after reading from the DB).

If you want to still provide access to file I/O you can always fall back to FileStream's for "files" you can't find.

Mulderig answered 6/11, 2010 at 3:34 Comment(4)
I am working on a solution that uses this as a starting point (I found some useful stuff scattered about (efreedom.com/Question/1-3264029/… and mail-archive.com/[email protected]/msg06080.html). The current problem seems to be deciphering how the various Platform Adaptation Layer methods get called. I can't seem to find any documentation on this at all.Mccormack
The Silverlight host may be a reasonable example of how to do this if you want to see an existing implementation.Mulderig
See my answer for a fairly simple custom PlatformAdaptationLayerPuss
This approach seems useful if you want to just have the files stored somewhere other than the expected filesystem location, but not if you wanted to, say, precompile them.Revareval
A
0

You need to implement import hooks. Here's an SO question with pointers: PEP 302 Example: New Import Hooks

Amabelle answered 5/11, 2010 at 12:47 Comment(2)
Not sure if I was clear enough in my original phrasing of the question. The application calling the Python script is written in C# and I'd like to treat the Python script as a black box as much as possible, so I'd like to be able to intercept the imports on the C# side if at all possible. I've edited the original question to reflect this additional information.Mccormack
See above. This looks like it should work, but it doesn't. Perhaps the IronPython Import implementation breaks the standard approach?Mccormack

© 2022 - 2024 — McMap. All rights reserved.