Supporting lexical-scope ScriptBlock parameters (e.g. Where-Object)

Question

Consider the following arbitrary function and test cases:

Function Foo-MyBar {
    Param(
        [Parameter(Mandatory=$false)]
        [ScriptBlock] $Filter
    )

    if (!$Filter) { 
        $Filter = { $true } 
    }

    #$Filter = $Filter.GetNewClosure()

    Get-ChildItem "$env:SYSTEMROOT" | Where-Object $Filter   
}

##################################

$private:pattern = 'T*'

Get-Help Foo-MyBar -Detailed

Write-Host "`n`nUnfiltered..."
Foo-MyBar

Write-Host "`n`nTest 1:. Piped through Where-Object..."
Foo-MyBar | Where-Object { $_.Name -ilike $private:pattern  }

Write-Host "`n`nTest 2:. Supplied a naiive -Filter parameter"
Foo-MyBar -Filter { $_.Name -ilike $private:pattern }

In Test 1, we pipe the results of Foo-MyBar through a Where-Object filter, which compares the objects returned to a pattern contained in a private-scoped variable $private:pattern. In this case, this correctly returns all the files/folders in C:\ which start with the letter T.

In Test 2, we pass the same filtering script directly as a parameter to Foo-MyBar. However, by the time Foo-MyBar gets to running the filter, $private:pattern is not in scope, and so this returns no items.

I understand why this is the case -- because the ScriptBlock passed to Foo-MyBar is not a closure, so does not close over the $private:pattern variable and that variable is lost.

I note from comments that I previously had a flawed third test, which tried to pass {...}.GetNewClosure(), but this does not close over private-scoped variables -- thanks @PetSerAl for helping me clarify that.

The question is, how does Where-Object capture the value of $private:pattern in Test 1, and how do we achieve the same behaviour in our own functions/cmdlets?

(Preferably without requiring the caller to have to know about closures, or know to pass their filter script as a closure.)

I note that, if I uncomment the $Filter = $Filter.GetNewClosure() line inside Foo-MyBar, then it never returns any results, because $private:pattern is lost.

(As I said at the top, the function and parameter are arbitrary here, as a shortest-form reproduction of my real problem!)

Afforest

Can you recheck Test 3? As far as I know GetNewClosure does not capture private variables. – Amphitropous

@PetSerAl - you might be right, although now I'm experiencing different behaviour in one ISE window (not tab) to another. In the ISE window I was testing in, $private:pattern is being closed over, but I can't recreate in a fresh window. Question stands, though, given Test 1 still works as described. – Afforest

@PetSerAl - OK, Remove-Variable pattern reset it -- I must have defined $pattern at some point earlier. I'll update to remove reference to Test 3 – Afforest

Simple way to solve it would be to put Foo-MyBar in separate module. – Amphitropous

Answer 1

The example given does not work because calling a function will enter a new scope by default. Where-Object will still invoke the filter script without entering one, but the scope of the function does not have the private variable.

There's three ways around this.

Put the function in a different module than the caller

Every module has a SessionState which has its own stack of SessionStateScopes. Every ScriptBlock is tied to the SessionState is was parsed in.

If you call a function defined in a module, a new scope is created within that module's SessionState, but not within the top level SessionState. Therefore when Where-Object invokes the filter script without entering a new scope, it does so on the current scope for the SessionState to which that ScriptBlock is tied.

This is a bit fragile, because if you want to call that function from your module, well you can't. It'll have the same issue.

Call the function with the dot source operator

You most likely already know the dot-source operator (.) for invoking script files without creating a new scope. That also works with command names and ScriptBlock objects.

. { 'same scope' }
. Foo-MyBar

Note, however, that this will invoke the function within the current scope of the SessionState that the function is from, so you cannot rely on . to always execute in the caller's current scope. Therefore, if you invoke functions associated with a different SessionState with the dot-source operator - such as functions defined in a (different) module - it may have unintended effects. Variables created will persist to future function invocations and any helper functions defined within the function itself will also persist.

Write a Cmdlet

Compiled commands (cmdlets) do not create a new scope when invoked. You can also use similar API's to what Where-Object use (though not the exact same ones)

Here's a rough implementation of how you could implement Where-Object using public API's

using System.Management.Automation;

namespace MyModule
{
    [Cmdlet(VerbsLifecycle.Invoke, "FooMyBar")]
    public class InvokeFooMyBarCommand : PSCmdlet
    {
        [Parameter(ValueFromPipeline = true)]
        public PSObject InputObject { get; set; }

        [Parameter(Position = 0)]
        public ScriptBlock FilterScript { get; set; }

        protected override void ProcessRecord()
        {
            var filterResult = InvokeCommand.InvokeScript(
                useLocalScope: false,
                scriptBlock: FilterScript,
                input: null,
                args: new[] { InputObject });

            if (LanguagePrimitives.IsTrue(filterResult))
            {
                WriteObject(filterResult, enumerateCollection: true);
            }
        }
    }
}

Trichomoniasis

Can't believe I overlooked extending PSCmdlet, this should be the accepted answer, +1 – Chops

@mklement0 in the scenario of a caller from outside of a module dot sourcing a function from a module, the current scope would most likely be the highest scope for the session state. Basically dot sourcing something from a different session state will not cause it to be invoked in your current scope, but the current scope of the session state (module) the command is tied to. Additional reading – Trichomoniasis

Thanks for the clarification; to call this behavior suprising is an understatment. Re target scope inside module: could it ever be something other than the module's top-level scope? The link to your blog is broken, because the path component isn't all-lowercase; here's the working link: seeminglyscience.github.io/powershell/2017/09/30/… – Forfeiture

@Forfeiture good question! Yes it can, but it's a lot less likely. Lets say ModuleA has FunctionA and FunctionB, and ModuleB has FunctionC. If FunctionA calls FunctionC and FunctionC dot sources FunctionB, then FunctionB will be invoked in the scope created by the invocation of FunctionA. Hopefully that is at least vaguely comprehensible... scopes are... complicated. Also thank you for the URL correction! Autocorrect strikes again. – Trichomoniasis

Just to make it more complicated: scopes are not stack but tree, because single parent scope can have multiple active child scopes in the same time. – Amphitropous

@Forfeiture For example: each instance of f have its own -Scope 0 $a, but -Scope 1 $a is shared between them. – Amphitropous

Revisiting this, I realized that the cmdlet implementation in this answer works in principle, but the ScriptBlock argument doesn't support use of $_ to refer to the current pipeline object (that object is passed via a positional argument instead). The workaround is cumbersome - see this discussion on GitHub and, for a solution written in PowerShell, this answer. – Forfeiture

Answer 2

how does Where-Object capture the value of $private:pattern in Test 1

As can be seen in the source code for Where-Object in PowerShell Core, PowerShell internally invokes the filter script without confining it to its own local scope (_script is the private backing field for the FilterScript parameter, notice the useLocalScope: false argument passed to DoInvokeReturnAsIs()):

protected override void ProcessRecord()
{
    if (_inputObject == AutomationNull.Value)
        return;

    if (_script != null)
    {
        object result = _script.DoInvokeReturnAsIs(
            useLocalScope: false, // <-- notice this named argument right here
            errorHandlingBehavior: ScriptBlock.ErrorHandlingBehavior.WriteToCurrentErrorPipe,
            dollarUnder: InputObject,
            input: new object[] { _inputObject },
            scriptThis: AutomationNull.Value,
            args: Utils.EmptyArray<object>());

        if (_toBoolSite.Target.Invoke(_toBoolSite, result))
        {
            WriteObject(InputObject);
        }
    }
    // ...
}

how do we achieve the same behaviour in our own functions/cmdlets?

We don't - DoInvokeReturnAsIs() (and similar scriptblock invocation facilities) are marked internal and can therefore only be invoked by types contained in the System.Management.Automation assembly

Chops