Powershell: Use a variable to reference a property of $_ in a script block

Question

$var =@(  @{id="1"; name="abc"; age="1"; },
          @{id="2"; name="def"; age="2"; } );
$properties = @("ID","Name","Age") ;
$format = @();
foreach ($p  in $properties)
{
    $format += @{label=$p ; Expression = {$_.$p}} #$_.$p is not working!
}
$var |% { [PSCustomObject]$_  } | ft $format

In the above example, I want to access each object's property through a variable name. But it cannot work as expected. So in my case, how to make

Expression = {$_.$p}

working?

Coyotillo

My bad, suppose the "properties" contains different property names. Please see the edited $properties. – Coyotillo

You can just use Expression = $p. – Cerveny

Or you can use Expression = &{$p=$p; {$_.$p}.GetNewClosure()}, if you really want script block for Expression. – Cerveny

@PetSerAl - I don't see GetNewClosure() working in this scenario unless you re-factor it to a nested loop. As it stands it's going to have problems closing over either $_ or $p, depending on where you do the closure. – Paisano

@Paisano It work fine for me. Can you show any evidence, that it not working properly in this scenario? – Cerveny

@PetSerAl - I didn't test it before I made the comment, but it does appear to work. I wouldn't have expected the closure to work over $_, but apparently it does leave that as a literal string. Learn something new every day. As an aside, when Get-NewClosure() was first introduce and I tested it against [scriptblock]::create(). Oddly, it took longer to re-close an existing script block than it did to create a brand new one using an expandable string. – Paisano

@Paisano Oddly, it took longer to re-close an existing script block than it did to create a brand new one using an expandable string. For which script block size? I expect GetNewClosure() taking same time independent of script block size. While [ScriptBlock]::Create() should take more time as script block size increasing. – Cerveny

@PetSerAl - It was admittedly a small script block (probably less than 20 characters and involving only a single variable). – Paisano

Answer 1

^{The OP's code and this answer use PSv3+ syntax. Casting a hashtable to [pscustomobject] is not supported in PSv2, but you can replace [pscustomobject] $_ with New-Object PSCustomObject -Property $_.}

As in many cases in the past, PetSerAl has provided the answer in terse (but very helpful) comments on the question; let me elaborate:

Your problem is not that you're using a variable ($p) to access a property per se, which does work (e.g., $p = 'Year'; Get-Date | % { $_.$p }).

Instead, the problem is that $p in script block { $_.$p } isn't evaluated until later, in the context of the Format-Table call, which means that the same, fixed value is used for all input objects - namely the value of $p at that point (which happens to be the last value that was assigned to $p in the foreach loop).

The cleanest and most generic solution is to call .GetNewClosure() on the script block to bind $p in the script block to the then-current, loop-iteration-specific value.

$format += @{ Label = $p; Expression = { $_.$p }.GetNewClosure() }

From the docs (emphasis added; update: the quoted passage has since been removed, but still applies):

In this case, the new script block is closed over the local variables in the scope that the closure is defined in. In other words, the current values of the local variables are captured and enclosed inside the script block that is bound to the module.

Note that automatic variable $_ is undefined inside the foreach loop (PowerShell defines it only in certain contexts as the input object at hand, such as in script blocks passed to cmdlets in a pipeline), so it remains unbound, as desired.

Caveats:

• While .GetNewClosure() as used above is convenient, it has the inefficiency drawback of invariably capturing all local variables, not just the one(s) needed; also, the returned script block runs in a dynamic (in-memory) module created for the occasion.

• A more efficient alternative that avoids this problem - and notably also avoids a bug in Windows PowerShell in which the closure over the local variables can break, namely when the enclosing script / function has a parameter with validation attributes such as [ValidateNotNull()] and that parameter is not bound (no value is passed)^[1] - is the following, significantly more complex expression ^{Tip of the hat again to PetSerAl, and Burt_Harris's answer here}:

  $format += @{ Label = $p; Expression = & { $p = $p; { $_.$p }.GetNewClosure() } }
  

  • & { ... } creates a child scope with its own local variables.
  • $p = $p then creates a local $p variable from its inherited value.
    To generalize this approach, you must include such a statement for each variable referenced in the script block.
  • { $_.$p }.GetNewClosure() then outputs a script block that closes over the child scope's local variables (just $p in this case).
  • The bug was originally reported as GitHub issue #3144 and has since been fixed in PowerShell (Core) 7+ (but won't be fixed in Windows PowerShell).

• For simple cases, mjolinor's answer may do: it indirectly creates a script block via an expanded string that incorporates the then-current $p value literally, but note that the approach is tricky to generalize, because just stringifying a variable value doesn't generally guarantee that it works as part of PowerShell source code (which the expanded string must evaluate to in order to be converted to a script block).

To put it all together:

# Sample array of hashtables.
# Each hashtable will be converted to a custom object so that it can
# be used with Format-Table.
$var = @(  
          @{id="1"; name="abc"; age="3" }
          @{id="2"; name="def"; age="4" }
       )

# The array of properties to output, which also serve as
# the case-exact column headers.
$properties = @("ID", "Name", "Age")

# Construct the array of calculated properties to use with Format-Table: 
# an array of output-column-defining hashtables.
$format = @()
foreach ($p in $properties)
{
    # IMPORTANT: Call .GetNewClosure() on the script block
    #            to capture the current value of $p.
    $format += @{ Label = $p; Expression = { $_.$p }.GetNewClosure() }
    # OR: For efficiency and full robustness (see above):
    # $format += @{ Label = $p; Expression = & { $p = $p; { $_.$p }.GetNewClosure() } }
}

$var | ForEach-Object { [pscustomobject] $_ } | Format-Table $format

This yields:

ID Name Age
-- ---- ---
1  abc  3  
2  def  4  

as desired: the output columns use the column labels specified in $properties while containing the correct values.

^{Note how I've removed unnecessary ; instances and replaced built-in aliases % and ft with the underlying cmdlet names for clarity. I've also assigned distinct age values to better demonstrate that the output is correct.}

---

Simpler solution, in this specific case:

To reference a property value as-is, without transformation, it is sufficient to use the name of the property as the Expression entry in the calculated property (column-formatting hashtable). In other words: you do not need a [scriptblock] instance containing an expression in this case ({ ... }), only a [string] value containing the property name.

Therefore, the following would have worked too:

# Use the property *name* as the 'Expression' entry's value.
$format += @{ Label = $p; Expression = $p }

Note that this approach happens to avoid the original problem, because $p is evaluated at the time of assignment, so the loop-iteration-specific values are captured.

---

^{[1] To reproduce: function foo { param([ValidateNotNull()] $bar) {}.GetNewClosure() }; foo fails when .GetNewClosure() is called, with error Exception calling "GetNewClosure" with "0" argument(s): "The attribute cannot be added because variable bar with value would no longer be valid."
That is, an attempt is made to include the unbound -bar parameter value - the $bar variable - in the closure, which apparently then defaults to $null, which violates its validation attribute.
Passing a valid -bar value makes the problem go away; e.g., foo -bar ''.
The rationale for considering this a bug: If the function itself treats $bar in the absence of a -bar parameter value as nonexistent, so should .GetNewClosure().}

Vanpelt

@PetSerAl: As an aside: It seems that the attribute is only a problem in parameters, not variables; e.g., [ValidateNotNull()] $o = ''. Any idea why? – Vanpelt

Variables can not violate validation attributes. Such violation would be caught on assignment [ValidateNotNull()] $o = $null, or when you try to add attribute $o = $null; (gv o).Attributes.Add([ValidateNotNull]::new()). But unused parameters would have default value even if it violate validation attributes. When GetNewClosure() capture such parameter, then it fail. It should be question about this on Stack Overflow already. – Cerveny

P.S. The source code for variable capture is in a method called CaptureLocals() in github.com/PowerShell/PowerShell/blob/… – Lessard

The closure bug is a tough little worm in 5.14393. I had it on the scriptblock argument of Register-ArgumentCompleter in a module. The module itself loaded and ran fine with the session or any CLI imports, but when I ran my create backup script that did Remove-Module and Import-Module -Force (to load changes), it produced this error on the "update" variable (presumably somewhere in Register-ArgumentCompleter). Could not fix it, so I just catched and suppressed it since argument completion failures are irrelevant for the backup, and I'm losing access to the system soon anyways. – Pathogenesis

Answer 2

While the whole approach seems misguided for the given example, just as an exercise in making it work the key is going to be controlling variable expansion at the right time. In your foreach loop, $_ is null ($_ is only valid in the pipeline). You need to wait until it gets to the Foreach-Object loop to try and evaluate it.

This seems to work with a minimum amount of refactoring:

$var =@(  @{id="1"; name="abc"; age="1"; },
      @{id="2"; name="def"; age="2"; } );
$properties = @("ID","Name","Age") ;
$format = @();
foreach ($p  in $properties)
{
    $format += @{label=$p ; Expression = [scriptblock]::create("`$`_.$p")} 
}
$var | % { [PSCustomObject] $_ } | ft $format

Creating the scriptblock from an expandable string will allow $p to expand for each property name. Escaping $_ will keep it as a literal in the string, until it's rendered as a scriptblock and then evaluated in the ForEach-Object loop.

Paisano

Answer 3

Accessing anything inside an Array of HashTables is going to be a bit finicky, but your variable expansion is corrected like this:

    $var =@(  @{id="1"; name="Sally"; age="11"; },
          @{id="2"; name="George"; age="12"; } );
$properties = "ID","Name","Age"
$format = @();

$Var | ForEach-Object{
    foreach ($p  in $properties){
        $format += @{
            $p = $($_.($p))
        }
    }
}

You needed another loop to be able to tie it to a specific item in your array. That being said, I think that going with an array of Objects would be a much cleaner approach - but I don't know what you're dealing with, exactly.

Roydd

We (now, possibly after you answered) know that the OP wants $format to contain calculated properties for later use with Format-Table, so this won't work. – Vanpelt