Promoting External Parameters

Posted on Updated on by ChrisCategories:Blog Entries, Coding

Dynamically Adding Parameters at Runtime

I am fully aware that I have been writing a lot of posts recently but, in my defense, work has recently left me unsupervised. This means I’ve had lots of time to spend working on fun PowerShell side projects, which naturally also means more blogs.

As part of my current pet project, I had been playing around with the NameIt module by Doug Finke and Chris Hunt. If you haven’t played with this module yet, it’s definitely worth giving a look if you ever find yourself needing to generate various types of random values, particularly if you need a lot of them. One of the elements of this module that I found to be particularly interesting though, was that it was essentially providing a framework for execution of various generators created by both the original authors and others. Some examples include things like random City, company, or person names, as well as simpler items like colors, numbers, dates, words, etc. The idea was that different people could figure out how to generate random instances of X item, and then ‘register’ it with the main module. Once registered, you could call a single cmdlet (Invoke-Generate), along with the name of the desired generator. You could provide values for parameters specific to the individual generator within the same string. The primary cmdlet would also enable you to specify how many instances of an generated item you wanted, and it would handle the call for you.

Unfortunately for me, the out of the box tool didn’t meet my needs, due to the disconnected nature of the generated items. I wanted to generate complex data sets that were associated to each other in specific ways. For example, rather than just person names, I wanted to be able to create identities that would include an email address and username created from their name parts. Further, I wanted to be able to use a common email domain for these personas based on a company, and to avoid duplications. While I could possibly have scripted the use of the NameIt tool to get some of what I needed, it would still have been necessary to augment a number of the existing generators. I did reach out to Doug about possibly contributing, but it turns out he’s very busy with other concerns right now, and he wasn’t sure when he might be able to review my proposed changes.

Long story short (ish), I decided that a new module with a more specific focus was in order. While the NameIt tool is fantastic for what it is, the current iteration is focused on random item names, not complex datasets. On top of this, there were some things I felt were fundamental challenges that I hoped to improve upon. One of those challenges was in the form of discoverability and execution of the generators themselves. As this was somewhat of a quickie module for the original authors, these were not the only challenges, but it is where my journey started, and is the focus of today’s blog.

Getting Started

Let’s start off by quantifying the challenges we are trying to solve first, which come down to the below items.

  1. Without opening the manifest, or browsing to the directory, how could I make it easier for users of my new module to ‘discover’ what generators were available
  2. Once the user figured out which generator they wanted to use, how could I make it easier and faster to call that generator, with or without generator specific parameters, in a uniform manner
  3. How could I reliably solve both of these problems without requiring advanced knowledge of the available generators and their options

What I essentially needed was a way to dynamically augment the available parameters at runtime, while still having at least some predictability. PowerShell does offer the ability to create parameters dynamically at runtime (source), but you still have to know what parameters you are needing to create.

Incorporating the Generators

The first step was to identify what generators were available, the calling commands, and then to gather the details on the available parameters and constraints. This mean that my new module would need a degree of proscribed structure to it, but not so much that would-be generator authors would be overly boxed in.

To solve this first challenge, I took a page from the ModuleBuild scaffolding framework module by Zachary Loeber. The gist of the module is that it provides a standardized template for developing your own modules in a more consistent manner. On the surface, not really anything you couldn’t technically get from Plaster, which this module actually uses under the covers. What makes this scaffold somewhat different however, is the concept of ‘plugin’ modules.

Normally, when you build a module that depends on other modules, you would simply specify those modules in the manifest. If you wanted to lock things to a specific version, you could list the dependencies in a hash table that included the version you were dependent upon. This would cause that version of the module to be automatically installed when your own module was.

The plugin approach however, uses a separate ‘plugins’ folder. Within this folder you create subfolders for each module you will use as a plugin. Within the module folder, you create load and unload PS1 files, and then copy in the entire source module folder. The load and unload files are used to import the module as a nested module any time the main module is imported. The key difference between a module dependency and a nested module is in the visibility of the cmdlets. For module dependencies, the module you depend on is installed normally, and the associated cmdlets are visible to the user directly. The nested module cmdlets are only visible to the primary module, similar to a private function that isn’t exported for public use. Outside of creating the loader, and copying in the module, nothing else is required to ‘register’ the module using the scaffold, as the scaffold already includes a process for dynamically finding and discovering all the load.ps1 files during import.

I leveraged a similar approach, though I created a separate root folder for Generators, and I skipped the ‘load.ps1’ aspect. Instead, to keep things secure, I configured a set of tests that run during the load sequence to verify that all of the generators comply with the minimal standards for my budding little framework. Among other items, these tests included the following elements:

  • Verify only a single public cmdlet was exported (the ‘generator’) with a simple call of the generator name (no verb, just noun)
  • Verify that no code execution, outside of ‘Export-ModuleMember’, is executed as part of the import
  • Verify that a ‘FriendlyName’ attribute exists in the Module manifest within PSData that matches the cmdlet name

Gathering the Data

Once I had a home for the generators, and a mechanism for reliably loading them during import of my module, the next step is to gather the data required for surfacing in my main cmdlet. To accomplish this feat, I needed to know what information was required. This included the following pieces:

  • Names of all of the available generator cmdlets in a readily consumed format
  • Detailed information on the available parameters for each cmdlet
    • Parameter Name
    • Parameter block designations (Position, Mandatory, ParameterSetName, etc)
    • Any validation blocks and their content (ValidateSet, ValidateRange, etc)
    • The object type, if specified (string, switch, int, etc)

The first step was to pull all of this data in, which I accomplished via a pre-load process that would use Abstract Syntax Tree (AST) to parse both the PSD1 and the PSM1. Parsing the PSD1 is super quick and easy, because it is essentially just a big hashtable that AST can parse into an appropriate object without much overhead. Parsing the PSM1 would potentially be more process intensive were it not for the framework standards I put in place. The PSM1 must contain the publicly accessible cmdlet definition, and the first thing that the loading process does is to find and bind only to this single function. The generator author can specify any number of additional functions, provided none are immediately called, per the current tests. Once I have an object representing the function, it is only one additional step to parse out all of the available ParameterAst instances. This AST block contains all of the details on each parameter, and I can filter out parameters I don’t need.

To keep everything organized and neat, I could have simply created an array of PSCustomObjects, but I instead opted to create my own custom object class. I elected to do this predominantly for the ability to craft my own custom methods. I needed to ensure that I could rapidly compare two objects in a reliable manner, which meant comparing the module name, the friendlyname, and the module version. This enables me to reliably determine if a given generator already exists, and to compare generators with the same name/friendlyname values to determine if one is a newer version. The other methods were for serialization of the data from a provided path to a folder that contained a PSD1 and PSM1 with the required elements. The pre-load process gets a list of the sub-folders in the Generators folder and uses the custom class to serialize the paths into GeneratorMetaInfo objects stored in a static array.

Surfacing the Generators

As one would hopefully expect, the primary static parameter for my main cmdlet is for specifying the name of the Generator to be called. This element was obviously somewhat tricky to achieve, because I needed to provide tab completion and validation. Since these details were dynamic in nature, I could not simply tag on a ValidateSet attribute to the parameter definition. I could theoretically have leveraged the script validation option, but this would not help with the discoverability.

To solve the problem, I once again leveraged my ‘pre-load’ process, which executes before I import my public cmdlets. As part of this process, which executes after the array containing the generator metadata, I created a dynamic ‘enum’ using the information from the metadata. The parameter on my Start cmdlet for the generator uses the Enum as the expected type, and I register the same Enum values as an argument completer. When creating the Enum, I use a here-string construct to build out what will be contained in the Enum, and then call Invoke-Expression on the here-string to initialize it. Creating the Enum this way provides two benefits; ability to dynamically set the content of the Enum, and the ability to regenerate the Enum in the event the values need to change. This was key, since an Enum cannot be modified after it is created, only overwritten.

Once I had this up an running, I was able to import the module and verify tab completion for my Generator parameter. At present, the module was structured more or less as shown below.

  • DataGen (main module folder)
    • DataGen.psd1
    • DataGen.psm1 -> calls preload.ps1 and postload.ps1
    • Other (supplemental items folder)
      • preload.ps1 -> loads class; serializes available Generators; creates enum for Generator names; initializes argument completers
      • postload.ps1 -> loads private and public functions
    • Generators (contains subfolders for each generator ‘module’ and main Load file)
    • Public (contains all public functions)
      • Start-DataGen.ps1 -> main module function, with Generator parameter that uses argument completer and enum for generator names
    • Private (contains all private functions)

Surfacing Generator-Specific Parameters

The next step in the plan was dynamically surfacing parameters for each specific generator. This required use of the Dynamic Parameter functionality offered by PowerShell, and it had the potential to be somewhat…messy.

Once upon a time, I had frequent complaints around the lack of sufficient guidance on effectively utilizing dynamic parameters, as well as the pain of coding them. Fortunately for me, the PowerShell community felt the same way, and some enterprising person or another created a function cmdlet called ‘New-DynamicParameter’. Doubly fortunate for me, a PS1 with this specific function is included as part of the ModuleBuild scaffold as a supplemental ‘build tool’. This function was subsequently copied into my Private functions folder with alacrity, as it not only makes the process of creating dynamic parameters far easier, it also makes for more readable code.

Now, just in case there are people out there who are not familiar with dynamic parameters that also elected not to read up on them using the link I provided above, I’ll provide a brief overview.

Dynamic Parameters are a feature that enables you to conditionally include parameters as part of your advanced functions, which means they aren’t available in standard functions or scripts. You can add them by placing a ‘DynamicParam {}’ block immediately following your standard ‘param ()’ block, and before your ‘begin {}’, ‘process {}’, and ‘end {}’ blocks. If you try to add the DynamicParam block without at least a ‘process’ block, VSCode will throw a fit, and the block will not execute.

You don’t want to confuse Dynamic Parameters with ParameterSets by the way. ParameterSets are used to group statically defined parameters together. While this does in a way limit what parameters you see, this is based only on what other parameters you are using on the command line. Dynamic Parameters, on the other hand, can change names, values, and just about everything else, based on the value you use for a parameter. Theoretically you could also leverage dynamic parameters to respond environmental factors as well, though I can’t think of any instances where I’ve seen this done.

In my case, I am triggering the Dynamic Parameters based on the value specified to the Generator parameter. As soon as a value is set, and a user goes to tab complete the next parameter, the DynamicParam block will fire to determine what other parameters need to be created. To provide these values, I filter the array of GeneratorMetaInfo objects to the specified generator. I then use the Generator specific parameter details to create the ‘new’ parameters using the code shown below.

DynamicParam {
    $GenMeta = ($Script:GeneratorMeta).Where({$_.FriendlyName -eq $Generator})
    $DynamicParams = @()
    foreach($cParam in $GenMeta.CommandParameters){
        $param = @{
            Name = "$($cParam.ParameterName)"
            Type = "$($cParam.ObjectType)"
        }

        foreach($Dec in $cParam.Decorations){
            $param.Add($($Dec.Decoration),$($Dec.Value))
        }

        $paramObj = [PSCustomObject]$param
        $DynamicParams += $paramObj
    }

    $DynamicParams | New-DynamicParameter
}
Expand

As you can see above, the first step is to filter the array of meta info to the specific generator. Since generators may have multiple parameters, I first create an empty array that will hold the PSCustomObjects required by the New-DynamicParameter cmdlet. After that, it is simply a matter of collecting the required values from the meta info and formatting them as a custom object. Once all the parameters are accounted for, I pass the entire array to the New-DynamicParameter cmdlet via the pipeline, and my parameters show up.

Wrapping Up

I would like to provide a caution here in terms of performance. While the dynamic parameter functionality is indeed powerful, there is a cost in overhead. Part of the reason I am creating the array of GeneratorMetaInfo objects is to offset this cost. If I tried to parse the PSM1 files to gather this data in real-time, as opposed to using ‘cached’ information, the impact on the user would be very noticeable. As it currently stands, unless someone is typing very fast, by the time the user hits dash and the tab key, the parameters show up without a truly noticeable pause. Down the road, it may be necessary to change my metadata storage type from an array to a generic list, in the event that there are dozens and dozens of Generators, but this isn’t a need I would expect at present.

Of course, there is another pseudo drawback to the Dynamic Parameters, which is that they don’t show up as part of the help information.

Normally, even if you don’t include help information, the help system will at least provide the limited parameter details it can find by parsing the cmdlets. Because our dynamic parameters don’t exist until runtime however, and then only in a triggered scenario, there is nothing for the help system to parse. This means that there would remain a bit of a discoverability challenge. Yes, the available generators themselves can now be discovered, as can the associated parameters, but what the parameters actually do remains a mystery to the user unless they look at the code.

Ultimately this is the other reason for using the ‘module’ type of approach, and setting up minimum standards that include providing Comment-Based Help. The tests I have now not only verify that this help exists, but also that every parameter is at least listed in said help, though this doesn’t directly solve the problem entirely. Since these are functionally ‘nested modules’, that means they are not directly visible to users so they can run ‘Get-Help’ on the Generator. Fortunately, this one was also pretty easy to solve.

The only reason that the help isn’t available to the user is because of scope. As mentioned previously, these modules are effectively ‘private’, which means they are only visible inside the module. The solution then, is to create a proxy cmdlet that is exported as a public function, which I did via a ‘Get-DGNHelp’ cmdlet. This cmdlet has most of the same parameter options as the standard Get-Help cmdlet, just with the cmdlet name being replaced with the Generator friendlyname. This information is then used to create a hashtable with the specified options that is splatted to the standard Get-Help cmdlet. Since the Get-DGNHelp cmdlet is executing within the module scope when it calls Get-Help, the help system is able to parse and display the Generator specific CBH without any problems. Even if that, for some reason, did not work, it is also possible to provide a path to Get-Help, which enables the help system to parse help details from modules that haven’t been loaded into the current scope. I felt that approach was less elegant however.

Anyway, that’s all for this time. I’m nearly at a point now that I can publish an early version of the module, so expect some more posts detailing the module further soon.

Until next time, stay fresh cheese bags! (I’m sure I’ll tire of that expression eventually…probably).

Chris