Andrew Pearce

Speaking at the PowerShell and DevOps Global Summit 2019

Fri, 12 Apr 2019 21:00:00 +0000

Disclaimer: The views expressed on here are mine alone and not (necessarily) those of my employer (AWS).

In a few weeks I will be attending the PowerShell and DevOps Global Summit 2019, my fourth PowerShell Summit in as many years since joining Amazon in late 2015. The biggest difference this year is I’m attending as a speaker, presenting on two of my favorite topics, Serverless computing with PowerShell in AWS Lambda and how the Kinesis Agent for Microsoft Windows can help make your logs more useful.

The PowerShell language support in AWS Lambda is one of my favorite topics. It’s no secret that I’m a huge fan of Serverless computing and the operational benefits it can provide to engineering teams. In September 2018, when AWS Lambda launched support for PowerShell Core, it opened the doors for the PowerShell community to leverage the benefits of Serverless computing without switching to another language, and in my opinion, it is a big deal for the PowerShell community. AWS Lambda lets you run code without provisioning or managing servers. It is highly available by default, has automatic scaling and charges you only when your code is executed.

The benefits of AWS Lambda have already existed for developers leveraging other runtimes, such as Python, .NET Core or Node.js, however a lot of developers in the PowerShell community are keen to stay with PowerShell where they can, which can make a lot of sense. For example, at one end of the scale are the developers who want to choose the best language for every situation, and at the other end are those who only ever want to learn and develop in a single language. However there’s often an even more important decision that engineering teams need to make, one of operational excellence. At 2am when your on-call gets woken up by the annoying, horrifying sound of their pager, what language do they want to be troubleshooting? The language they’re the most familiar with? Or the language they only just started to learn because “it was the best language for the situation”? Don’t get me wrong, I’m all for engineers learning as many languages as they can as I believe it makes them a stronger engineer, however when you’re building that next production service, language familiarity for the engineering team has to be part of the decision.

One of the reasons I’m such a big fan of leveraging Serverless, is it’s abstractions. With Serverless, engineering teams can go about choosing the best language for them, their service, and their operational excellence. Customers likely don’t care what language a service is developed in, and they likely don’t care about the engineering team being paged at 2am page for a service outage; they just want the service back online as soon as possible. For a lot of administrators, engineers and developers attending the PowerShell and DevOps Global Summit, PowerShell is likely going to be one of the best operational decisions.

To circle back, having the PowerShell Core language support available in AWS Lambda is a game changer. It opens the door to all the capabilities of AWS and Serverless for PowerShell developers.

Switching focus a little to the Kinesis Agent for Microsoft Windows. I think this agent is one of the best ways for Windows focused engineering teams to start using their logs to deliver service improvements. The plugins included in the default installation allow you to stream your logs and performance counters to services such as Amazon Kinesis or Amazon CloudWatch. Once your data is streamed to these services, you can do almost anything you can think of… stream them into Amazon Elasticsearch, dump them to S3 for both archival and SQL-like queries with Amazon Athena or process them in near-realtime with PowerShell in AWS Lambda. Think about that last one for a second… you can process your logs in a centralized location, with a service that scales automatically, with code you develop, to do almost anything you want. That is a very powerful capability.

Centralizing your logs is a key requirement as you move towards disposable systems. It’s great that when you’re woken up in the middle of the night you can forget about everything and simply terminate the EC2 Instance, allowing it to rebuild and get your service back to an operational state. The downside is that without centralized logs you’ll also lose the data required to help perform a root cause analysis so you can prevent the issue from re-occurring. The Kinesis Agent for Microsoft Windows can help get you out of that situation, while also allowing you to take action from your logs and leveraging them in ways you may never have thought possible.

So there you have it, a little insight into why I wanted to present on these two topics, and why I’m excited to share some knowledge about these services and their capabilities. It’s likely to be a very different experience for me at this years PowerShell and DevOps Global Summit, but it’s one that I’m definitely looking forward to. It will be great to catch up with the community members I’ve gotten to know over the years, and to meet those I haven’t. See you all there in a few short weeks!

Cross-platform PowerShell, Unit Testing and Automatic Variables

Tue, 10 Jul 2018 21:00:00 +0000

Now that PowerShell Core is gaining traction in the community, it’s only a matter of time before cross-platform PowerShell Module development is the standard. Let’s face it, it’s a no-brainer, why would you exclude potential customers!? However we need to remember that PowerShell Core is not equal to or the same as Windows PowerShell. They are different products, with different cmdlets, namespaces and functionality, and I have no doubt you’ll end up with platform specific logic paths in your code (at least in the near term).

In addition to this, PowerShell developers are increasingly using CI/CD tooling to automate their unit tests on multiple Operating Systems; whether it’s with AppVeyor and TravisCI, or perhaps with AWS CodeBuild, which now supports Linux and Windows build containers.

This blog will provide insight and guidance into a common issue PowerShell developers are likely to face when they start their cross-platform development, specifically in regards to PowerShell automatic variables and Pester unit tests.

A little background

With the release of Nano Server and Windows IOT, ‘Windows PowerShell Core’ was released, running on .NET Core as those platforms didn’t have the full .NET stack available. To help developers identify which platform their code was executing on, the Constant variable ‘$PSEdition’ was introduced. It resolves to ‘Desktop’ on Windows Server and Windows client platforms, and ‘Core’ on Nano Server and Windows IOT.

When PowerShell Core was released cross-platform, a similar issue was introduced; PowerShell developers still needed a way to identify the underlying platform. The automatic variables ‘$IsCoreCLR’, ‘$IsLinux’, ‘$IsMacOS’ and ‘$IsWindows’ were introduced to resolve the issue. ‘$IsCoreCLR’ is true on all PowerShell Core platforms, and it’s pretty obvious when the other variables would evaluate to true or false.

PowerShell developers have five automatic variables to identify the underlying platform for cross-platform PowerShell development. However, if you’re in a mature environment or even just starting out writing Pester unit tests, rest assured that when you use these automatic variables directly in your code you will very likely run into unit testing troubles.

Note: The PowerShell automatic variables documentation does not include details about these recently introduced variables yet.

Automatic variables and unit testing don’t mix

I’m going to walk through what most PowerShell developers do when they start developing cross-platform; the goal is to write a function that returns ‘Windows’ or ‘Not Windows’. I am going to ignore both Nano Server and Windows IOT for this example as they aren’t really end-user, customer platforms in most cases.

I’ll start with a simple function and unit tests to demonstrate the logic works. I also note that ‘$IsWindows’ is now an option so I’ll use that.

# The sample code
function Invoke-Foo
{
    if ($IsWindows)
    {
        'Windows'
    }
    else
    {
        'Not Windows'
    }
}

Describe 'Invoke-Foo' {
    It 'Runs the Windows logic path' {
        $IsWindows = $true

        $assertion = Invoke-Foo
        $assertion | Should -BeExactly 'Windows'
    }

    It 'Runs the non-Windows logic path' {
        $IsWindows = $false

        $assertion = Invoke-Foo
        $assertion | Should -BeExactly 'Not Windows'
    }
}

Code written, lets run our tests:

Describing Invoke-Foo
  [+] Runs the Windows logic path 29ms
  [+] Runs the non-Windows logic path 18ms

Perfect! The tests passed; our code works. Now lets go try out our new function:

# Invocation of 'Invoke-Foo' on Windows PowerShell
PS>Invoke-Foo
Not Windows

# Invocation of 'Invoke-Foo' on PowerShell Core on Windows
PS>Invoke-Foo
Windows

Well, that didn’t work as planned did it. Clearly something needs to change. Now we remember that ‘$IsWindows’ is only valid for PowerShell Core, so we’ll add ‘$PSEdition’ to our evaluation to fix the function on Windows PowerShell:

# PSEdition check added
function Invoke-Foo
{
    if ($IsWindows -or $PSEdition -eq 'Desktop')
    {
        'Windows'
    }
    else
    {
        'Not Windows'
    }
}

# Setting PSEdition to 'Core' so we can test the non-Windows path
Describe 'Invoke-Foo' {
    It 'Runs the Windows logic path' {
        $IsWindows = $true

        $assertion = Invoke-Foo
        $assertion | Should -BeExactly 'Windows'
    }

    It 'Runs the non-Windows logic path' {
        $IsWindows = $false
        $PSEdition = 'Core'

        $assertion = Invoke-Foo
        $assertion | Should -BeExactly 'Not Windows'
    }
}

Let’s run our tests and invoke the function to validate:

# Pester Test
Describing Invoke-Foo
  [+] Runs the Windows logic path 32ms
  [-] Runs the non-Windows logic path 14ms
    SessionStateUnauthorizedAccessException: Cannot overwrite variable PSEdition because it is read-only or constant.

# Invocation of 'Invoke-Foo' on Windows PowerShell
PS>Invoke-Foo
Windows

# Invocation of 'Invoke-Foo' on PowerShell Core on Windows
PS>Invoke-Foo
Windows

We now have a working function, but our unit tests fail so we can’t prove it. Why is the test failing? Because we can’t modify the automatic variable ‘$PSEdition’, which is a Constant. A Constant variable as outlined in the ‘Option’ parameter of Set-Variable:

The acceptable values for this parameter are:

None. Sets no options. ("None" is the default.)
ReadOnly. Can be deleted. Cannot be not changed, except by using the Force parameter.
Constant. Cannot be deleted or changed. Constant is valid only when you are creating a variable. You cannot change the options of an existing variable to Constant.
Private. The variable is available only in the current scope.
AllScope. The variable is copied to any new scopes that are created.

That Constant ‘$PSEdition’ variable cannot be modified, which makes is essentially useless in our unit test.

Some of you are probably seeing the underlying problem and have already figured out the fix, however I’ll continue walking through the process as I believe this is a very common workflow when PowerShell developers start developing for cross-platform. Additionally, the next step for a lot of developers is to drop the unit tests because “the code works!”. This is very clearly the wrong answer.

So how is this resolved?

The answer is actually quite simple! Firstly, do not drop the unit tests. These are critical to operational maturity, your personal development and is simply good development practice. To resolve this, we are going to use another function instead of directly calling the automatic variables. The new function will perform the exact same logic we used initially, except we can mock the function with whatever we want, which means we can control the logic and validate it.

# A function to evaluate the platform
function IsWindows
{
    if ($IsWindows -or $PSEdition -eq 'Desktop')
    {
        $true
    }
    else
    {
        $false
    }
}

# Our user facing public function
function Invoke-Foo
{
    if (IsWindows)
    {
        'Windows'
    }
    else
    {
        'Not Windows'
    }
}

# Unit tests now mock the private function
Describe 'Invoke-Foo' {
    It 'Runs the Windows logic path' {
        Mock IsWindows {$true}

        $assertion = Invoke-Foo
        $assertion | Should -BeExactly 'Windows'
    }

    It 'Runs the non-Windows logic path' {
        Mock IsWindows {$false}

        $assertion = Invoke-Foo
        $assertion | Should -BeExactly 'Not Windows'
    }
}

This change makes the code just as easy to read, and if used in a PowerShell Module, the ‘IsWindows’ function would be made private. To demonstrate that this change has not broken the function while allowing us to perform our unit testing validation:

#############
# Pester
#############

# Unit tests on Windows PowerShell
Describing Invoke-Foo
  [+] Runs the Windows logic path 63ms
  [+] Runs the non-Windows logic path 48ms

# Unit tests on PowerShell Core on Windows
Describing Invoke-Foo
  [+] Runs the Windows logic path 136ms
  [+] Runs the non-Windows logic path 27ms

# Unit tests in PowerShell Core in Bash WSL
Describing Invoke-Foo
  [+] Runs the Windows logic path 606ms
  [+] Runs the non-Windows logic path 123ms

# Unit tests in PowerShell Core on MacOS
Describing Invoke-Foo
  [+] Runs the Windows logic path 653ms
  [+] Runs the non-Windows logic path 123ms

#############
# Invocation
#############

# Windows PowerShell
PS>Invoke-Foo
Windows

# PowerShell Core on Windows
PS>Invoke-Foo
Windows

# PowerShell Core in Bash WSL
PS /> Invoke-Foo
Not Windows

# PowerShell Core on MacOS
PS /> Invoke-Foo
Not Windows

Wrap up

Automatic variables are extremely useful and provide the required detail to identify the platform your PowerShell code is being executed on. However, remember that using them directly can cause you headaches. Simply swap them out for private helper functions and off you go.

Happy PowerShell Developing!

Announcing 'Convert', a PowerShell Module for common object conversions.

Thu, 31 May 2018 22:15:00 +0000

I recently released a module to the PowerShell Gallery, called simply, Convert. Convert provides PowerShell Functions for common object conversions. The initial release enables conversions to and from base64 encoded strings, clixml, and MemoryStream objects with optional gzip compression. It wraps the appropriate .NET methods with easy to use PowerShell Functions. The module is cross-platform ready and comes with a minimum requirement of PowerShell 5, which I believe is fair game for 2018.

Installation is the standard process from the PowerShell Gallery:

Install-Module -Name 'Convert' -Scope 'CurrentUser'

I hear some of you asking, “what’s the benefit of a module that simply wraps standard .NET calls?” My answer, simplicity. I often perform object conversions between these object types, and I find myself forgetting the underlying .NET Namespace, or the conversion process. Instead, I want to use PowerShell to make this easier, and with Convert, hopefully to help other people while I’m at it.

How can Convert help you?

Do you ever want, or need to leverage clixml for handing objects around? Export-Clixml will handle the conversion, except it requires writing to disk, something I won’t do unless I absolutely must. You can use the System.Management.Automation.PSSerializer Namespace to perform the conversion in-memory, but I always forget the Namespace. With Convert, I only need to remember ConvertTo-Clixml and ConvertFrom-Clixml, both of which follow the standard “ConvertTo” and “ConvertFrom” naming convention for object conversions.

PS C:\> 'This is an object' | ConvertTo-Clixml

<Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04">
  <S>This is an object</S>
</Objs>

What about some other examples where Convert may help?

Passing objects between disconnected systems with AWS Systems Manager, Parameter Store

Let’s say you have a process that invokes some PowerShell on a remote system, and you want to save a PowerShell Object somewhere for future retrieval, through a log file, database entry, or some other form of storage. Normally you would need to hunt down the code to do so, or write to the filesystem. Here’s an example using Convert and the AWS Systems Manager, Parameter Store to make this easy.

# On the remote system, save the 'Convert' module specification to AWS Parameter Store:
$base64 = Get-Module -Name 'Convert' | ConvertTo-Clixml | ConvertTo-Base64 -Compress
Write-SSMParameter -Name 'ConvertModuleSpec' -Value $base64 -Type String

# On a different system, retrieve the object back in its original form:
(Get-SSMParameterValue -Name 'ConvertModuleSpec').Parameters.Value | ConvertTo-String -Decompress | ConvertFrom-Clixml | Format-Table -AutoSize

ModuleType Version  Name    ExportedCommands
---------- -------  ----    ----------------
Script     0.1.1.13 convert {ConvertTo-String, ConvertFrom-MemoryStreamToString, ConvertTo-Clixml, ConvertFrom-MemoryStreamToBase64...}

Convert the output of an AWS Lambda Function

The Invoke-LMFunction PowerShell cmdlet (from AWSPowerShell or AWSPowerShell.NetCore) can be used to invoke an AWS Lambda Function. The invocation itself is easy, however the output is returned as a System.IO.MemoryStream object. Converting a MemoryStream to a String is a common process for Software Developers, but most PowerShell Developers won’t know this off the top of their head. With Convert this is easy; simply pipe the Payload property to ConvertTo-String.

$json = ConvertTo-Json -Compress -InputObject @{ 'name' = 'Andrew' }
(Invoke-LMFunction -FunctionName 'GetHelloWorld' -Payload $json).Payload | ConvertTo-String

{"Hello": "Andrew"}

To take this one step further, pipe the output to ConvertFrom-Json to create a PowerShell object to work with.

(Invoke-LMFunction -FunctionName 'GetHelloWorld' -Payload $json).Payload | ConvertTo-String | ConvertFrom-Json

Hello
-----
Andrew

Stream a PowerShell object to Amazon Simple Storage Service (S3)

The Write-S3Object PowerShell cmdlet (also from AWSPowerShell or AWSPowerShell.NetCore) is used to upload an object to S3. Common use-cases are to upload files and folders from your local filesystem, however the cmdlet also supports uploading data from a MemoryStream. Convert enables this with in-memory conversions from any object, to a MemoryStream object. Let’s see how.

$memoryStream = Get-Module -Name 'Convert' | ConvertTo-Clixml | ConvertTo-MemoryStream
Write-S3Object -BucketName 'mys3bucket' -Key 'ConvertModuleSpec.clixml' -Stream $memoryStream

So there you have it, I hope I’ve outlined why I released Convert to the PowerShell Gallery. It is available for installation today, and you can view the source code on GitHub.