PowerShell Module Adventures: Part 2

Way back in part 1 of this series, I started working on trying to build out the basics of a web API to use as part of my new PowerShell module. If you haven’t checked that one out yet, you can find it here. In that article, I spent a lot of time trying to cover a bunch of concepts that, in reality, I barely understand myself. That said, I’ve committed to documenting my process of discovery and…ahem…adventure, as I work on getting this new module put together. For this article, I’ll finish up what I started on last time, and move a little further along the process to actually get something we can see up and running!

For this exercise, I’ved used the database from my older ADDeploy module, which is available on my Github page here. I migrated from from SQLite to MS SQL Server, which better accommodates the centralized data requirement, but otherwise left the structure unchanged. If starting from the basic Visual Studio Web API for EntityFrameworkCore, you’ll have a project folder structure that looks like the one shown in the screenshot of my Windows Terminal session shown below.

The Program and Startup files are the main files that will launch our application, though these files do not contain the main logic for the app. The appsettings and appsettings.Development JSON files contain some runtime information for when the app is compiled and run. There will also potentially be a third of these files, but with ‘Production’ instead of ‘Development’ in the title. The ADDeploy CSPROJ and SLN files contain values used by the IDE, which in my case is Visual Studio 2019. Finally, there are four sub-directories, where the majority of the actual API functionality will live. In this article, I’ll go over each bit in turn, making adjustments as required to the templated files along the way.

Program.cs

The Program.cs file from the project template is very simple initially, and contains only a single public class called ‘Program’ that does not return any values or objects. As outlined in the last article, the use of ‘public’ for the scope of the class indicates that it is visible to other classes and enables interaction from outside of the class by other classes or users.

The template creates two components within the single publicly defined class, which are Main and IHostBuilder. The only job of the first component is to start up the application, which in this case is a web service. The second component in the class adds configuration elements for the application by calling the startup class defined in the startup.cs. I will be making a few tweaks to this file, though there’s nothing that really needs to be changed here.

For my needs, and the purposes of this blog entry, I really wanted to ensure that I have good logging. This means logging that provides details that go to both the console and to a flat file, in case I need to troubleshoot later. While AspCore.NET 5 has come a long way from where it started, it still doesn’t appear to include flat file logging. More, I also want some flexibility in the logging framework so I have the option to add other log destinations down the road. I determined via a quick interweb search that one of the more popular logging solutions is ‘Serilog’. The main Serilog library is supposedly quite fast and highly flexible, and it offers a very large variety of other ‘sinks’ libraries to support almost any destination I might need. I also felth it important to be able to configure Serilog using an external file, such as appsettings. Fortunately this is something that Serilog supports, though it does require an additional module.

I may still have to make changes in the code at some point, but I’m trying to anticipate some of the common scenarios, such as writing to Splunk for example, that I might need. Careful planning for things like this can save painful recoding later. For the moment, I don’t need it, but by adding it up front, I avoid potentially painful retrofitting later.

If using Visual Studio, the various packages can be added using the Nuget package manager. The dotnet CLI command, or Nuget itself, may also be used to find and install required libraries as an alternative to Visual Studio.

After a library has been installed, you must add the associated ‘using’ statement for the relevant classes to your class file. Conversely, you should avoid adding using statements for libraries that are not required, as this may have adverse effects on things such as memory usage.

In the case of some libraries, it may also be necessary to create an instance of the services offered by the library within your class file. For example, you must first create an instance of a ‘logger’ for Serilog in order to actually log information with it. I obtained the below code snippet from the official Serilog wiki, which was added to the top of the ‘Main’ compont of the Program class.

            Log.Logger = new LoggerConfiguration()
                .Enrich.FromLogContext()
                .CreateLogger();

The first line creates a new LoggerConfiguration instance. Since there is no semi-colon at the end of the first line, the system knows that the next lines belong to this construct and my IDE automatically indented appropriately. The second line uses a supporting library to add more logging details than would come out of the default functionality. The final line creates a fully configured logger intsance for use in the application. In the rest of my Program.cs file, I can now use ‘Log’ as a shortcut to access my logger to write out information, debug messages, warnings, etc. I made a handful of additional adjustments to the Main component section to make use of this, as shown below.

        public static int Main(string[] args)
        {
            Log.Logger = new LoggerConfiguration()
                .Enrich.FromLogContext()
                .CreateLogger();

            try
            {
                Log.Information("Starting Web Host");
                CreateHostBuilder(args).Build().Run();
                return 0;
            }
            catch (Exception ex)
            {
                Log.Fatal(ex, "Host terminated unexpectedly.");
                return 1;
            }
            finally
            {
                Log.CloseAndFlush();
            }
        }

The first change I made is in the first line, where I dropped ‘void’ in favor of ‘int’. Now, instead of not returning anything at all, I can output an integer value that tells me if something went wrong in a succinct manner. Next, I wrapped the CreateHostBuilder line in a try/catch/finally block. If you’ve spent much time writing advanced PowerShell functions, you’re probably already familiar with this setup. One difference in .NET however, is that I believe you have to always include all three blocks, but don’t quote me on that.

I configured the first line of the ‘Try’ block to log an informational message using the new logger instance. I also configured the ‘Try’ block to return a numerical value of 0 if successfully executed. In the catch block, I set it up to catch any exception thrown, then pass the exception details and a message to my logger before returning a numerical value of ‘1’ to indicate an issue. In the finally block, I’m closing the logger instance and flushing everything to disk.

We have one more bit to modify here, which is the IHostBuilder section. We need to tell it to use Serilog when it launches the startup file, as well as to finish configuring Serilog. As mentioned previously, we want to pull our configuration from our appsettings file, so this is where we tell Serilog to read in the configuration from the file as part of calling the startup process. I definitely do not understand all of what is going on in this component just yet, so I took the code directly from the Serilog examples, resulting in the below snippet.

        public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .UseSerilog((hosting, loggerconfig) =>
                    loggerconfig.ReadFrom.Configuration(hosting.Configuration))
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.UseStartup<Startup>();
                });

Startup.cs

Now we move on to the startup.cs file. One of the other key things I want to ensure I support with this API is compliance with leading industry practices. One of those, that may actually make our lives easier down the road when we build the PowerShell module portion, is support for Swagger. If you aren’t familiar with it, it basically offers a framework that describes the API for consumers, and makes it easier to document in a standardized manner. There are lots of different libraries to add this support, but I went with Swashbuckle. I added the required additional libraries, which included the ‘Swashbuckle.AspNetCore’ and ‘Microsoft.OpenApi.Models’.

Add in all the required ‘using’ statements…admittedly I probably added too many that weren’t strictly necessary while trying to figure things out. In my case, I made a bunch of changes to this file from the default template. Some of these were in relation to examples for implementing the various elements, and some I added from other samples and conventions I noted while troubleshooting, because I thought they made good sense. The result is that I have three components defined in this file; Startup for the IWebHostEnvironment, ConfigureServices for IServiceCollection, and Configure for IApplicationBuilder and the DB connection.

The first section is where we pull in our settings from our appsettings files. We use the information from these files to configure elements of the environment, and to build the webhost service, as shown in the below code snippet.

        public Startup(IWebHostEnvironment env)
        {

            var builder = new ConfigurationBuilder()
                .SetBasePath(env.ContentRootPath)
                .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
                .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
                .AddEnvironmentVariables();
            Configuration = builder.Build();

        }

        public IConfiguration Configuration { get; }

Based on my understanding, the various dependencies are pulled in from other files or classes using dependency injection via the various builders. This type of approach enables projects to be logically separated out, or be deployed as discreet microservices. The sharp-eyed among you will no doubt notice the add-on line that actually sits outside the Startup component definition. Everything inside Startup indicates where to get all of the configuration details and builds it out, then this last line imports the built configuration for use.

The ‘ConfigureServices’ component is used to spin up, and sometimes to configure, the services that will be used within the application. This includes services we get from incorporated libraries, such as Swagger, as well as any services we define within our application. I suspect I’ll have some of my own at some point, but for now the only one I specifically add of my own is the database context, as shown below.

        public void ConfigureServices(IServiceCollection services)
        {
            var connection = Configuration.GetConnectionString("ADDeployDatabase");
            services.AddDbContextPool<ApplicationDbContext>(options =>
            options.UseSqlServer();
            services.AddAutoMapper(typeof(Startup));
            services.AddSwaggerGen(setup =>
            {
                setup.SwaggerDoc("v1", new OpenApiInfo
                {
                    Title = "ADDeploy API",
                    Version = "v1",
                    Description = "API for interacting with the core DB back end"
                });

            });
            services.AddControllers();

            services.AddMvcCore()
                .AddApiExplorer();
        }

The first line gets the connection string from our appsettings file (I promise, I’m getting to it), which in this case is called ‘ADDeployDatabase’. The second line adds the database using a conext pool, which I’m using over the standard dbcontext because I may need to have multiple threads access the DB as different users perform different tasks, of if running multi-threaded deployment. An individual connection is apparently limited in scope and usage until the process calling the connection releases it. From what I understand, using a pool will allow multiple additional contexts to be spun up and down as needed. The third line tells the application to use MS SQL server to connect to the database. All of these are part of the EntityFrameworkCore library.

The next line adds in automapper, which simplifies wiring up different parts of our model to each other. I don’t have anything that specifically needs this yet, but as I add functionality, I will need it, so I went ahead and included it.

The next little section of settings adds the Swagger generator, which is what looks at our structure and documents it for consumption via the Swagger-UI component. Since Swagger is an open specification, it uses OpenAPIDocument standards to put some pieces in place. We theoretically don’t have to add this bit to get the basic functionality going, but defining it allows us to control the presentation of various elements, such as the app title and description, the API version, copyright details, and contact/source links.

The next line adds our controllers, which will reside in our Controllers folder. So far as I know, there’s nothing specific that has to be set for this one, and it just loads any controllers we have defined. The next two lines go together, and configure MvcCore to enable the ApiExplorer which, according to the documentation I looked at for Swagger, is required base functionality. I think the template actually adds just the regular Mvc, but since we are looking to make this fully cross-platform, I switched it out for MvcCore.

        public void Configure(IApplicationBuilder app, ApplicationDbContext dbContext)
        {

            app.UseStaticFiles();
            app.UseRouting();
            
            app.UseSerilogRequestLogging();
            app.UseSwagger();
            app.UseSwaggerUI(setup =>
            {
                setup.RoutePrefix = string.Empty;

                setup.SwaggerEndpoint(
                    url: "/swagger/v1/swagger.json",
                    name: "ADDeploy API");
            });

            app.UseEndpoints(endpoints =>
            {
                endpoints.MapControllers();
            });
        }

The last section adds the final bits for our setup. The first line allows us to use static files, and I added it to support future capabilities I plan to add. UseRouting allows requests to be properly routed to your controllers and the like. The next two add in the Serilog and and Swagger functionality, while the section after that configures some of the Swagger-UI elements. The RoutePrefix was part of several examples and lets the Swagger interface be accessible from the root. The last section actually maps the controllers as API endpoints. I didn’t initially include this and I lost half a day trying to figure out why the app would load the Swagger interface, and show all the correct pieces, but the APIs just kept returning 404.

Appsettings.json

See, I told you I’d get here…don’t look at me like that…anyway.

These files are used to configure various settings for use within our source, and to differentiate them between environments without having to change the code. The main appsettings file should contain any settings that are common to all environments, while the Development and Production variants naturally hold values that are specific to those environments. Which environment you are accessing is defined in the launchSettings.json file that lives in our Properties folder, in the form of an environment variable definition. For this setup, I have the Serilog configuration and host restrictions defined within the main file, and I have my DB connection string defined in the Development copy of the file. The snippet of the main file is shown below.

{
  "Serilog": {
    "Using": [ "Serilog.Sinks.Console", "Serilog.Sinks.File"],
    "LevelSwitches": { "controlSwitch": "Verbose" },
    "MinimumLevel": {
      "Default": "Debug",
      "Override": {
        "Microsoft": "Warning"
      }
    },
    "WriteTo": [
      {
        "Name": "Console"
      },
      {
        "Name": "File",
        "Args": {
          "path": "EADDLog.txt",
          "rollingInterval": "Day",
          "rollOnFileSizeLimit": "true",
          "shared": "true"
        }
      }
    ]
  },
  "AllowedHosts": "*"
}

Now, you’re probably thinking to yourself ‘Why all the fuss? Why a separate section?’. This is for this particular paragraph right here. When connecting to a DB, you obviously need to use an identity, which you should absolutely never put into clear text in your code. Ideally you will pull this from a vault service/solution, such as Azure KeyVault or CyberArk. That said, it can be a pain to get that set up while in early stages of development, so you can place the account and password details into the Development copy of the appsettings file, which is where mine presently is (and why the snippet isn’t included). In a later article, I’ll cover the option I end up going with, but this article is about getting an API up and accessible.

Adding a Controller

The API functionality itself, in its simpleist form, comes from controllers that define the web methods and logic used to interact with our database using the EntityFramework. The easiest way to do this in Visual Studio, is to add a new scaffolded item by right-clicking on the Controllers folder and selecting those options as shown below.

In Visual Studio, this will then provide a list of templates you can select from. We’re using the Entity Framework, so you will select ‘API Controller with actions, using Entity Framework’ as shown below.

If you followed along with the first article, and are using Visual Studio, you should have an existing DB connection. Using the scaffolded approach makes it easier to generate the code for the controller pretty automatically, along with predefined CRUD (Create, Read, Update, Delete) web actions. In the final window, you should be able to select a class using a dropdown selection, for which the dialog box is shown below.

Clicking the dropdown for the Model class should provide a list of all of the classes defined in the solution. Selecting the AP_Objects class will allow us to generate a new API controller for interacting with that particular table. The Data Context Class should already be selected, but can be selected from a list if needed. The controller name gets set automatically, though you can override it if desired. Finally, click the add button to generate the new file. A bunch of build type tasks will execute, as it appears as though a partial build gets completed to enable all the pieces to be pulled together to define the controller.

AP_ObjectsController

At this point, we could theoretically build and access our new controller, but we still need to make a few minor edits if we want to take true advantage of Swagger, as well as adding some helpful options for users. In theory we would want to add code to integrate logging via Serilog as well, but I’ll save that for another time. I won’t cover all of the code for this one, just some of the key pieces for demonstration purposes.

The first thing we’ll do is add some helpful functionality to the HttpGet option for our controller. It’s reasonable to assume that people consuming our new API may wish to filter the results, rather than just returning everything all the time. To do this, we’ll need to add some parameters, as well as the logic to process them, which I’ve done already in the code snippet shown below.

        [HttpGet]
        [ProducesResponseType(StatusCodes.Status200OK)]
        [ProducesResponseType(StatusCodes.Status404NotFound)]
        public async Task<ActionResult<IEnumerable<coreObjectType>>> GetAP_Objects(bool? isEnabled, bool? hasAcls)
        {
            var AP_Objects = _context.AP_Objects.AsQueryable();

            if (isEnabled != null)
            {
                AP_Objects = _context.AP_Objects.Where(i => i.typeIsEnabled == true);

                if (hasAcls != null)
                {
                    AP_Objects = AP_Objects.Where(i => i.typeHasADACLs == true);
                }
            }

            if (hasAcls != null && isEnabled == null)
            {
                AP_Objects = _context.AP_Objects.Where(i => i.typeHasADACLs == true);
            }

            return await AP_Objects.ToListAsync();
        }
        [HttpGet("{isEnabled},{hasAcls}")]
        [ProducesResponseType(StatusCodes.Status200OK)]
        [ProducesResponseType(StatusCodes.Status404NotFound)]
        public async Task<ActionResult<IEnumerable<coreObjectType>>> GetAP_Objects(bool? isEnabled, bool? hasAcls)
        {
            var AP_Objects = _context.AP_Objects.AsQueryable();

            if (isEnabled != null)
            {
                AP_Objects = _context.AP_Objects.Where(i => i.OBJ_enabled == true);

                if (hasAcls != null)
                {
                    AP_Objects = AP_Objects.Where(i => i.OBJ_AssignACLs == true);
                }
            }

            if (hasAcls != null && isEnabled == null)
            {
                AP_Objects = _context.AP_Objects.Where(i => i.OBJ_AssignACLs == true);
            }

            return await AP_Objects.ToListAsync();
        }
        [HttpGet]
        [ProducesResponseType(StatusCodes.Status200OK)]
        [ProducesResponseType(StatusCodes.Status404NotFound)]
        public async Task<ActionResult<IEnumerable<coreObjectType>>> GetAP_Objects(bool? isEnabled, bool? hasAcls)
        {
            var AP_Objects = _context.AP_Objects.AsQueryable();

            if (isEnabled != null)
            {
                AP_Objects = _context.AP_Objects.Where(i => i.typeIsEnabled == true);

                if (hasAcls != null)
                {
                    AP_Objects = AP_Objects.Where(i => i.typeHasADACLs == true);
                }
            }

            if (hasAcls != null && isEnabled == null)
            {
                AP_Objects = _context.AP_Objects.Where(i => i.typeHasADACLs == true);
            }

            return await AP_Objects.ToListAsync();
        }

First, I’ll point out two lines directly underneath the HttpGet tag, which indicate the response types that this particular method could return. These are Http response codes, and I don’t know to what level I can modify these. For this use case right now, I’m just going to put in two possibilities; either we get a response (Status200OK), or we don’t (Status404NotFound).

We start by adding the values inside of the parenthesis of ‘GetAP_Objects()’. We know that items could be enabled or disabled (OBJ_enabled column), as well as whether or not there are ACLs to assign (OBJ_AssignACLs column). In the current version of the module, I have different points at which I make these specific queries separately, so I need to accommodate both use cases individually. Further, while these show up in the SQLite DB as a 1 or a 0 value, it makes sense to convert these into proper boolean values when we make the jump to SQL Server.

By adding a question mark after the object type (bool?), we indicate that a value may be provided. The second value is how we expect to refer to this parameter value within the code. This can be a completely arbitrary value, but keep in mind that this is exposed via the URL to users, and we do have some conventions we need to adhere to. For example, the first character should be capitalized (yeah, I blew that one already lol), and we should avoid spaces or special characters that might interfere with accessing the API via the browser. Obviously some are more conventions, while others are less flexible.

Now we obviously also have to have the logic to actually filter the results using our new parameters. In my case, I used two If constructs, with the first one having a second nested If construct. I honestly probably need several more to account for the various possible permutations, but we’ll keep it simple for now. Essentially, we are only attempting to filter if the value of our parameter is not null (empty). If it has a value, then we call our database context using the Model definition, and then look for the specific values we want to include or exclude within the results. Finally, we wait on all the results to be processed before creating a generic list that is returned. This list is presented as a serialized JSON object containing the results.

The only other modification we’re making for this particular article is adding in Swagger support. Much like PowerShell, Swagger uses keywords that are found in comments to generate documentation. There are a number of libraries that extend what keywords are recognized and processed by Swagger, and you can always write your own of course too if you have specific needs. I added a few in my case, not all of which will be covered today. A code snippet for the component we have been working on here is shown below.

        /// <summary>
        ///  Gets object model types
        /// </summary>
        ///  <remarks>
        ///  Returns only enabled object model types by default 
        /// </remarks>
        /// <param name="isEnabled"></param>
        /// <param name="hasAcls"></param>
        /// <returns>List of available object model types</returns>

The text between the two summary tags is the short title that indicates what the method does. We can also add a ‘description’ block if we need to provide more details. Remarks are useful for showing hints or quick notes and information. The parameter names should also be captured, each in their own block. The Returns indicates what kind of items or response is returned. You can also add in response codes, with further details on what would cause each code to be returned.

Wrapping Up

At this point, you should be able to start the app and test accessing your new API. By going to ‘http://localhost:{port}/index.html’, which happens automatically when you run the app, and see your nice pretty Swagger UI. The UI shows your API description, your endpoints, and your controllers, as well as letting you test out your controllers in similar fashion to using something like Postman, as shown in the two screenshots below.

Obviously Swagger is generating the index.html file at this stage, as we haven’t defined any pages or web site components yet. That will be done in a separate project down the road, and that UI will ostensibly consume this API to interact with our database. This ensures that each element of our application can be maintained and scaled as completely independent microservices. They could, of course, all be hosted on a single server, but in my case I wanted portability and cloud support, so I’m setting everything up for Docker Containers.

I’m not really covering Docker here, but if you have Docker Desktop installed and configured on your dev system, you can add Docker functionality to your app pretty quickly within Visual Studio via a right-click on the project, and then selecting ‘Add Docker Support’. This run a brief wizard asking a few details, such as what type of OS, and then will proceed to download and configure the required image. The helper will also add a ‘Dockerfile’ to the root of your project, as well as adding a new entry to your launchSettings.json file in the Properties folder. The values in this file impact what you see at the top of Visual Studio for launching and testing your application. What ports are used, as well as the Environment Variables and target types, are all defined in the launchSettings file.

That’s all for this time my friends. I’m planning to build out the rest of my controllers and functionality, then start working on the focus of my next planned blog covering securing the API. At present, I’m thinking this will involve ‘super users’ that will be permitted to register endpoints, which will in turn have system specific security keys that work sort of like an IoT scenario, but we’ll see how it plays out when I actually start working on it. Still lots to learn!!