About now, it is time for some housekeeping items I think. We’ve been trudging along in this series working through a logical set of steps. In the last article, I talked about a couple of pretty important helpers; DTOs and AutoMapper. In addition to this, the earliest articles in the series were focused a lot on another helper, in the form of Entity Framework itself.
Helpers such as these make coding something as daunting as a back-end for a PowerShell module a lot more achievable, at least for a non-programmer like myself. This is because it takes a lot of difficult pieces off of my plate entirely. For example, instead of having to learn how to build and configure a database, create connections to it, and write native T-SQL queries, EFCore enables me to define a few classes and some LINQ, and I’m off the to races. AutoMapper saves me tons of repetitive code by providing more dynamic access to DTOs.
These things are, no doubt, awesome and welcome additions to the arsenal, but what happens when things go wrong? The simple fact of the matter, is that each of these helpers comes with their own unique potential issues and quirks, and it is up to each developer to figure out whether the trade-off is worth it. While I’m still entirely too novice to know all of these potential trade-offs still, I have run into a few things already that are worth covering, as there doesn’t seem to be much out there for them.
But First, a Note on Source Control…
While the bulk of this article will focus on the helpers I mentioned above, an area that I sorely neglected out of the gate is that of source control functionality. If you find yourself wondering why it took so long to get my next article out, as well as why this detour now, it’s because I didn’t do enough with source control in the beginning for myself either.
If you come from the pure infrastructure world like myself, your idea of source control might just be that it’s a more complex OneDrive or DropBox, so why bother. Maybe you’ve spent enough time doing PowerShell or working with developer types that you understand what source control is, but think it’s just for ‘real developers’. I can tell you that this is not the case, and that you should, in fact, be using it for everything you do code wise, even just basic scripts.
In my case, I did set up a Git-based source control for my project in Azure DevOps (which is free even for private projects to a degree btw), but I wasn’t religious about it. After an initial commit, I just kept messing with code locally and saving, but wanted to save my commit till I had a whole working setup. It was this delayed mindset that caused me to lose pretty much everything when my hard drive up and died, and all my hard work to that point went up in a puff of smoke.
Long story short(ish), I learned a valuable lesson, and did a bunch more research into repository patterns and using Git. I now have my project split into multiple repositories, all organized into a single project. I’m likely going to reorganize things again, for even greater granularity in the near future as well, as I move towards a more effort oriented approach. As part of this, I’ll also shift how I work on my project, by focusing on just one collective set of work units at a time.
The DB components will remain in a repo by themselves, along with the base classes. That piece is fairly stable, and I’ve not been tinkering with it. This repo will handle the base structure definition for the DB, as well as contain the connection class for everything.
The next unit of work, I’ve set up to be my API web server setup. This will house my core web startup code, and various core services. For example, my global error and log handling services will live in this repository. Much like the DB repo, this will remain relatively stable for the moment, as the associated pieces for this are written and usable elsewhere. My AutoMapper service will live in this repo as well, though it is potentially the only thing that will be changing in relation to other build pieces.
The next piece is that of the DTOs and the controllers that will be a part of my main application. Here I’m a bit torn as the goal is to set the controllers up in such a way that they will remain fairly static, while only the underlying elements will change. With how I’ve been building things out however, I’ve found need to modify the controllers in lots of little ways to accommodate things as I learn and adjust. In light of this, I’m instead choosing to organize each element by the controller. I currently only have four controllers, and each of these is associated to several tables, services, and contract repositories.
From a change of work perspective, I will now do my best to focus on just one controller element at a time, until I get it to a ‘stable’ point. Once I have things ‘working’, even if it’s not entirely where I want them to be, I’ll commit my work and do a sync to my online repo. With my other base repos being unchanged, and committing my work more or less every couple of days, I won’t lose everything again if I suffer another drive failure. Another benefit is that my work will be more organized, as opposed to me figuring something out, and then trying to rush to apply it to all the other controllers.
EFCore Hiccups
I could go on and on about all the things I wish I had known about EFCore when I started, versus where I find myself today. I promise I won’t do that, at least not this time, because this isn’t a rant piece. Suffice it to say, there could be a bit more guidance around preferred patterns of practice, as well as better coverage of differences between Entity Framework, and Entity Framework Core. I’ve run into entirely too many articles that I think have a solve for an issue, but then come to realize the article doesn’t apply to me because it was about Entity Framework, not EFCore.
One of the issues I ran into when re-writing everything after my data wipe, was a compile and run issue. The error in question that I was receiving was about converting int16 to text, but the point at which the error was being thrown was only on execution of the Save method. At no point during any other aspect of walking the code did I see any values or elements that were out of alignment. With the DTO in use, I wasn’t even engaging with more than a handful of fields, and none of them were int16.
After weeks of searching and asking for help, and having everyone tell me my code seemed fine. For the longest time, I thought the issue was with the HierarchyId column. I even reached out to the main guy who developed the hierarchyid component for EFCore, who ended up responding to a stack exchange post. His suggestion was a simple one; scaffold my DB table using the EFCore tools, and compare the generated class to my definition and see if it was different.
It turns out that one of the columns that I had set up to be auto-calculated by SQL had been created as an int16, even though my class had it defined as a text. This field was not part of my DTO, nor was it being queried or interacted with in my code, however when I hit the point at which I attempted to commit my changes, the code broke. While I did look at the columns and do a compare, I had focused only on the columns I was actively using. I’m still not sure how the column got the int16 data type, as I did code-first, but at least I now had the answer. As soon as I updated my class definition, the error went away, and I was able to make it to my next problem.
Data Transfer Objects
DTOs are a wonderful helping hand, as they allow you to customize your input and output data to just what is needed. The problem with DTOs, in my humble opinion, is in properly harnessing them, and in knowing what goes where.
First up is in the vein of creating and naming your DTOs. I have seen any number of approaches, but few consistencies. The larger projects that focus on more scaffolding type approaches all leverage a similar convention however.
From a naming perspective, the practice seems to be Method, then name, then the direction (in or out) that data will flow when using the DTO. The second portion of the convention is that every class that will be interacted with has at least one in, and one out DTO, even if you aren’t doing anything with them initially beyond stripping the ID field off. I can see this helping in a lot of ways, particularly when you are checking your code for errors, because you should only ever have ‘in’ DTOs in your controllers as inputs, and ‘out’ DTOs in your returns.
The other piece that I found horribly confusing at first is how DTOs play into things when interacting with your DB. For example, if you are querying for an item from the DB, it is going to return your regular class. If you are modifying an item, attempting to submit just your DTO will result in errors.
Admittedly, this is the purpose of tools such as AutoMapper, whose job it is to map back and forth between base classes and DTOs, but I’ll get to that in a moment. Before you reach the AutoMapper point however, you still need to know when to put what where, which is something that doesn’t seem to have much in the way of coverage…everything I’ve read just seems to assume you already know.
So, here is my unasked for 2 cents on the matter, based on what I’ve observed combined with no small amount of trial and error.
- Initial inputs collected from your API controller should be an ‘in’ DTO, even if working with the full object properties, since we want to be consistent
- Before submitting the data to the service for processing, the DTO should be mapped back to the original base class, either manually or through AutoMapper
- In the back-end service area, all activities should work exclusively with the base classes, and the full base class should be returned to the controller
- Prior to returning the data, if any, to the requestor, the base class should be mapped to an ‘out’ DTO, even if working with the full object properties
As far as mapping to a DTO, even if working with the full properties, I think this is a good practice even from the start. While you may not initially believe you need a DTO, and might in fact not end up needing one, it reduces rework later should the situation change. By doing it consistently, you leave more of your main structure alone if/when you need to start manipulating content down the road, and it’s a good practice to be in.
As far as the always working with the full class, and remapping in the controller, this is one of practicality. You can, of course, remap in the service and pass that back, but to me it makes the most sense to remap in the same place. By doing all your remapping in a consistent manner, it should make it easier for others to make sense of your code, as well as making it more intuitive as to where changes need to be made.
At the end of the day, it could be a case of six of one, half a dozen of the other, but I think the key piece is being consistent with your approach. If you pick a practice, and then implement it the same every time, you give yourself a better base to work from by ensuring good habits regardless of project size early on. I do the same thing when I build PowerShell modules, with every module being built from the same base elements regardless of how complex the final result is going to be, and I’ve found it makes my life easier when the requirements inevitably change and grow way beyond what they started out as.
AutoMapper Kickers
The last little bit I am going to write about for this article is AutoMapper, and some of the pains I’ve run into. All-in-all, it’s a great item to have in your tool belt, but just like any other tool, it has its own quirks, and not all of them are sufficiently documented. I have seen some developers who swear by it, and I’ve seen just as many others who would rather laboriously map field-by-field than delve into AutoMapper.
In general, and theoretically, AutoMapper is pretty straight forward. You create an AutoMapper profile class, and then you specify the DTO and base class you want to map. After that, everything is butterfly’s and rainbows and unicorns and…yeah, no, it’s not that.
First thing I ran into is that the order of the mapping matters. If you want to map a base class over to an Out DTO, as I talk about above, then you have to have an entry with the base class first, and the DTO class second. For inbound, it is naturally the opposite, with the In DTO first, and the base class second. If done properly, then a simple call to the mapping is all that is required, provided all of your DTO column names are the same as your base class anyway.
If you want to change things up, you have to create the one-to-one mappings within the AutoMapper profile definition for that direction. For example, if my base case has ‘baseClassColumnStuff’, but I want to just use ‘MyStuff’ in the input or output, I’m probably going to have to map that myself. I believe that the tool does employ some logic on its own to try and map automatically, provided you have followed certain conventions, but I haven’t put this effect to the test myself. For now, I’m just keeping the column names the same on both sides.
Despite my playing it safe however, I’ve still run into at least one issue that required updating the profile; Foreign key conversion.
In my table, any time that I have a lookup type item, I’ve been using a table for that purpose. As an example, my framework that I’ve developed currently defines five layers of organizational unit. The third layer of this structure can be between one and three levels, and the number of levels may vary broadly within a single deployment. To ensure that I always correctly identify the layer type correctly, I use a foreign key lookup that defines the supported types. D
Before any of the more experienced dev types points out enums, I purposely elected to use tables instead for a couple of reasons. First, each time I deploy the model, it seems like I identify some new use case that requires that I make adjustments to the framework. I’ve not had any related to the OU layer types yet, but I did encounter one that required adding more flexibility. In the event that I did need to make a change at some point, I’d rather have it be dynamically supported by just adding a value to a table, rather than needing to change code.
Down the road, I’m hoping to build a business rules layer that will inter-relate with the layer types, among other things, to enforce certain minimal structures and the like. When I get there, I’ll likely be looking at possible code changes when adding layers, but my intent will be to make that dynamic as well, and so tables and foreign keys it is.
Getting back to how this relates to AutoMapper, the challenge I ran into was when I wanted to return the more friendly value in the name column, rather than the ID, as part of my output. Figuring out how to do this was an absolutely nightmare, even though the end effort required was relatively simple. The issue came from the fact that I had to find the exact right combination of search terms before I happened across an answer on Google. I’m normally pretty good at finding what I need in relatively short order, but that’s because normally whatever I’m searching for has been discussed in so many different ways that my chances are pretty good if I word my searches the right way.
As a small side bit here, I should point out that there are a number of elements specific to EFCore and what are called ‘navigational properties’ that came into play here as well. These increased the complexity of finding what I needed to solve the problem, because it wasn’t clear where exactly the problem was coming from.
Without using AutoMapper, I could build out a return object one property at a time, and I could walk the navigation to specify the related name property from the foreign table. In EFCore terms, I believe this is a form of lazy loading, but I’m not entirely sure. I also looked into lazy loading and eager loading as well. When talking about EFCore 3.x and earlier, this was apparently a really big problem, and required a lot of careful planning to ensure you didn’t consume too much memory and crash your box. This led to trying things like using ‘.include’ code in my repository code, which did not solve the issue.
The problem I was having, of course, is that I didn’t want to have to manually map all of those properties, and I shouldn’t have to because that’s what AutoMapper was for right? Unfortunately for me, no matter how I tried to go about loading in the property values, I either got a null value, or I got a vague AutoMapper error indicating an unmapped property. I get that being a programmer means that more of the training wheels are off, but I feel like it could at least tell me what property it was trying to map that failed. Instead, I got nothing from VS in warning, nothing specific error wise, and generally too little info to go on.
One of the things I ended up learning about was improvements to the EFCore engine as part of 5.x, in terms of navigational properties and loading. As I walked through debugger, I noticed that I had the required data when I looked at what was returned from the DB, but then was getting an error only when I hit AutoMapper as part of the conversion to the output object. This caused me to switch the focus of my search from something specific to EFCore to trying to figure out what was off from an AutoMapper perspective.
The solution, in the end, was first to add the ‘.ForMembers’ code element to the AutoMapper profile entry for the specific conversion I wanted to make. I had to tell AutoMapper that ‘fkName’ was the same thing as ‘myObject.fkTable.fkName’ directly, even though the name was the same. The second part of the equation was on the EFCore side, and involved tracking of changes. In all of my queries, since I wasn’t making changes, I had tracking set to false. When you do this however, EFCore doesn’t perform the additional relationship fixup steps that enable it to return the values I wanted.
Conclusion
Another lengthy article with a lot of content in it, but hopefully a lot of helpful content for someone out there. I had honestly hoped to be further along in my journey, and therefore further along in this series of posts, but the loss of data, coupled with a new project and the challenges faced above, has conspired against me somewhat. Here’s hoping my next article will come along a bit faster, now that I’ve gotten through a few of these growing pains though.
Until next time!
