bash – James' Blog

Concatenating files in Powershell

When working with databases, it’s best practice to create migration scripts to which apply your database changes, rather than using GUI/designer tools which need you to manually apply changes across multiple environments.

I’ve been working with a client where there isn’t a specific migration tool to run all these migration files at the end of project, but they need to be bundled into a single file and attached to the change tracking system.

I keep these files under version control and prefix migration files with M_ and rollback files with R_

In Bash concatenating these files is simple:
cat M_* > ./migration.sql

In Powershell it isn’t too different:
Get-Content M_* | Set-Content .\migration.sql

When dealing with rollback scripts I create the rollback which opposes the migration with the same number which makes the files easy to navigate:

./M_001_AddsTable.sql ./M_002_AddsView.sql ./R_001_AddsTable.sql ./R_002_AddsView.sql

Rollback scripts are slightly more interesting as they need to be run in reverse.

We can also achieve this in Powershell with a minor tweak to our previous command:
Get-Content R_* | Sort-Object -Property Name -Descending | Get-Content | Set-Content .\migration.sql

This will firstly get all our rollback files and then reverses the order, then gets the content of each file and writes it to a single file.

Although I prefer doing these types of tasks with Bash, it’s still quite easy to remember the Powershell commands if you don’t have access to Bash on a Windows system.

Sharing bash functions between scripts

I’ve spent most of the past week writing bash functions to help with automating build and deployment at work, and something I’ve never really paid much attention to is how to keep these .sh files structured. Trying to keep functions small and generic helps with reuse and saves me from having to write more code, and once I’ve tested something is working hopefully I won’t need to touch it again.

To quote the first item of the summarised Unix philosophy:

Write programs that do one thing and do it well.

It’s also made me think about another horrible mess of code that I’ve come to depend on, my .bash_aliases file.

This has truly become a dumping ground over the years to save me from having to remember almost everything I have done at one point or another. At what I feel is a whopping 1474 lines, my little black book of functions is something that I lean on a lot, but have never really taken any time to trim or polish, apart from some headings in the form of comments.

Creating some structure

So the first step in organising this mess is trying to group similar functions together. Near the very top of my .bash_aliases file I find this:

##########  Git aliases ##################
alias gs='git status'
alias ga='git add'
alias gd='git diff'
alias gc='git commit'
alias gca='git commit -a'
alias gb='git branch'
alias gbr='git branch --remote'
#########################################

So there are some aliases that I use because I’m lazy, and shaving off those extra keystrokes should save some wear and tear on my little digits. I’ve already noted in the file what this group applies to GIT so that is a great starting point.

Let’s create a new directory to store my newly structured files:
mkdir ~/scripts
There’s a new directory in my home directory called scripts. Next is to copy and paste the contents from .bash_aliases to a new file called git.sh within this folder.

Setting up the files

I could just copy and paste the contents in Emacs, but where’s the fun in that. I can use sed to grab the lines that I want and output them to this new file without having to leave the terminal.

sed -n '19,27p' ~/.bash_aliases > ~/scripts/git.sh

So here we are using sed to get the line between 19 and 27 from the file ~/.bash_aliases and outputting them to the file ~/scripts/git.sh easy. Bash files should have the following at the top ‘#!/bin/bash’, to tell the OS which interpreter should be used for the file. This way seems slightly hacky, but again, means I don’t need to leave the terminal.

sed -i '1s:^:#\!/bin/bash\n:' ~/scripts/git.sh

This sed command inserts the required text on the first command, followed by a new line, giving us what we want.

Next we need to delete the text from .bash_aliases. Again, let’s stay in the terminal, we’ve almost written the command we need already.

sed -n '19,27d' ~/.bash_aliases

So, sed lessons are over (almost), our files are set up and I’ll try and get this post back on track.

If we were to open a terminal at this point, the commands we’d receive a command not found message. This can easily be fixed, and I’m going to use sed again because why not?

sed -i '2s:^:\n\. ~/scripts/git\.sh\n\n:' ~/.bash_aliases

Here we are inserting a new line to our .bash_aliases file using the . command to source the contents of our git.sh file. Because .bash_aliases is run every time we open a bash shell, we now have contents of the ~/scripts/git.sh available at all times, whilst also keeping our git alias code separate.

I’ve got lots of functions that can be grouped together, so I envisage creating a azure-cli.sh file, docker.sh, code-generation.sh file and many others in the future.

Repeating this over the coming months should really help whip my .bash_aliases file into shape.

Conclusion

So this post devolved into a post about the Power of sed rather than what it was initially supposed to be, but the TD;DR is that:

1) massive files are bad
2) the . command will source or import code from another file
3) grouping similar functions into separate files helps make navigation of code easier
4) with the . command we can share code between files without repeating ourselves
5) we can use sed to manipulate text files succinctly, precisely with a couple of keystrokes

If you’ve made it this far, thanks for reading.

Running MSBuild on Windows in Git Bash

So I’m in the process of automating the build and publish of a .Net WinForms application with Squirrel in Windows.

The first step in achieving this is to get the project building programmatically, outside of Visual Studio.

Coming from a Linux background I prefer working with Bash over Powershell or Batch and Git Bash is my terminal of choice within Windows.

Discovering the tools we need

The application uses framework 4.6.1 and I wasn’t sure what version of the tools were needed. To find out what tools were already installed on my system I ran:

dir HKLM:\SOFTWARE\Microsoft\MSBuild\ToolsVersions\

This listed a couple of versions, 2.0, 3.5 and 4.0.30319. I decided to try to build the project with the latest installed tools using:

C:\Windows\Microsoft.NET\Framework64\v4.0.30319\MSBuild.exe C:\path\to\solution.sln

Which caused the following error:
Project file contains ToolsVersion="15.0".

So now I know I need version 15.0 of the build tools.

One mystery solved.

Downloading and Installing the BuildTools

A quick search led me to this download page and after downloading the Build Tools for Visual Studio 2017, I ran the executable.

As I’m only currently trying to build a particular framework, under the Windows section I checked .NET desktop build tools and only checked the optional installation of Testing tools core features - Build Tools, as .NET Framework 4.6.1 SDK and targeting pack is included by default.

Now we’re cooking with gas.

So now we’ve got the correct build tools, let’s check we can build the project successfully. From a Powershell terminal, type:

C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\15.0\Bin\MSBuild.exe "C:\path\to\project\solution.sln"

Build Successful!

Working with Bash

Now we’ve proven that the tools work, we just need to make them accessible from Git Bash.

In order to be able to access MSBuild.exe from Git Bash we’ll need to add the path to MSBuild.exe used above to the $PATH environment variable. This is quick and easy.

PATH="$PATH:/c/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/MSBuild/15.0/Bin"

Some things to note from the above command:
– Forward slashes are used instead of backslashes for paths in Bash
– We can access the C:\ drive using just the drive letter in Git Bash

Now we can run the following command from our Git Bash window:

MSBuild.exe "/c/path/to/solution/solution.sln"

We should get identical output to when we ran the command initially in Powershell.

Adding switches to MSBuild

The final thing we will want to do is add some switches to the build command so we can Clean/Build/Rebuild and set the Configuration to Debug or Release.

In Powershell we could run the following:

MSBuild.exe "/c/path/to/solution/solution.sln" /t:Rebuild /p:Configuration=Release

Due to Bash using / for file paths, we need to escape any / characters with another /. So in Git Bash the above command becomes:

MSBuild.exe "/c/path/to/solution/solution.sln" //t:Rebuild //p:Configuration=Release

And there we have it, building a .NET solution with Git Bash and it only took a few minutes. By putting this in a script, we can now automate building of our WinForms solutions, ready for packaging and deployment with Squirrel.

Resources

In discovering the above, the below answers on StackOverflow helped steer me in the right direction.

Bonus Points

As described in the answer for this question, we could also script the install of the build tools, rather than using the GUI, because who in their right mind wants to use the GUI???

https://stackoverflow.com/questions/42696948/how-can-i-install-the-vs2017-version-of-msbuild-on-a-build-server-without-instal

That will be the next step so we can script the install of our build tools, making it effortless to configure a new build server.