Jekyll2022-06-04T15:58:08+02:00https://dynamicautomation.nl/feed.xmlDynamic AutomationWrite an awesome description for your new site here. You can edit this line in _config.yml. It will appear in your document head meta (for Google search results) and in your feed.xml site description.Sudesh Jethoe, MSc. MBA.Kustomize for fun and profit2021-06-23T00:00:00+02:002021-06-23T00:00:00+02:00https://dynamicautomation.nl/articles/kustomize-for-fun-and-profit<h1 id="introduction">Introduction</h1>
<p>Managing a container based infrastructure on <a href="https://kubernetes.io/">kubernetes</a> can be a daunting task. Often, many resources are involved, each requiring specific adjustments for environments against they need to be deployed against. Managing the resources individually for many environments is error-prone and should therefore be avoided.</p>
<h2 id="kustomize">Kustomize</h2>
<p><a href="https://kustomize.io/">Kustomize</a> is a configuration management solution which allows developers to reuse existing resource and tailor them to specific environments. In contrast to other configuration management solutions which compose infrastructure by combining unique elements together, kustomize uses an additive approach by patching existing resources and overwriting or adding properties to existing resources.</p>
<p>In this article I will explain various methods and best practices for setting up configuration for kubernetes using <a href="https://kustomize.io/">Kustomize</a>.</p>
<h1 id="alternatives">Alternatives</h1>
<p>Kustomize vs Helm vs Crossplane vs Terraform vs …</p>
<p>When considering configuration management systems for Kubernetes in particular, we can ofcourse also use Helm or something such as Crossplane or Pulumi. Why would you want to use one or the other and what are the advantages of Kustomize in particular?</p>
<h2 id="kustomize-vs-helm">Kustomize vs Helm</h2>
<h3 id="helm">Helm</h3>
<p><a href="https://helm.sh/">Helm</a> is a configuration management tool for Kubernetes which is mainly focused on the <em>distribution</em> of distributed applications running on top of Kubernetes.
Distributed applications are often composed of multiple microservices which each can have their own requirements with regards to availability, system- and network-resources.
Helm provides package management for applications where all required resources have been prefedined and only the input parameters for running the distributed application in your own cluster are required.</p>
<p>In theory it should be possible to deploy a helm package (<em>helm chart</em>) <em>as is</em> with providing merely the required input parameters through a yaml file. However, in practice, helm charts are often highly opinionated, the package makes assumptions on the features of the cluster it will be running in. Such assumptions could be:</p>
<ul>
<li>The version of Kubernetes which it is running on</li>
<li>Underlying platform capabilities (Azure, AWS, GCP)</li>
<li>Permissions available to the deployment agent (to be able to run as Daemonset or permissions to create ServiceAccounts)</li>
<li>Cluster resource access (Storage, CPU)</li>
</ul>
<p>As Kubernetes can be deployed in many different contexts and is also used in secured environments these assumptions often require some adjustments. What happens is that these packages are downloaded by the DevOps/SRE squads and then adjusted to make the package run in their own context.</p>
<p>Personally I consider this an anti-pattern, the purpose of these packages is to be able to run the software the way that the developer intended it and by using the package we can leverage the work of the community on its deployment. When we adapt such packages manually we incur a technical debt as now we are ourselves responsible for managing and adapting the deployment process in the future (in case of software updates).</p>
<h3 id="kustomize-1">Kustomize</h3>
<p>Although <em>Kustomize</em> allows us to manage the configuration of deployments, it does not offer us a feature for package management and distribution.
Kustomize instead merely focusses on patching existing Kubernetes resource definitions (manifests) to be able to adapt them to different environments.
As this is such a common use case, Kustomize has been integrated into the <em>kubectl</em> command itself and can be triggered during any regular deployment of Kubernetes resources.</p>
<h1 id="directory-structure">Directory Structure</h1>
<h1 id="resources">Resources</h1>
<h2 id="deployment">Deployment</h2>
<h2 id="configmap">Configmap</h2>
<h2 id="service">Service</h2>
<h2 id="ingress">Ingress</h2>
<h1 id="compose">Compose</h1>
<h1 id="pipeline">Pipeline</h1>
<h1 id="references">References</h1>
<ol>
<li><a href="https://kubernetes.io/">Kubernetes</a></li>
<li><a href="https://kustomize.io/">Kustomize</a></li>
</ol>
<hr />Sudesh Jethoe, MSc. MBA.Developing a container based infrastructure with kubernetes and kustomize.Developing a Container Platform on Azure2019-02-01T00:00:00+01:002019-02-01T00:00:00+01:00https://dynamicautomation.nl/portfolio/containerplatform-on-azure<p><a href="https://www.ing.com/About-us/Profile/ING-at-a-glance.htm">ING</a> is a global financial institution headquartered in The Netherlands. ING is a global leader in finance and innovation and has been mentioned as one of the world’s best banks by <a href="https://www.forbes.com/sites/antoinegara/2020/06/08/the-worlds-best-banks-the-future-of-banking-is-digital-after-coronavirus/">Forbes</a>.<br />
In their journey to digital transformation the retail domain of ING Netherlands was looking for possibilities to move some of its workloads to the public cloud.<br />
I was asked to develop a strategy and architecture to land IT-services in the public cloud without compromising operational risk, whilst still enabling 300+ DevOps squads to work in an agile fashion.</p>
<p>The public cloud offers many services, all of which can be enabled “at the flick of a button”, however, in order to migrate a heavily regulated organization, many measures need to be put in place before public cloud services are even allowed to be consumed. Being closely inspected by the public and the regulators, any deficiencies might lead to hefty fines or, even worse, negative publicity. That a migration to the public is not without risk is also something we can see when looking at other <a href="https://edition.cnn.com/2019/07/29/business/capital-one-data-breach/index.html">organizations which have moved workloads to the public cloud</a>.</p>
<h1 id="references">References</h1>
<ol>
<li><a href="https://edition.cnn.com/2019/07/29/business/capital-one-data-breach/index.html">A hacker gained access to 100 million Capital One credit card applications and accounts, Rob McLean, CNN Business</a></li>
</ol>
<hr />Sudesh Jethoe, MSc. MBA.Designing a container platform for mission critical systems.Booting SystemRescueCD ISO from EFI on Fedora 28 Linux2018-07-26T00:00:00+02:002018-07-26T00:00:00+02:00https://dynamicautomation.nl/articles/booting-systemrescuecd-iso-from-efi-on-fedora-28-linux<h1 id="introduction">Introduction</h1>
<p>Yesterday I noticed that my Thinkpad X1 laptop had assigned too little diskspace to its OS partition. As it was running in a physical partition and not a logical one, it was time to move it to LVM. Unfortunately I had no USB sticks anymore.<br />
As such I decided to move the whole system partition to LVM by using the SystemRescueCD (sysrcd) livecd booting directly from the disk itself.<br />
It was quite a puzzle to get this done, therefore I decided to write it down for whoever might need to do the same thing.</p>
<h1 id="steps">Steps</h1>
<ol>
<li>Confirm startup mode and disk layout</li>
<li>Download sysrcd and copy to the right location</li>
<li>Configure grub2 to read the sysrcd iso</li>
<li>Update grub2</li>
<li>Update EFI/BIOS boot settings</li>
<li>Boot into sysrcd</li>
</ol>
<h1 id="checking-startup-mode-and-disk-layout">Checking startup mode and disk layout</h1>
<p>As its possible to run modern systems in EFI and BIOS mode and this tutorial is mode specifically for systems in EFI mode with GPT disk layout check these before you continue.</p>
<h2 id="check-efi-or-bios">Check EFI or BIOS</h2>
<p>To check if you are running in EFI, you can use the following commands</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># if this folder exists you're running in EFI mode</span>
<span class="nv">$ </span><span class="nb">ls</span> /sys/firmware/efi/config_table
efivars esrt fw_platform_size fw_vendor runtime runtime-map systab
<span class="c"># this command will show output when running in EFI mode (not installed by default)</span>
<span class="nv">$ </span>efibootmgr
BootCurrent: 0012
Timeout: 0 seconds
BootOrder: 0012,0013,0000,0001,0002,0003,000C,0007,0008,0009,000A,000B,000D
Boot0000 Setup
Boot0001 Boot Menu
Boot0002 Diagnostic Splash Screen
Boot0003 Lenovo Diagnostics
<span class="nt">---</span>
Boot0012<span class="k">*</span> Fedora
Boot0013<span class="k">*</span> Windows Boot Manager
</code></pre></div></div>
<h2 id="check-disk-layout">Check disk layout</h2>
<p>EFI systems support the classical disk layout (MBR or master boot record layout), but also a modern one which is called the GPT or GUID Partition Table layout. There are quite some differences between both, one of them being that MBR only supports 4 primary partitions with sizes up to 2 TB, GPT can go up to at least 128 and up to 1 ZB partition size (256 TB on Windows). As GRUB2 addresses these layouts differently it’s important to ensure you’re using GPT before we continue (if you’re not using GPT, just use the normal instructions described on the <a href="https://help.ubuntu.com/community/Grub2/ISOBoot/Examples#SystemRescueCD" target="_blank" rel="noopener">Ubuntu GRUB2 Example pages</a>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>gdisk <span class="nt">-l</span> /dev/sda
GPT fdisk <span class="o">(</span>gdisk<span class="o">)</span> version 1.0.3
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present
Found valid GPT with protective MBR<span class="p">;</span> using GPT.
</code></pre></div></div>
<p>Output of <b>gdisk</b> when ran against a GPT formatted disk.</p>
<h1 id="download-sysrcd-and-copy-to-the-right-location">Download sysrcd and copy to the right location</h1>
<p>The first step is to download the SystemRescueCD from the website here: <a href="http://www.system-rescue-cd.org/Download/" target="_blank" rel="noopener">Download</a>. For GRUB2 to be able to read the ISO file, it needs to be placed <strong>on the same partition as the /boot partition</strong>. In my case this is the system partition itself. For convenience I follow the instructions described on the <a href="http://www.system-rescue-cd.org/manual/Installing_SystemRescueCd_on_the_disk/" target="_blank" rel="noopener">SystemRescueCD documentation page</a> and place the ISO in:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/boot/sysrcd/systemrescuecd-x86-x.y.z.iso
</code></pre></div></div>
<p>Now let’s continue to the next step and configure GRUB2</p>
<h1 id="configuring-grub2">Configuring GRUB2</h1>
<p>Before making changes to GRUB2, we need to dive a bit into how GRUB2 is actually setup in Fedora systems.<br />
The GRUB2 system can be quite daunting and complex, but for this tutorial you only need to know the following commands and locations:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># write a new Grub2 EFI configuration for fedora on EFI</span>
grub2-mkconfig <span class="nt">-o</span> /boot/efi/EFI/fedora/grub.cfg
<span class="c"># location of default grub configuration settings</span>
/etc/default/grub
<span class="c"># location of grub2 configurable OSes</span>
/etc/grub.d/
<span class="c"># final location of compiled grub2 configuration (!do not edit this file directly!)</span>
/boot/efi/EFI/fedora/grub.cfg
</code></pre></div></div>
<p>Now let’s create our new GRUB2 configuration, by creating a new grub configuration in</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/etc/grub.d/20_sysrcd
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/sh</span>
<span class="nb">exec tail</span> <span class="nt">-n</span> +3 <span class="nv">$0</span>
<span class="c"># This file provides an easy way to add custom menu entries. Simply type the</span>
<span class="c"># menu entries you want to add after this comment. Be careful not to change</span>
<span class="c"># the 'exec tail' line above.</span>
menuentry <span class="s1">'SystemRescueCD (64-bit) '</span> <span class="o">{</span>
<span class="nb">set </span><span class="nv">root</span><span class="o">=</span><span class="s1">'hd0,gpt6'</span>
<span class="nb">set </span><span class="nv">isofile</span><span class="o">=</span><span class="s2">"/boot/sysrcd/systemrescuecd-x86-5.2.2.iso"</span>
loopback loop <span class="o">(</span>hd0,gpt6<span class="o">)</span><span class="nv">$isofile</span>
linuxefi <span class="o">(</span>loop<span class="o">)</span>/isolinux/rescue64 <span class="nv">setkmap</span><span class="o">=</span>us <span class="nv">isoloop</span><span class="o">=</span><span class="nv">$isofile</span> docache
initrdefi <span class="o">(</span>loop<span class="o">)</span>/isolinux/initram.igz
<span class="o">}</span>
</code></pre></div></div>
<p>After writing the file, don’t forget to make it executable</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">chmod</span> +x /etc/grub.d/20_sysrcd
</code></pre></div></div>
<h1 id="update-efibios-boot-settings">Update EFI/BIOS boot settings</h1>
<p>Now run the update grub command for Fedora on EFI</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>grub2-mkconfig <span class="nt">-o</span> /boot/efi/EFI/fedora/grub.cfg
</code></pre></div></div>
<p>This command writes a new configuration to /boot/efi/EFI/fedora/grub.cfg, a new section in this file should pop-up now:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">### BEGIN /etc/grub.d/20_sysrescd ###</span>
<span class="c"># This file provides an easy way to add custom menu entries. Simply type the</span>
<span class="c"># menu entries you want to add after this comment. Be careful not to change</span>
<span class="c"># the 'exec tail' line above.</span>
menuentry <span class="s1">'SystemRescueCD (64-bit) '</span> <span class="o">{</span>
<span class="nb">set </span><span class="nv">root</span><span class="o">=</span><span class="s1">'hd0,gpt6'</span>
<span class="nb">set </span><span class="nv">isofile</span><span class="o">=</span><span class="s2">"/boot/sysrcd/systemrescuecd-x86-5.2.2.iso"</span>
loopback loop <span class="o">(</span>hd0,gpt6<span class="o">)</span><span class="nv">$isofile</span>
linuxefi <span class="o">(</span>loop<span class="o">)</span>/isolinux/rescue64 <span class="nv">setkmap</span><span class="o">=</span>us <span class="nv">isoloop</span><span class="o">=</span><span class="nv">$isofile</span> docache
initrdefi <span class="o">(</span>loop<span class="o">)</span>/isolinux/initram.igz
<span class="o">}</span>
<span class="c">### END /etc/grub.d/20_sysrescd ###</span>
</code></pre></div></div>
<p>After restarting the system you should now be able to select “System Rescue CD” as a boot option, this will only work when you have disabled “secure boot” in your BIOS!</p>
<h1 id="sources">Sources</h1>
<ul>
<li><a href="https://docs.fedoraproject.org/f27/system-administrators-guide/kernel-module-driver-configuration/Working_with_the_GRUB_2_Boot_Loader.html" target="_blank" rel="noopener">Fedora 27 “Working with the Grub2 bootloader”</a></li>
<li><a href="http://www.system-rescue-cd.org/manual/Installing_SystemRescueCd_on_the_disk/" target="_blank" rel="noopener">Installing SystemRescueCd on the disk</a></li>
<li><a href="https://help.ubuntu.com/community/Grub2/ISOBoot" target="_blank" rel="noopener">Ubuntu docs, Grub2 ISOBoot</a></li>
<li><a href="https://help.ubuntu.com/community/Grub2/ISOBoot/Examples#SystemRescueCD" target="_blank" rel="noopener">Grub2 ISOBoot example SystemRescueCD</a></li>
</ul>Sudesh Jethoe, MSc. MBA.Introduction Yesterday I noticed that my Thinkpad X1 laptop had assigned too little diskspace to its OS partition. As it was running in a physical partition and not a logical one, it was time to move it to LVM. Unfortunately I had no USB sticks anymore. As such I decided to move the whole system partition to LVM by using the SystemRescueCD (sysrcd) livecd booting directly from the disk itself. It was quite a puzzle to get this done, therefore I decided to write it down for whoever might need to do the same thing.Configure postfix with Gmail as relayhost2018-05-03T00:00:00+02:002018-05-03T00:00:00+02:00https://dynamicautomation.nl/articles/configure-postfix-with-gmail-as-relayhost<p>On Linux servers it is often useful to be able to send e-mail for logging and alerting.<br />
When using a VPS setup of the SMTP service is often not included and needs to be configured manually.<br />
As Google already has a pretty good mail service with Gmail, it’s useful to use this for relaying your emails.<br />
In this post we will discuss the requirements for using gmail as a relayhost and steps necessary to configure postfix.</p>
<p>As Google services require credentials and you shouldn’t use your own username and password for login,
it’s a good idea to create an “app password” for the postfix service, instructions can be found
<a href="https://support.google.com/mail/answer/185833?hl=en" target="_blank" rel="noopener">here</a>.</p>
<p>After creating the app password we can login to the VPS and configure Postfix.<br />
<strong>! Be aware there are many instructions available online, but I found many of them incomplete or just plain wrong.</strong></p>
<h1 id="outline">Outline</h1>
<p>The general steps are:</p>
<ol>
<li>Create <strong>sasl_passwd</strong> file which contains the plaintext credentials for google</li>
<li>Generate <strong>sasl_passwd.db</strong> file which is used by postfix</li>
<li>Create a <strong>tls_policy</strong> file which contains a plaintext version of the tls_policy used by postfix</li>
<li>Generate <strong>tls_policy.db</strong> file which is used by postfix</li>
<li>Edit settings and references to previous files in postfix main configuration file <strong>main.cf</strong></li>
</ol>
<h1 id="1-create-sasl_passwd-file">1. Create <strong>sasl_passwd</strong> file</h1>
<p>Create a new file: <em>/etc/postfix/sasl_passwd</em> with content:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>smtp.gmail.com]:587 USER@DOMAIN:PASSWORD
<span class="c"># Where</span>
USER <span class="o">=</span> your gmail or gsuite username
DOMAIN <span class="o">=</span> gmail.com or yourgsuitedomain.com
PASSWORD <span class="o">=</span> the app password you previously created
</code></pre></div></div>
<h1 id="2-generate-sasl_passwddb">2. Generate <strong>sasl_passwd.db</strong></h1>
<p>This command generates the sasl_passwd.db file.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd</span> /etc/postfix
postmap /etc/postfix/sasl_passwd
</code></pre></div></div>
<h1 id="3-create-a-tls_policy">3. Create a <strong>tls_policy</strong></h1>
<p>Create another file: <em>/etc/postfix/tls_policy</em>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">[</span>smtp.gmail.com]:587 encrypt
</code></pre></div></div>
<h1 id="4-generate-tls_policydb">4. Generate <strong>tls_policy.db</strong></h1>
<p>Generate the tls_policy.db file</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>postmap /etc/postfix/tls_policy
</code></pre></div></div>
<h1 id="5-edit-postfix-maincf-configuration-file">5. Edit postfix <strong>main.cf</strong> configuration file</h1>
<p>Now add/ensure the following configuration parameters are set</p>
<h2 id="51-tls-settings">5.1 TLS settings</h2>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># TLS parameters</span>
<span class="nv">smtpd_tls_cert_file</span><span class="o">=</span>/etc/ssl/certs/ssl-cert-snakeoil.pem
<span class="nv">smtpd_tls_key_file</span><span class="o">=</span>/etc/ssl/private/ssl-cert-snakeoil.key
<span class="nv">smtpd_use_tls</span><span class="o">=</span><span class="nb">yes
</span>smtpd_tls_session_cache_database <span class="o">=</span> btree:<span class="k">${</span><span class="nv">data_directory</span><span class="k">}</span>/smtpd_scache
smtp_tls_session_cache_database <span class="o">=</span> btree:<span class="k">${</span><span class="nv">data_directory</span><span class="k">}</span>/smtp_scache
smtp_tls_policy_maps <span class="o">=</span> <span class="nb">hash</span>:/etc/postfix/tls_policy
</code></pre></div></div>
<h2 id="52-host-and-relay-configuration">5.2 Host and Relay Configuration</h2>
<p>Replace HOSTNAME with the actual FQDN of the VPS on which you are configuring postfix</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>myhostname <span class="o">=</span> HOSTNAME
mydestination <span class="o">=</span> HOSTNAME, localhost, localhost.localdomain, localhost
relayhost <span class="o">=</span> <span class="o">[</span>smtp.gmail.com]:587
</code></pre></div></div>
<h2 id="53-gmail-sasl-configuration">5.3 Gmail SASL Configuration</h2>
<p>! One of my sources incorrectly stated that the sasl_mechanism should be <strong>plain</strong>, however this didn’t work for me, however <strong>login</strong> did!</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Enable SASL authentication (for Gmail)</span>
smtp_sasl_auth_enable <span class="o">=</span> <span class="nb">yes
</span>smtp_sasl_password_maps <span class="o">=</span> <span class="nb">hash</span>:/etc/postfix/sasl_passwd
smtp_sasl_security_options <span class="o">=</span> noanonymous
smtp_sasl_tls_security_options <span class="o">=</span> noanonymous
<span class="c">#smtp_sasl_mechanism_filter = plain</span>
smtp_sasl_mechanism_filter <span class="o">=</span> login
smtp_tls_security_level <span class="o">=</span> encrypt
smtp_tls_CAfile <span class="o">=</span> /etc/ssl/certs/ca-certificates.crt
</code></pre></div></div>
<h1 id="finalize">Finalize</h1>
<p>After changing everything, just restart postfix and send a test email with:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>systemctl restart postfix
<span class="nb">echo</span> <span class="s2">"this is a test email"</span> | mail <span class="nt">-s</span> <span class="nb">test </span>youremail@gmail.com
</code></pre></div></div>
<h1 id="debugging">Debugging</h1>
<p>If you run into errors it might be useful to check the following files for information</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/var/log/mail.log
/var/log/mail.err
</code></pre></div></div>
<h1 id="sources">Sources</h1>
<ol>
<li><a href="https://linode.com/docs/email/postfix/configure-postfix-to-send-mail-using-gmail-and-google-apps-on-debian-or-ubuntu/">https://linode.com/docs/email/postfix/configure-postfix-to-send-mail-using-gmail-and-google-apps-on-debian-or-ubuntu/</a></li>
<li><a href="https://serverfault.com/questions/627128/postfix-sasl-authentication-failed-internal-authentication-error">https://serverfault.com/questions/627128/postfix-sasl-authentication-failed-internal-authentication-error</a></li>
</ol>Sudesh Jethoe, MSc. MBA.On Linux servers it is often useful to be able to send e-mail for logging and alerting. When using a VPS setup of the SMTP service is often not included and needs to be configured manually. As Google already has a pretty good mail service with Gmail, it’s useful to use this for relaying your emails. In this post we will discuss the requirements for using gmail as a relayhost and steps necessary to configure postfix.Developing a Big Data Strategy2017-11-01T00:00:00+01:002017-11-01T00:00:00+01:00https://dynamicautomation.nl/portfolio/developing-a-big-data-strategy<p><em>If you were the Chief Data Officer in your current organization and were assigned with the task to build a strategy that leverages Data Analytics, what would the strategy look like? In case you work for yourself, you can answer this question for a client organization.</em></p>
<h1 id="develop-a-data-strategy-for-your-current-organization">Develop a Data Strategy for your current organization</h1>
<p>The organization that I am currently in, has a big problem with the balance between sharing and protecting of its data.
Various systems within the company generate large amounts of metric and logging data. These types of data are extremely useful in detecting and possibly predicting failure of components which are critical for supporting business functions. However, in order to gain proper insights from this data it is important to share the data between teams within the company.
As the company is dealing with privacy sensitive information, it is company policy to apply security measures depending on the sensitivity of the exposed data. Security measures vary from non-existent (public data), data masking to restricting access to specific teams or even persons.
As there is no clear policy to classify risks regarding the exposure of metric data, the risk teams, by default, tend to give metric collection systems high sensitivity classifications. Such classifications require teams to implement a strict set of security measures, effectively preventing the sharing of the metric- and logging data. This exposes an additional issue that teams are inexperienced in the practice of acquiring and sharing their data with other teams.</p>
<h2 id="improvement-proposals">Improvement Proposals</h2>
<p>As the Chief Data Officer I would enable teams to share their data with other teams. This will allow them to be able to gain insight from each other’s data. Although it is a small step in creating a more data centric organization, the transformations required to implement an insights-driven organization will also be beneficial when the company is ready to expand its data-driven capabilities in other areas.
To develop the capability successfully, two key concerns have to be addressed, namely the skill gap of the employees regarding data-driven practices and the privacy and security regulations within the organization.
In <a href="https://doi.org/10.1016/j.bushor.2017.01.002">1</a> it is proposed to have organizations collaborate with educational institutions in order to address the skill gap in data engineering and -science. In my experience it is impossible for people to understand and successfully apply data-driven practices successfully with merely a training. A better way would be to attract experts which can guide and drive the teams in the right direction by a period of intensive training and cooperation, effectively building the practice together. Such an approach has two benefits, namely that people get to work with real experts and gain insight in the thought process and paradigms of data science. Also, by building the practice themselves, the team acquires a greater sense of ownership of the implemented solutions.
To address the issue of privacy we are required to follow the law, and as is proposed in <a href="https://doi.org/10.1016/j.bushor.2017.01.002">1</a> is the incorporation of “general best practices”. This is interesting, as we are dealing with possibilities created by the utilization of new technology. These possibilities did not exist before and therefore it can be difficult or even impossible to find “best practices”. It would be better to give teams the freedom to develop their data science practices, whilst at the same time making them responsible for the risks which might emerge from their progress. This approach prevents teams from being held back by defensive departments such as risk. While a risk department´s primary concern is mitigating risk, a team is better capable of finding the balance between both risk and development of new solutions.</p>
<h1 id="developing-a-data-strategy-for-the-dutch-government">Developing a data strategy for the Dutch government</h1>
<p><em>You have been asked to become the Minister of Data Analytics in the Netherlands? What would be the key interventions in Dutch government or society that you would want to implement in the coming 4 years, and why these?</em></p>
<p>As the Minister of Data Analytics in the Netherlands, I would focus on three main areas, namely education, open data and governance. Although the area of data analytics has gained traction only recently, it is important for the Netherlands as knowledge economy to stay ahead in such developments. Education is a key driver, enabling people to come with their own ideas and having the knowledge to realize them as well. Data analytics can only exist by having data, therefore it is important to be able to easily expose data. At the same time, it is still important to expose data in such a way that it does not cause harm to individuals or the public, therefore governance must be taken into account.</p>
<h2 id="education">Education</h2>
<p>Data-literacy of the public has to be taught before going into any specialized education. Therefore, it is important to start mandatory data-programs in high-schools. Data being the next technological frontier, people should be educated to be critical not only about its risks, but also about its possibilities.</p>
<h2 id="open-data">Open Data</h2>
<p>Open data initiatives exist in the government already for some time. However, when looking at the datasets exposed at https://data.overheid.nl/data/dataset it can be seen that there are many different formats and suppliers. To enable the people to consume these data, it would be better if there would be a standardized interface based on, for example, REST and JSON. The IT organization of the government could develop a standard, possibly open source, webservice based framework which should be used by any government institution which wishes to expose its data. Standardization on webservices for external data consumers has the additional benefit that it can also be utilized for “internal” data consumers, such as data sharing between government institutions.</p>
<h2 id="governance">Governance</h2>
<p>In the past several initiatives regarding data and public health have been halted by public outcry. Ofcourse such initiatives should be well thought of and not taken lightly. However, data-science applied to public (health) data can lead to extremely valuable insights for society as a whole. Because of the potential value, it is quite likely and probably already happening that this kind of data is already being harvested and analyzed by companies. Collecting and exposing such data through a government body would ensure that this data is kept under strict control, while on the other hand also making it available for research in, for example, anonymized form. The Dutch Data Protection Authority (DPA) exists solely to protect and ensure lawful processing of personal data. A good approach would be to heavily involve this organization in development of the “open data webframework” and following implementation efforts.</p>
<h2 id="after-four-years">After four years</h2>
<p>By focusing on education, open data and governance, in four years, we might have standardized publicly accessible data and people that understand and know how to work with it. We have built a sustainable environment where data is turned into insights which can drive efficiency in government and strengthen the economic position of the Netherlands. I expect that future efforts will naturally lead to novel applications.</p>
<h1 id="building-a-data-analytics-culture">Building a Data Analytics Culture</h1>
<p><em>What would you do specifically to build a Data Analytics culture in your Ministry, that will stimulate decisions to be based on data and analysis?</em></p>
<p>Decision making processes in ministries affect the lives of many people in the country. It is therefore very important that these decisions are made on a basis which, at the least, attempts to be free of bias. Unfortunately, especially ministries are subject to shifting political views, cultural backgrounds and spatial bias. These issues emerge from various origins. The most obvious being change of leadership every four years after the elections. Cultural bias emerges in the ministries themselves as there is often a difference in education and social class between the people working in the ministries and the people who are affected by their decisions. Finally, spatial bias, is introduced by the fact that most ministries are based in The Hague, however their decisions also have effects in rural areas such as Friesland or Groningen.</p>
<h2 id="objectives">Objectives</h2>
<p>To build a data-analytics culture in ministries these biases need to be taken into account and challenged.
<em>Political biases</em> are the hardest to battle, as it requires a change in te mindset and behavior of the leadership, which are (or used to be) politicians. Government leadership needs to be made aware of their biases and tought to observe without judgment. Keep an open mind and ask questions.</p>
<p>The <em>cultural bias</em> within government can be countered by hiring people from different social and cultural backgrounds into the departments. Ofcourse the work done in ministries requires a certain level of education. This makes such work unsuitable for people of all classes, however by being aware of this bias, government officials may find other methods for engaging with the citizens who are affected by their policy.</p>
<p><em>Spatial bias</em>, as with cultural bias it might be good to diversify on the origins of the personnel of the ministry. However, it should be taken into account that people working for a longer time in a specific location might “disconnect” them from the places which are affected by their policy. However, in the age of “internet” it shouldn’t be too much of a problem to enable personnel to work remote. By working remote, personnel stays connected with the issues in remote regions and at the same time, is able to use this knowledge for delivering the right information to their department.</p>
<p>Other factors which need to be taken into account when transforming ministries into data-driven organizations are technical and skills related.</p>
<p>The technical platform should enable personnel to experiment and uncover potential bias. Existing initiatives could be extended or migrated to a new platform which fits the requirements of all involved parties.</p>
<p>Besides training personnel in getting insight in their own biases, they can be trained to develop their own models and test them on the technical platform.</p>
<h2 id="expected-outcomes">Expected Outcomes</h2>
<p>By addressing the issues with regard to bias, technical platforms and skills I expect that data-driven decision making will grow within government. Ofcourse there can always be other pitfalls which might surface during the implementation, but focusing on these key issues will give governments a good start in their transformation.</p>
<h1 id="reflection-on-organizational-efforts-to-become-data-driven">Reflection on organizational efforts to become data-driven</h1>
<p><em>Please reflect on your or your own organization’s efforts to become more data-driven, and build differentiating and more sustainable Data Analytics capabilities. What would you have done differently now, given your improved understanding of driving successful Big Data transformations?</em></p>
<p>Successfull transformations require adjustments on multiple aspects of the organization. These are adjustments on policy, culture, technique and skills. In the organization where I am consulting at the moment, there seems to be a strong focus on the culture and skills aspect. However, the policies and technical capabilities are underdeveloped. Also it seems that even for the aspects of culture and skills it is hard to nurture the development of real critical mindsets as it is common in “high performing” organizations to present the reality a lot brighter than it actually is. Teams get a basic training to be able to work with data. However, since a good platform is missing, they don’t have the opportunity to work with real data. Obviously, real insights require real and proper data. It is ofcourse not necessarily true for all teams, as the data teams themselves do get access to some data. But it is highly generalized and is hardly used for other things than support of generalized opinions.</p>
<p>As long as leadership is not critical of their own capabilities, that of their teams and -systems, “building a data mindset” within the organization does not go any further than having an army of engineers which can “hello world” python. But after that:</p>
<ol>
<li>lack the critical mindset to apply their skills correctly and be critical of their models</li>
<li>lack the tools to apply their models to production data</li>
<li>will still be turned down by a risk department which is not aligned with an offensive data-strategy</li>
</ol>
<p>Looking at these issues, I would still keep parts of the current strategy, however extend it not only to the research (data-science) teams, but especially teams who build and run business-critical applications. To enable teams which can have real world impact, some components need to be in-place.</p>
<ol>
<li>A clear open data policy (risk)</li>
<li>An open data sharing platform (technical)</li>
<li>Applied experimentation on real-world data (training)</li>
<li>An open platform where teams can discuss and challenge findings freely (without pressure from management for “delivering results”)</li>
</ol>
<p>Besides improvements in the organizational strategy I would also take more steps in creating a platform which can actually create value from data. With the investment power of a large organization it should be possible to develop the data-strategy in multiple directions at the same time. As it seems, at the moment there is a strong focus on driving the business through insights. By having teams involved in the decision making process, it should be possible for them to experiment with insights in the real environment (A/B testing). Such experimentation can be achied by implementing (human controlled) triggers in the business-process such that it is easy to test hypotheses. The results of these experiments not only can be utilized to tune the models which drive the business-processes, but also provide valuable information on which processes can actually be replaced by prediction models.</p>
<p>To summarize, I would recommend the organization to not only focus on training and culture, but also put a strong focus on development of the technical platform and an open policy. Besides, I would encourage the teams to do more practical experimentation and think about ways how they can safely experiment and apply their theories in the real environment.</p>
<h1 id="references">References</h1>
<ol>
<li><a href="https://doi.org/10.1016/j.bushor.2017.01.002">Alharthi, A., Krotov, V., Bowman, M. (2017). <em>Addressing barriers to big data.</em> Business Horizons, 60(3), 285-292</a></li>
<li><a href="https://doi.org/10.1177%2F0013916512458579">Schultz, P. W., Milfont, T. L., Chance, R. C., Tronu, G., Luís, S., Ando, K., … & Gouveia, V. V. (2014). <em>Cross-cultural evidence for spatial bias in beliefs about the severity of environmental problems.</em> Environment and Behavior, 46(3), 267-302.</a></li>
</ol>
<hr />Sudesh Jethoe, MSc. MBA.Coursework for MBA Big Data & Business Analytics, devise strategies for government and corporate enterprises on big data.Setup nagios event handlers using passive checks and NRPE2016-02-14T00:00:00+01:002016-02-14T00:00:00+01:00https://dynamicautomation.nl/articles/how-to-setup-nagios-event-handlers-using-passive-checks-and-nrpe<h1 id="introduction">Introduction</h1>
<p>Nagios event handlers allow nagios to automatically apply remedial actions when certain events occur (e.g. automatic restart of a service when it goes down).</p>
<p>The NSCA (Nagios Service Check Acceptor) daemon allows nagios to receive passive checks from hosts. Passive checks can be advantageous since they do not require the nagios server to initiate a connection to the client. Instead the client reports its results itself to the server.</p>
<p>By combining event handlers with passive checks and NRPE, nagios can apply automatic remediation when hosts get online.<br />
To configure and enable event handling on the nagios server follow these steps</p>
<h1 id="1-set-path-to-script">1. Set path to script</h1>
<p>Enable commands to be triggered when the event handler should be run on the server side</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># /etc/nagios/private/resource.cfg</span>
<span class="c"># uncomment $USER2$, as the path to the event handlers</span>
<span class="nv">$USER2$=</span>/usr/lib64/nagios/plugins/eventhandlers
</code></pre></div></div>
<h1 id="2-place-the-server-side-event-handler-in-the-user2-path">2. Place the server-side event handler in the $USER2$ path</h1>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/usr/lib64/nagios/plugins/eventhandlers/handle_windows
</code></pre></div></div>
<h1 id="3-add-handle_windows-as-a-new-command-in-nagiosql">3. Add handle_windows as a new command in nagio(sql)</h1>
<p>See also: <a href="https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/3/en/macrolist.html#servicestate" target="_blank">https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/3/en/macrolist.html#servicestate</a></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Command: handle_windows
Commandline: <span class="nv">$USER2$/</span>handle_windows <span class="nv">$SERVICESTATE$ $SERVICESTATETYPE$ $SERVICEATTEMPT</span><span class="err">$</span>
<span class="nv">$HOSTADDRESS$ $SERVICEDISPLAYNAME$ $SERVICEDESC</span><span class="err">$</span>
define <span class="nb">command</span> <span class="o">{</span>
command_name handle_windows
command_line <span class="nv">$USER2$/</span>handle_windows <span class="nv">$SERVICESTATE</span><span class="err">$</span>
<span class="nv">$SERVICESTATETYPE$ $SERVICEATTEMPT$ $HOSTADDRESS$ </span>&nbsp<span class="p">;</span> &nbsp<span class="p">;</span> <span class="nv">$SERVICEDISPLAYNAME$ $SERVICEDESC</span><span class="err">$</span>
register 1
<span class="o">}</span>
</code></pre></div></div>
<h1 id="4-create-additional-service-in-nagiosql">4. Create additional service in Nagiosql</h1>
<p>Go to the services tab in nagiosql and</p>
<ol>
<li>create a new service with a dummy check (will be reported by NSCA)</li>
<li>make sure the service check is configured and available on the client (next section)</li>
<li>enable the “Event handler” for this service (and select the right one (handle_windows)</li>
</ol>
<h1 id="5-start-services">5. Start services</h1>
<p>Now make sure that both Nagios and the NSCA client are active and that Nagios is able to execute commands via NRPE and receive results from NSCA (check nagios.cfg for this).</p>
<h1 id="6-configuring-the-nagios-client">6. Configuring the Nagios client</h1>
<p>For Windows setup NSClient++
Make sure the following is configured:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#### nagios.cfg ####</span>
<span class="o">[</span>/modules]
CheckExternalScripts <span class="o">=</span> 1
CheckSystem <span class="o">=</span> 1
NRPEServer <span class="o">=</span> 1
NSCAClient <span class="o">=</span> 1
<span class="nv">Scheduler</span><span class="o">=</span> 1
<span class="o">[</span>/settings/default]
allowed hosts <span class="o">=</span> <span class="c">#ipaddress of nagios server</span>
<span class="nb">timeout</span> <span class="o">=</span> <span class="c">##</span>
<span class="c"># enable NRPE daemon</span>
<span class="o">[</span>/settings/NRPE/server]
insecure <span class="o">=</span> <span class="nb">true
</span>allow arguments <span class="o">=</span> <span class="nb">true
</span>port <span class="o">=</span> 5666</pre>
<span class="o">[</span>/settings/NSCA/client]
<span class="nb">hostname</span> <span class="o">=</span> auto</pre>
<span class="o">[</span>/settings/NSCA/client/targets/default]
address <span class="o">=</span> <span class="c">#ipadress of nagios server</span>
encryption <span class="o">=</span> 0
<span class="c"># set targets of external scripts for service checks and remediation</span>
<span class="o">[</span>/settings/external scripts/scripts]
handle_domaintrust <span class="o">=</span> cmd /c <span class="nb">echo </span>scripts<span class="se">\h</span>andle_my_issue.ps1 <span class="p">;</span> <span class="nb">exit</span><span class="o">(</span><span class="nv">$lastexitcode</span><span class="o">)</span> | powershell.exe <span class="nt">-command</span> -
check_my_issue <span class="o">=</span> cmd /c <span class="nb">echo </span>scripts<span class="se">\c</span>heck_my_issue.ps1 <span class="p">;</span> <span class="nb">exit</span><span class="o">(</span><span class="nv">$lastexitcode</span><span class="o">)</span> | powershell.exe <span class="nt">-command</span> -</pre>
<span class="c"># set the passive check run interval</span>
<span class="o">[</span>/settings/scheduler/schedules/default]
interval <span class="o">=</span> 30s</pre>
<span class="c"># schedule the check</span>
<span class="o">[</span>/settings/scheduler/schedules]
check_my_issue <span class="o">=</span> check_my_issue
</code></pre></div></div>
<h1 id="debugging">Debugging</h1>
<p>To debug</p>
<ol>
<li>tail -f nagios.log on the nagios server</li>
<li>net stop nscp on the client</li>
<li>nscp.exe test on the client</li>
<li>Now trigger the scripts / commands from either the server or the client (inside the nscp console)
(type queries to see if all scripts are listed)</li>
</ol>Sudesh Jethoe, MSc. MBA.Introduction Nagios event handlers allow nagios to automatically apply remedial actions when certain events occur (e.g. automatic restart of a service when it goes down).Installation SQLplus2016-02-14T00:00:00+01:002016-02-14T00:00:00+01:00https://dynamicautomation.nl/articles/installation-sqlplus<p>For a customer I was required to extract and report data from an Oracle database.
Ofcourse I want to automate this task to the full extent.
Normally I would use a scripting language like bash/perl/python for this task.
Unfortunately, the environment did not allow for “easy” retrieval of external modules to interface with an Oracle database.<br />
However, the environment did allow access tot the Oracle SQLplus instantclient.
The instantclient allows running SQL-queries directly against Oracle databases,
much like running “mysql -u root -p$password $mydatabase” in a MySQL environment.<br />
By piping or redirecting commands to such a “connection” we are able to extract the required information from the database.</p>
<p>Below I will outline the required steps and after I will show you snippets of the code I used for the reporting mechanism.</p>
<h3 id="install-oracle-client">Install oracle client</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>yum <span class="nb">install </span>oracle-instantclient11.2-sqlplus-11.2.0.3.0-1.i386
</code></pre></div></div>
<h3 id="create-oracle-user">Create oracle user</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>useradd oracle
</code></pre></div></div>
<h3 id="edit-database-configuration">Edit database configuration</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># /usr/lib/oracle/12.1/client64/network/admin/tnsnames.ora</span>
TDIGJ <span class="o">=</span>
<span class="o">(</span>DESCRIPTION <span class="o">=</span>
<span class="o">(</span>ADDRESS_LIST <span class="o">=</span>
<span class="o">(</span>ADDRESS <span class="o">=</span> <span class="o">(</span>PROTOCOL <span class="o">=</span> TCP<span class="o">)(</span>HOST <span class="o">=</span> <span class="k">${</span><span class="nv">IPADDRESS</span><span class="k">}</span><span class="o">)(</span>PORT <span class="o">=</span> <span class="k">${</span><span class="nv">PORT</span>:1521<span class="k">}</span><span class="o">))</span>
<span class="o">)</span>
<span class="o">(</span>CONNECT_DATA <span class="o">=</span>
<span class="o">(</span>SID <span class="o">=</span> <span class="k">${</span><span class="nv">SID</span><span class="k">}</span><span class="o">)</span>
<span class="o">)</span>
<span class="o">)</span>
</code></pre></div></div>
<h3 id="test">Test</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo</span> <span class="s2">"SELECT table_name FROM DICTIONARY ORDER BY table_name;"</span> | <span class="se">\</span>
sqlplus <span class="k">${</span><span class="nv">USER</span><span class="k">}</span>@<span class="k">${</span><span class="nv">DBNAME</span><span class="k">}</span>/<span class="k">${</span><span class="nv">DBPASSWD</span><span class="k">}</span>
</code></pre></div></div>Sudesh Jethoe, MSc. MBA.For a customer I was required to extract and report data from an Oracle database. Ofcourse I want to automate this task to the full extent. Normally I would use a scripting language like bash/perl/python for this task. Unfortunately, the environment did not allow for “easy” retrieval of external modules to interface with an Oracle database. However, the environment did allow access tot the Oracle SQLplus instantclient. The instantclient allows running SQL-queries directly against Oracle databases, much like running “mysql -u root -p$password $mydatabase” in a MySQL environment. By piping or redirecting commands to such a “connection” we are able to extract the required information from the database.Utilizing lvm thin provisioned snapshots2016-02-14T00:00:00+01:002016-02-14T00:00:00+01:00https://dynamicautomation.nl/articles/utilizing-lvm-thin-provisioned-snapshots<p>To fully utilize the few bits of space which is provided by my SSD, I always take the following approach:</p>
<ol>
<li>Create an “as large as possible” LVM volume group to hold all my data.</li>
<li>Create a thin_pool within the volume group for my virtual machines.</li>
<li>Create one “base VM” which contains just a bare installation of my preferred OS (inside the thinpool).</li>
<li>Now for each VM I need for my labs I create a snapshot from the base VM</li>
</ol>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>lvcreate <span class="nt">--snapshot</span> <span class="nt">--name</span> thin_snapshot vg/thin_base
</code></pre></div></div>
<ol>
<li>Now to ensure that I can actually utilize the newly created snapshot I have to activate it first.</li>
</ol>
<p>LVM (thin) snapshot are “by default” created in inactivated mode, but it also has the “<a href="http://man7.org/linux/man-pages/man7/lvmthin.7.html#Thin_Topics" target="_blank">activation skip flag</a>”. This flag prevents the system from activing the partition with the usual commands. To active the partition use:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>lvchange <span class="nt">-ay</span> <span class="nt">-K</span> vg/thin_snapshot
</code></pre></div></div>
<p>To remove the skip flag completely add:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>lvchange <span class="nt">-kn</span> vg/thin_snapshot
</code></pre></div></div>
<p>Finally the newly added thinly provisioned snapshot is available for usage by my virtualisation tool.</p>Sudesh Jethoe, MSc. MBA.To fully utilize the few bits of space which is provided by my SSD, I always take the following approach: Create an “as large as possible” LVM volume group to hold all my data. Create a thin_pool within the volume group for my virtual machines. Create one “base VM” which contains just a bare installation of my preferred OS (inside the thinpool). Now for each VM I need for my labs I create a snapshot from the base VMPreparing Powershell Development Environment2015-11-02T00:00:00+01:002015-11-02T00:00:00+01:00https://dynamicautomation.nl/articles/preparing-powershell-scripting-development-environment<p>The Windows Powershell scripting language is a powerful advancement from older scripting language in Windows (Batch, VBScript). The language integrates features from other scripting languages (bash) and it is able to utilize existing (.Net) libraries.<br />
A big difference between other scripting languages and Powershell is that it is fully object-based and not text-based. Therefore it is important to keep in mind that what you might see as output on your screen is only a representation of the object, but not the object itself.<br />
In this post I will explain the basic steps to set up a “sane” working environment, which allows you similar experience as a Bash shell with TMUX.</p>
<p>In specific I will discuss the following tools/modules:</p>
<ul>
<li>ConEmu</li>
<li>Environment Settings</li>
<li>Powershell Profile</li>
<li>Using Modules</li>
<li>PSGet / NuGet</li>
<li>PSReadline</li>
</ul>
<h1 id="conemu">ConEmu</h1>
<p><a href=""https://conemu.github.io/"">ConEmu</a> is a console emulator with tabs and panes.<br />
It is able to provide a similar management experience as <a href=""https://tmux.github.io/"">Tmux</a> on Linux.</p>
<p>For usage with Powershell I use the following settings in ConEmu:</p>
<ul>
<li>Startup
<ul>
<li><code class="language-plaintext highlighter-rouge">"Specified named task" = {Shells::Powershell}</code></li>
<li>Tasks > <code class="language-plaintext highlighter-rouge">"5 {Shells::Powershell}" = "C:\Windows\System32\WindowsPowershell\v1.0\powershell.exe -new_console:d:C:\gitrepo\powershell"</code></li>
<li>Environment > <code class="language-plaintext highlighter-rouge">"Set up environment variables" = "set HOME=C:\gitrepo\powershell"</code></li>
</ul>
</li>
<li>Features
<ul>
<li><code class="language-plaintext highlighter-rouge">Colors = <xterm></code></li>
</ul>
</li>
<li>Keys & Macro
<ul>
<li>create new split /vertical /horizontal</li>
<li>new tab</li>
<li>zoom</li>
<li>switch windows</li>
</ul>
</li>
</ul>
<p>The above settings ensure that any new shell is spawned from the directory containing my Powershell scripts and also allows to run scripts directly from this path.</p>
<h1 id="environment-settings">Environment Settings</h1>
<p>Only one environment variable should be set for Powershell, namely the “PSModulePath” variable. This variable allows the usage of modules (and functions within these modules) straight from any Powershell CLI, it doesn’t matter whether it is the powershell.exe, the Powershell ISE, or a custom execution from a different path.</p>
<p>To set the variable go the following:</p>
<div class="language-powershell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># get current path</span><span class="w">
</span><span class="nv">$curpath</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="n">Environment</span><span class="p">]::</span><span class="n">GetEnvironmentVariable</span><span class="p">(</span><span class="s2">"PSModulePath"</span><span class="p">)</span><span class="w">
</span><span class="c"># update path with new path</span><span class="w">
</span><span class="p">[</span><span class="n">Environment</span><span class="p">]::</span><span class="n">SetEnvironmentVariable</span><span class="p">(</span><span class="w"> </span><span class="s2">"PSModulePath"</span><span class="p">,</span><span class="w"> </span><span class="s2">"C:\gitrepo\powershell\modules;"</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nv">$curpath</span><span class="p">)</span><span class="w">
</span><span class="c"># check if settings were updated correctly</span><span class="w">
</span><span class="n">get-childitem</span><span class="w"> </span><span class="nx">env:psmodulepath</span><span class="w">
</span></code></pre></div></div>
<p>Source: <a href=""https://technet.microsoft.com/en-us/library/dd878326(v=vs.85).aspx"">MS TechNet</a></p>
<h1 id="powershell-profile">Powershell Profile</h1>
<p>Powershell also allows you to specify a set of commands which will be run before spawning a new shell. This is convenient for pre-loading modules, setting aliases and setting the path.</p>
<p>The powershell powerfile can be set up by creating a new textfile in the following location:</p>
<div class="language-powershell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># $HOME\Documents\WindowsPowerShell\Microsoft.Powershell_profile.ps1</span><span class="w">
</span><span class="c"># powershell profile</span><span class="w">
</span><span class="c">#set path</span><span class="w">
</span><span class="nv">$</span><span class="nn">env</span><span class="p">:</span><span class="nv">Path</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s2">"C:\gitrepo\powershell"</span><span class="w">
</span><span class="c"># import modules</span><span class="w">
</span><span class="n">import-module</span><span class="w"> </span><span class="nx">psreadline</span><span class="w">
</span><span class="c"># set aliases</span><span class="w">
</span><span class="n">set-alias</span><span class="w"> </span><span class="nx">vim</span><span class="w"> </span><span class="s1">'C:\Program Files (x86)\Vim\Vim74\vim.exe'</span><span class="w">
</span><span class="c"># To edit the Powershell Profile</span><span class="w">
</span><span class="kr">Function</span><span class="w"> </span><span class="nf">Edit-Profile</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">vim</span><span class="w"> </span><span class="nv">$profile</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="c"># To edit Vim settings</span><span class="w">
</span><span class="kr">Function</span><span class="w"> </span><span class="nf">Edit-Vimrc</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="n">vim</span><span class="w"> </span><span class="nx">D:\powershell\_vimrc</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Besides some convenient shortcuts, the most important module I load here is the PSReadline module.</p>
<h1 id="using-modules">Using Modules</h1>
<p>Powershell modules allow you to reuse existing code of yourself and others.<br />
Modules are basically plain powershell scripts without any directly executed statements, but merely (groups of) functions. Also the extension ends with .psm1 instead of .ps1 .</p>
<p>Powershell modules can also contain some documentation and things like author and license however this is not required.</p>
<p>To be able to use modules the PSModulePath needs to be set and your modules should be available in this path.</p>
<p>Alternatively, from Powershell 4, the following path is <a href="https://technet.microsoft.com/en-us/library/dd878350(v=vs.85).aspx" target="_blank" rel="noopener noreferrer">added by default</a>, which can be used for global modules:</p>
<div class="language-powershell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$EnvProgramFiles</span><span class="n">\WindowsPowerShell\Modules\</span><span class="w">
</span></code></pre></div></div>
<h1 id="psget">PSGet</h1>
<p>PSGet is a module and repository of useful powershell modules, a.o. the PSReadline module, to install it follow the instructions <a href="http://psget.net/" target="_blank" rel="noopener noreferrer">on the site</a></p>
<h1 id="psreadline">PSReadline</h1>
<p>PSReadline enables readline support for Powershell terminals. The features are a.o.</p>
<ul>
<li>enhanced tab completion</li>
<li>history</li>
<li>history search (Ctrl+r)</li>
<li>command line editing</li>
</ul>
<p>See also Scripting Guy’s blog at: <a href="http://blogs.technet.com/b/heyscriptingguy/archive/2014/06/17/a-better-powershell-command-line-edit.aspx">http://blogs.technet.com/b/heyscriptingguy/archive/2014/06/17/a-better-powershell-command-line-edit.aspx</a></p>
<p>To install PSReadline, first install PSGet, then execute the following command in your environment:</p>
<div class="language-powershell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># install psreadline with psget</span><span class="w">
</span><span class="n">install-module</span><span class="w"> </span><span class="nx">PsReadLine</span><span class="w">
</span></code></pre></div></div>Sudesh Jethoe, MSc. MBA.The Windows Powershell scripting language is a powerful advancement from older scripting language in Windows (Batch, VBScript). The language integrates features from other scripting languages (bash) and it is able to utilize existing (.Net) libraries. A big difference between other scripting languages and Powershell is that it is fully object-based and not text-based. Therefore it is important to keep in mind that what you might see as output on your screen is only a representation of the object, but not the object itself. In this post I will explain the basic steps to set up a “sane” working environment, which allows you similar experience as a Bash shell with TMUX. [Environment]::SetEnvironmentVariable( “PSModulePath”, “C:\gitrepo\powershell\modules;” + $curpath)Installing an SSD (Fedora 16)2015-03-08T00:00:00+01:002015-03-08T00:00:00+01:00https://dynamicautomation.nl/articles/installing-an-ssd-fedora-16<p>I just bought an SSD drive and I want to make sure it works best as it can.<br />
I migrate my old / (ext4) partition to this drive.<br />
For best performance I decide to switch to a GPT-based disk layout and use btrfs for the filesystem.\</p>
<h1 id="install-ssd-and-configure-partition-table">Install SSD and configure partition table</h1>
<p>First step is to physically install the disk into your laptop<br />
After, boot from livecd/usb (systemrescuecd) and create GPT partition table and partitions, using gdisk (g fdisk).<br />
Create at least two partitions!</p>
<ul>
<li>..BIOS_boot partition code: EF02 size: 1 MiB</li>
<li>..Linux/Windows data, root partition code: 0700 size: any</li>
</ul>
<p><strong>! gdisk automatically aligns partitions on 2048-sector boundaries for best performance.</strong></p>
<h1 id="format-root-partition-as-btrfs">Format root partition as btrfs</h1>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mkfs.btrfs <span class="nt">-L</span> MyLinuxOS /dev/sda2
</code></pre></div></div>
<h1 id="copy-data-from-old-to-new-partition">Copy data from old to new partition</h1>
<p>Copy data from old partition to new partition using rsync with -a(rchive) option to make sure all filepermissions are preserved.</p>
<h1 id="install-grub2-on-new-partition">Install Grub2 on new partition</h1>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mount /dev/sda2 /mnt
mount <span class="nt">-o</span> <span class="nb">bind</span> /dev /mnt/dev
mount <span class="nt">-t</span> proc /proc /mnt/proc
mount <span class="nt">-t</span> sysfs /sys /mnt/sys
<span class="nb">chroot</span> /mnt
<span class="c"># possibly fedora specific:</span>
grub2-mkconfig <span class="nt">-o</span> /boot/grub2/grub.cfg
grub2-install /dev/sda
</code></pre></div></div>
<h1 id="edit-fstab-to-use-on-startup">Edit fstab to use on startup</h1>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># /etc/fstab #</span>
<span class="nv">LABEL</span><span class="o">=</span>Fedora / btrfs defaults,noatime,discard,ssd 0 0
</code></pre></div></div>
<h1 id="sources">Sources</h1>
<ul>
<li><a href="https://wiki.archlinux.org/index.php/Solid_State_Drives">https://wiki.archlinux.org/index.php/Solid_State_Drives</a></li>
<li><a href="http://tincman.wordpress.com/2011/01/20/installing-arch-linux-onto-a-gpt-partitioned-btrfs-root-ssd-on-a-legacy-bios-system/">http://tincman.wordpress.com/2011/01/20/installing-arch-linux-onto-a-gpt-partitioned-btrfs-root-ssd-on-a-legacy-bios-system/</a></li>
<li><a href="http://fedoraproject.org/wiki/Grub2">http://fedoraproject.org/wiki/Grub2</a></li>
<li><a href=""https://wiki.archlinux.org/index.php/Solid_State_Drives"">ArchLinux SSD tutorial</a></li>
</ul>Sudesh Jethoe, MSc. MBA.I just bought an SSD drive and I want to make sure it works best as it can. I migrate my old / (ext4) partition to this drive. For best performance I decide to switch to a GPT-based disk layout and use btrfs for the filesystem.\