Friday, April 21, 2017

Puppet RSpec set up and testing with Puppet Forge module dependencies

Puppet & RSpec

As at the 21st April 2017, Puppet and RSpec testing is still an issue, with .fixtures.yml not full functioning as it should, that is loading dependencies into the spec/fixtures directory.

Even when you place your .fixtures.yml in the root of your project directory, which is to say the module where the spec directory lives along with the modules and manifest, and you run the puppet-spec-init or rspec or rake commands your fixtures file is ignored, and the only module linked is the current module.

It is essential that after creating your module with the puppet module generate command you will get the necessary layout for your puppet module.  Once you start to build your module you’ll eventually want to add some code spec tests to ensure that someone doesn’t break the existing code.

ServerSpec tests can be run independently of your code, so the key difference is that RSpec is there to perform tests on your code and not the final server build.

To fix this means you still need to deal with all your module dependencies manually, so here are the steps.

Example project space;

The Rakefile

require 'rspec-puppet/rake_task'
begin
  if Gem::Specification::find_by_name('puppet-lint')
    require 'puppet-lint/tasks/puppet-lint'
    PuppetLint.configuration.send('disable_autoloader_layout')
    PuppetLint.configuration.ignore_paths = ["spec/**/*.pp""vendor/**/*.pp"]
    task :default => [:rspec, :lint]
  end
rescue Gem::LoadError
  task :default => :rspec
end

This file controls what will happen when the tests run, such as in this example a linter will run against your code first, and certain files will be ignores by the linter as they are not puppet code.

The spec_helper.rb file

This file defines other files required to help perform the testing, and where the files to test can be located.

require 'rspec-puppet/spec_helper' 

fixture_path = File.expand_path(File.join(__FILE__, '..', 'fixtures')) 

RSpec.configure do |c| 
  c.module_path = File.join(fixture_path, 'modules') 
  c.manifest_dir = File.join(fixture_path, 'manifests') 
  c.environmentpath = File.join(Dir.pwd, 'spec') 
end


Preparing the module dependencies

First let’s take a look at a sample directory layout of what is required;

mysqlmodule
├── Gemfile
├── Gemfile.lock
├── manifests
│   └── init.pp
├── Rakefile
├── spec
│   ├── classes
│   │   └── mysqltest_spec.rb
│   ├── defines
│   ├── fixtures
│   │   ├── manifests
│   │   │   └── site.pp
│   │   └── modules
│   │       ├── mysql
│   │       │   ├── lib -> ../../../../../mysql/lib
│   │       │   ├── manifests -> ../../../../../mysql/manifests
│   │       │   └── templates -> ../../../../../mysql/templates
│   │       ├── mysqlmodule
│   │       │   ├── manifests -> ../../../../manifests
│   │       │   └── templates -> ../../../../templates
│   │       ├── staging
│   │       │   ├── files -> ../../../../../staging/files
│   │       │   ├── libs -> ../../../../../staging/libs
│   │       │   └── manifests -> ../../../../../staging/manifests
│   │       └── stdlib
│   │           ├── lib -> ../../../../../stdlib/lib
│   │           ├── manifests -> ../../../../../stdlib/manifests
│   │           └── types -> ../../../../../stdlib/types
│   ├── functions
│   ├── hosts
│   └── spec_helper.rb
└── templates

You’ll notice that in this module the user has made use of PuppetForge modules for MySQL, which in itself has other dependencies on Staging and StdLib. For RSpec to work these fixtures need to be available during the test to enable checking of your own modules code.  For example;

it { is_expected.to contain_class_mysql__db(‘mydb’) }

This would look for  mysql::db{‘mydb’:} in your code.
Since the .fixture.yml is not being read at the moment, you should manually add the dependencies to your 
spec/fixtures/modules directory.  Let’s take the MySQL module.

In our modules directory, that is where we would see mysqlmodule if listed, we would ensure that we have done a puppet module install of those modules require, and making sure we note the dependencies that have been downloaded too.

Here is the layout of the MySQL puppetlabs/mysql module;

mysql
├── CHANGELOG.md
├── checksums.json
├── CONTRIBUTING.md
├── examples/
├── Gemfile
├── lib/
├── LICENSE
├── manifests/
├── metadata.json
├── NOTICE
├── Rakefile
├── README.md
├── spec/
├── templates/
└── TODO

In our fixtures directory under modules we will create a directory called mysql and within that directory you will create symlinks back to the essential directories and files that make up the module.  In this case we will symlink to;
- lib
- manifests
- templates

We would then do the same for the other 2 dependent modules stdlib and staging to ensure that our tests will work with all elements.

Overcoming variable issues

When running rspec after configuring your module dependencies, you may then find that your test still fails.  Right near the top of the output you might see something like;

F

Failures:

  1) mysqlmodule should contain File_line[/etc/my.cnf] with bind-address => "0.0.0.0"
     Failure/Error: it {is_expected.to contain_file_line('/etc/my.cnf').with('bind-address' => '0.0.0.0')}
     
     Puppet::PreformattedError:
       Evaluation Error: Unknown variable: '::osfamily'. at /home/steve/web/puppet/modules/mysqlmodule/spec/fixtures/modules/mysql/manifests/params.pp:36:8 on node tpslaptop

From this output we are interested in the line that comes after Puppet::PreformatedError which states that the variable ::osfamily is missing.  This is actually a Puppet Facter variable, and would normally be collected by the Puppet Master during the compilation of the Puppet catalog.  We therefore need to define all of the values for the Facts in our classes spec test.

Our spec test file is called mysqltest_spec.rb in spec/classes, and originally looked like;

require 'spec_helper'
describe 'mysqlmodule' do
  it {is_expected.to contain_file_line('/etc/my.cnf').with('bind-address' => '0.0.0.0')}
end
  
The test is using the file_line type from the mysql module, and we need to tell our test to set the osfamily facter variable before this is run.  The change to the code that will allow the variable to be set is;

let(:facts) do
  { :osfamily => RedHat}
end

This piece of code is added within the describe block before the tests, and we have provided a value.

We would need to add each variable into the block as they appear during our tests.  For example, the next one to pop up would be ::operatingsystem, so we would have the following code;

require 'spec_helper'
describe 'mysqlmodule' do
  let(:facts) do
{
  :osfamily => 'RedHat' ,
      :operatingsystem => 'RedHat' 
    }
  end
  it {is_expected.to contain_file_line('/etc/my.cnf').with('bind-address' => '0.0.0.0')}
end

After running through the test we eventually find that to test the file_line type for the MySQL module we would need the following;

describe 'mysqlmodule' do
  let(:facts) do
    { :osfamily => 'RedHat',
    :operatingsystemmajrelease => '7.2',
    :root_home => '',
    :puppetversion => '4.10.0',
    :operatingsystem => 'RedHat' }
  end

  it {is_expected.to contain_file_line('Edit my.cnf')}
end

This will finally result in your module being checked, or at least the file_line type;

.

Finished in 2.24 seconds (files took 0.93215 seconds to load)
1 example, 0 failures



Now you repeat for the other modules.

Friday, March 24, 2017

Getting installed services in RHEL

Today I was asked how you could discover those services that are installed on a system.

For packages installed today;
rpm -ql $(rpm -qa --last | 
grep "$(date +'%a %d %b %Y')" | 
awk '{print $1}') | 
grep systemd | 
grep service | 
awk -F"/" '{print $NF}' | 
grep -v '@'

From the beginning of time;
rpm -ql $(rpm -qa --last | 
awk '{print $1}') | 
grep systemd | 
grep service | 
awk -F"/" '{print $NF}' | 
grep -v '@'

From using these you can then ask which are enabled on start up;

systemctl list-unit-files --type=service |
grep enabled |
egrep $(
  rpm -ql $(rpm -qa --last | 
  grep "$(date +'%a %d %b %Y')" | 
  awk '{print $1}') | 
  grep systemd | 
  grep service | 
  awk -F"/" '{print $NF}' | 
  grep -v '@' | 
  tr '\n' '|' | 
  sed 's/|$//'
)

Tuesday, May 24, 2016

Dead Icons on Mac OS X

Having used a Mac for the last 6 months again (how annoying), I went through the process of clearing it up ready for the next person to use (as I'd never personally own an Apple Mac, since I was the person who coined the term "Crapple Mac").

However, in the process of working out how to remove applications that were not installed through iTunes (I have no intention of putting my credit card details into a laptop I don't own, or to Apple), most of my applications were installed via brew.

I had the following shell script to install, search or delete my applications;
#!/bin/bash

if (( $# < 2 ))
then
echo "SYNTAX: $0 action package" >&2
echo "Where action is one of (search|install|uninstall)" >&2
exit 1
fi

action=$1
shift
package="$@"

case $action in
'install')
brew cask $action $package --force
;;
        remove|erase)
                brew cask uninstall $package
;;
*)
brew cask $action $package
;;

esac

So when coming to delete the applications were not always easy to remove with brew, as some were downloaded over the Internet.

So if brew odes not remove the application you can try one of the following 3 options;
  1. The simple method of clicking on Launchpad and then click and hold on the application to remove.  All the icons will then wobble.  If you can remove the application here then an X will appear against the application.  Clicking it will allow you to remove it.
  2. Open the application.  Then bring up the menu for the application in the dock so that you can "show in finder".  Then quit the application (don't just close, you need to quit it) and then drag the .app from Finder into the waste basket and then empty trash.
  3. This final one will help remove application Icons where the application has gone, but the icon refuses to budge from Launchpad.  The following describes how to do this.
To remove an icon or application still showing in Launchpad that could not be remove using steps 1 or 2 do the following;

1. To find the application and see if it is in Launchpad run the following command
for x in $(find /private/var/folders -name *launchpad*)
do
  sqlite3 $x/db/db "SELECT * FROM apps;"
done | grep -i appName

2. In the 2nd column is the title.  This next command requires you to type the title in in the exact case.  So if for example you have an application called Thunderbird you would delete it as follows;

for x in $(find /private/var/folders -name *launchpad*)
do
  sqlite3 $x/db/db "DELETE FROM apps WHERE title='Thunderbird';"
done
Replacing Thunderbird with the title you got from Step 1.

Your launchpad will now start to look cleaner.

Monday, April 4, 2016

GPG Sharing encrypted files

Occasionally you want to share a secret file, be it a password, or some credentials, or even somethings else with colleague(s) over the unsecure internet or via Email.

gpg is a useful program for achieving this but it does require you to have the public key of the person you are intending to encrypt the file for, or a shared public key to which everyone has the private one.


You ENCRYPT: gpg -z0 --output message.gpg --encrypt --recipient “Person-B" message.txt
Colleague DECRYPT: gpg --output message --decrypt message.gpg

Sunday, March 20, 2016

Making Live CDs with latest updates

Not too long back I bought a new laptop, and as always needed to put my favourite Linux OS on it to keep ahead of the enterprise version.  If you haven't guessed I'm referring to Fedora.  Being about 3 years ahead of what is about to hit RedHat is extremely useful.

However the laptop that I decided to buy, from pcspecialist looked like a Mac Airbook, or whatever they call their slimest one, but with the latest and greatest faster i7 6th Gen processor, etc.  So in short far superior to the Apple, but less than 1/2 the price at £620.

When it arrived, it was excellent - F23 installed, but after a short while it started to hang if it was stressed, so diverted over to Elementary OS, which although pretty, was a little old from a Kernel perspective and lacking some features.

Last weekend I had some time to sit down, since F23 wouldn't run as a Live USB stick from download I decided to build my own spin.  I've been building kickstart systems for a very long time, especially over networks and also on DVDs, however in recent times the process has changed a little since you can't get the entire OS DVD, but have to create a live one.

So using some useful sites around livecd-tools decided to install it and create F23 spin with all of the latest updates taken from rpmfusion.  Here is how it is done on a Fedora system;

1.Install livecd-tools
    1. sudo dnf -y install livecd-tools fedora-kickstarts
      1. The fedora-kickstarts will provide the basis for your files to build the livecd
2. Create a directory to perform the work and store your final ISO image
    1. mkdir -p $HOME/livecds/fc23
      1. I've added fc23 so that I can build different versions
    2. I also created a script called $HOME/livecds/livecd so that I don't have to remember the options;

#!/bin/bash

# URL location
# https://fedoraproject.org/wiki/How_to_create_and_use_a_Live_CD
if (( $# < 3 ))
then
  echo "Syntax: $0 " >&2
  exit 1
fi

pathToKickstartFileks="$1"
tmpcacheLocation="$2"
cdLabel="$3"

livecd-creator --verbose --config="$pathToKickstartFileks" --fslabel="CDLabel" \
--cache="$tmpcacheLocation"

3. The main element to creating a live CD with the latest kernel, etc is to include the repositories of rpmfusion, and to add the necessary packages to the kickstart file.
  1. Copy the example kickstart base file for Fedora 23 so that you can modify it;
    1. cp /usr/share/spin-kickstarts/fedora-live-workstation.ks $HOME/livecds/fc23
  2. Modify the kickstart file, changing the following to make it fit on an 8GB USB
    1. part / --size 8192  --fstype ext4 --grow
      1. The original only allows for a 4GB and doesn't have grow
    2. Under the %packages section you can add your extra Groups or individual packages that you need.  Note if you are adding a group then something like "Development tools" should be written as development-tools
    3. If you don't want a package then precede it with a hyphen (-)
      1. e.g. -shotwell
    4. You will also notice in this file the line;
      1. %include fedora-repo.ks
      2. Create this file with the following for F23, change version based on OS
repo --name=fedora --mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-23&arch=$basearch
repo --name=fedora-updates --mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=updates-released-f23&arch=$basearch
repo --name=fpmfusionfreerawhide --mirrorlist=http://mirrors.rpmfusion.org/mirrorlist?repo=free-fedora-rawhide&arch=$basearch
repo --name=rpmfusionnonfreerawhide --mirrorlist=http://mirrors.rpmfusion.org/mirrorlist?repo=nonfree-fedora-rawhide&arch=$basearch
repo --name=rpmfusionfree --mirrorlist=http://mirrors.rpmfusion.org/mirrorlist?repo=free-fedora-23&arch=$basearch
repo --name=rpmfusionfreeupdates --mirrorlist=http://mirrors.rpmfusion.org/mirrorlist?repo=free-fedora-updates-released-23&arch=$basearch
repo --name=rpmfusionnonfree --mirrorlist=http://mirrors.rpmfusion.org/mirrorlist?repo=nonfree-fedora-23&arch=$basearch
repo --name=rpmfusionnonfreeupdate --mirrorlist=http://mirrors.rpmfusion.org/mirrorlist?repo=nonfree-fedora-updates-released-23&arch=$basearch
repo --name=google --baseurl=http://dl.google.com/linux/chrome/rpm/stable/x86_64

You'll note that I've also included Chrome.  Each repo is one complete line.  The key ones are the rpmfusion and the updates so that we get the latest and greatest software installed.

4. To build the ISO image simply run your livecd script from within the F23 directory;
  1. $HOME/livecds/livecd ../fedora-live-base.ks ../cache F23-livecd
    1. This command then connects with the relevant repositories to download the rpms and builds the DVD ISO image
    2. You should watch the early output since it will tell you if you have any mistakes with package names or groups
5. Finally put the image on to you USB stick with;
  1. dd if=F23-livecd.iso of=/dev/sdd
    1. Where /dev/sdd should be changed to your USB drive
    2. If you don't know how to find out then;
      1. df -h
        1. Get a list of all the Filesystems
      2. Insert the USB stick
        1. df -h
        2. Look for the Filesystems that was not their previously
      3. Unmount the USB stick so you can write to it with dd
        1. sudo umount /dev/sdd

Troubleshooting

In most cases you are likely to see the following error if the partition size in the kickstart file is not large enough for the image.  If this is the case you will see;
Error creating Live CD : Unable to install: Could not run transaction.
Unmounting directory /var/tmp/imgcreate-vo3YLW/install_root
Losetup remove /dev/loop1
In the above case you should increase the part --size until the error no longer appears.

If you have an incorrect package you will see something like;
Retrieving http://www.mirrorservice.org/sites/download1.rpmfusion.org/nonfree/fedora/updates/testing/23/x86_64/repodata/d5d34f8bad2e14babefd4ff808715d34cdd5637475759a76c58852f211892340-primary.sqlite.xz ...OK
Skipping missing group 'Web Server'
Edit the fedora-live-base.ks file and modify the name in the %package section.

Sunday, November 8, 2015

PCSpecialist Lafite II - Fedora 22 - Backlight fixed

I recently purchased the Lafite II from PCSpecialist  with a high spec;
- i7-6500U
- Intel 520 Core Skylake integrated graphics
- 8GB Ram
- 250GB Samsung 850 EVO SSD, SATA 6GB/s (540MBs R/520MBs W)

URL: http://www.pcspecialist.co.uk/notebooks/lafiteII/

This laptop uses the same Aluminimum chassis as the Apple MacBook 13.3, but the spec of the internals is obviously way more than that of the Apple and for 1/2 the price of their top spec.  Given that Apple still only use an inferior i5 processor (4th Gen) and not even an i7, this laptop is far superior.

I got it to run Linux as I always do, and in this case Fedora 22.  Installing F22 was the easiest install ever, and with the SSD doesn't take hours to do the fsck that it does on standard hard disks.  After installing from a live USB version of the OS the only extras required were;

  1. Tether my phone to the laptop to do an update so that the wireless driver installed.  The base image for the live image does not include the wireless driver, and there is no ethernet connector unless you by a USB adapter (which I have for the USB 3.1 port).
  2. The only other issue, even after the update was the screen brightness, which can be fixed through grub by adding the following;
    1. Edit the file /etc/default/grub
      GRUB_CMDLINE_LINUX="rhgb quiet acpi_backlight=vendor i915.preliminary_hw_support=1"
    2. Run the following command to update the boot loader;
      grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
After that the laptop is fully working, with all buttons.  The HDMI port though needs to be connected on boot to make it work with an external screen.  I also purchased HDMI to VGA connector since I have to connect to projectors that are so out of date, and also a mic/headphone combo cable.  The 3 cables (includes the ethernet/3 USB connector) were purchased from Amazon for less than £20.

Makes those of you with an Apple look like you are using something from the 1950s ;-)


Tuesday, August 11, 2015

Setting the keyboard and language RHEL and CentOS 7 command line

Setting the console keyboard RHEL and CentOS 7

When transferring VMs and systems built in one locale to another the keyboards become a problem.  In the earlier versions of Linux we simply changed a symlink or edited the /etc/sysconfig/keyboard file, but as of version 7 we now have to use localectl or edit the correct files to make the change permanent.

Having read various pages scattered across the Internet a lot of them talk about using the localectl command to change your keyboard and locale settings for a console installation.  However, the localectl command did not update the console keyboard and things such as the | key didn't work when changing from UK to US locale.  Even after a reboot the keyboard would still be wrong.

So how do you get the keyboard changed for the console in RHEL and CentOS 7 permanently?

Persistant Changing The Keyboard and Locale

Here we will assume that the system is currently set to en_GB.UTF-8 and uk keyboard.

1. Edit the /etc/locale.conf file and make sure the following is in it;
LANG="en_US.utf8"

2. Edit the /boot/grub2/grub.cfg file and change
vconsole.keymap=uk    change to    vconsole.keymap=us
Do that for all occurrences, a quick method is;
sed -i 's/vconsole.keymap=uk/vconsole.keymap=us/g' /boot/grub2/grub.cfg

3. Edit /etc/sysconfig/grub and change
vconsole.keymap=uk    change to     vconsole.keymap=us
Like step 2 you can use sed to perform the change as follows;
sed -i 's/vconsole.keymap=uk/vconsole.keymap=us/g' /etc/sysconfig/grub

By doing these steps and then rebooting your system your keyboard and locale would set itself to the new locals for the console.