Wednesday, April 7, 2021

The non conforming shell (zsh)

Today whilst working with some graduates who are all using Apple Macs I find out that the default zsh that is now used as the command line does not follow the common Shell standards!

 This particular nugget that was found will actually cause you a problem if you're a real system administrator who knows how to clear out log files without having to use rm or rebooting the systems or restarting the process.

The issue I refer to is the use of the redirection symbols > and >>.

Most of you are familiar with doing things such as;

echo "Hello" >somefile

ps -ef > allprocesses

But the real system administrators reading this also know that you should be able to do;

>/var/log/messages

Obviously as root.

This command should empty the log file without removing it, and freeing up disk space on that partition.

This is the conforming standard for the use of redirection in the majority of Unix and Linux (GNU) shells.


ZSH however does not do this any more!  So beware.

Instead ZSH when you do;

>somefile

will wait for your to type something in until you press ^D on an empty line.  This as all sysadmins know is;

cat >somefile

So in ZSH to perform the same action as real shells now need to;

>somefile

^D


So the question begs, who did this and why?


Yet another reason I tell people to buy a better regular laptop without an operating system and simply install a version of Linux that you like the look of and customise it to your preferred look and feel.


Mac OS != Unis

Mac OS == Broken Unix.

Wednesday, July 22, 2020

Apache Restricting Content From Download

The Scenario

After releasing our latest Youtube video https://youtu.be/QWjub-nKNL4 we had some extra content that we wanted to share, but not allow download, since it was mentioned in the video.

What to do, what to do.  Previously I tried looking for options on how to stop the right click download, but that wouldn't stop people using other ways to download the content, and owning the web server wanted a way that would restrict the straight forward methods of downloading the content and only allowing it to be streamed using the player on the web site.

A few Apache configuration lines later, and the use of JWPlayer I solved my problem.  The joys of having your own web server - which means you can't do this if you're on a hosted platform.

Most players are client based, so the requirement to not have direct access to the file is something I'm very familiar with, but didn't want to write my own player.  In the past I've streamed Word documents without the URL being available so that clients had to be logged in to access the document, and never knew the real URL, but this time I want to make sure the content is not downloadable directly.

The Research

First I needed to see if anyone had worked out how to prevent download of files in Apache (httpd), so a quick Google search for "apache htaccess deny download of specific files" came up with;
RewriteEngine On
RewriteCond %{HTTP_REFERER} !^http://(www\.)?yourwebsite\.com/ [NC]
RewriteCond %{REQUEST_URI} !hotlink\.(mp3|mp4|mov) [NC]
RewriteCond %{HTTP_COOKIE} !^.*wordpress_logged_in.*$ [NC]
RewriteRule .*\.(mp3|mp4|mov)$ http://yourwebsite.com/ [NC]
From this the lines I needed would be;
  • RewriteEngine On
  • RewriteCond %{HTTP_REFERER} !^http://......
  • RewriteRule .*\.(mp3|wav)$ ....
I would need to make some modifications to this, but the .htaccess file was the way to go, since other directories I still wanted people to download from.  So some extra help from https://300m.com/stupid-htaccess-tricks/ to understand what goes on in the square brackets.

Secondly, because I wanted to use the .htaccess file I would need to modify my Apache configuration to AllowOverride All for the root directory.

Last but not least a player that would allow the media to be played through the web browser - https://www.jwplayer.com/.

Configuring Apache

To ensure Apache allows you to use the .htaccess files you need to change the main configuration files.  If you intend to be able to do different things in different directories then you should set your DocumentRoot so that the Directory setting has AllowOverride  All instead of the usual AllowOverride None.

Excerpt from the httpd.conf file;

DocumentRoot "/usr/local/apache2/htdocs"
<Directory "/usr/local/apache2/htdocs">
    Options Indexes FollowSymLinks
    IndexOptions FancyIndexing
    DirectoryIndex index.html

    AllowOverride All

    Require all granted
</Directory>

You should also ensure that the following module is enabled in the httpd.conf file;
LoadModule rewrite_module modules/mod_rewrite.so
This will ensure that the directives we add to the .htaccess file will work. 

Denying Download

Now locate the directory in you web server where you want to restricted the download of the content.  In this directory we will add the .htaccess file to prevent our MP3 and WAV files being downloaded.

The content of the .htaccess file;

RewriteEngine on

RewriteCond %{HTTP_REFERER} !^http://(www\.)?tps\.local [NC]

RewriteRule .*\.(mp3|wav)$ - [NC,F,L] 


The Lines Explained

RewriteEngine on

This line ensures that Apache can perform rewrites to the HTTP headers and allow the other 2 actions to work correctly.

ReWriteCond

This line checks that the REFERER in the header is tps.local for this example, but should be your domain name.  This ensures that only requests from your own domain are allowed.  The (www\.)? state that the www part might not be supplied.
The [NC] at the end of the line makes the whole statement case insensitive = No Case.

RewriteRule

This line defines what extensions we will be denying the download of.  In this case .mp3 and .wav.  The regular expression uses the or | notation to allow us to supply a list of mp3|wav  which is the same as saying mp3 or wav.  The .*\. before this will be anyfile name before the final full stop.  The $ at the end specifies the the mp3 or wav are the last characters at the end of the data.
The - symbol is there to replace the mp3 or wav file names with just an invalid filename of -.

Again at the end we see NC for case insensitive, F for forbidden to drive the 403 and L being that this is the last rule, don't do anything more.

Testing Download

Firstly we can try a successful download by using wget or curl or our web browser.

We should ensure an html file in the directory, and that should be displayed;

curl http://www.tps.local/index.html
It Works!

Now the mp3
curl http://www.tps.local/my.mp3
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /media/meditation/body.mp3
on this server.</p>
</body></html>

Using JWPlayer

First you need to get yourself an account on JWPlayer.

Log in and from the dashboard menu on the right select Players;

By default you will have 2 example players.  Select one that suits your needs and make modifications to it;

Change items such as;

  • Size
  • Playback
  • etc


Save your player and make a note of Cloud-Hosted Player Library URL as you will need this in your JavaScript code.

The following HTML is an example of using the player with a single file, let's say this is called index.html to make it easier to have the player launch when people enter the restricted directory;

<html>

<body>

<div id="myElement"></div>

<script type="text/JavaScript" src="https://cdn.jwplayer.com/libraries/example.js"></script>

<script type="text/JavaScript">

    jwplayer("myElement").setup({ 

        "playlist": [{

                "file": "my.mp3"

        }]

    });

</script>

</body>

</html>

Change the src to the URL in your JWPlayer account.  The file will play using the player.

For further customisation of JWPlayer see;
 

 


Sunday, May 31, 2020

Terraform Runtime Data

Terraform and Runtime values


The other day I was asked if we could use operating system variables in Terraform like we can in Ansible?

An interesting question.  Obviously those who know Terraform will shout out and say, of course you can you just put TF_VAR_ in front of the variable in the OS and Terraform will find it.

However, that's not the same as being able to get hold of say, HOME or PWD, etc as can be done in Ansible using the lookup function;

vars:
  cwd: "{{ lookup('env', 'PWD' }}"

The above in Ansible obtains the value of the PWD operating system environment variable and stores it in an Ansible variable called cwd at runtime.

So, how on earth do I do such a thing in Terraform, rather than relying on people configuring variables up front, or using a Makefile or wrapper script?

The data sources

The data source allows you to obtain data from various sources, see https://www.terraform.io/docs/configuration/data-sources.html, of which the best data source for our requirement is the external data source.

Example


The downside is that your external data source needs to return JSON data, so you can't just run a command or echo a variable.  But you can create the relevant JSON data.  I wrote a simple use at https://bitbucket.org/stevshil/terraform/src/master/envvar/.

To explain that code directly here I took the following steps;

  1. Create a shell script that will return JSON data, this way you don't have to work out how to escape characters, etc.  In the example code we created a script called mypwd containing the following;

    #!/bin/bash
    cat <<_END_
    {
      "dir": "$PWD"
    }
    _END_

  2. Create the Terraform code (getvar.tf) to grab the printed output;

    data "external" "example" {
     program = ["bash","./mypwd"]
    }
    output "pwd" {
      value = data.external.example.result.dir
    }

You'll notice that using the external data source, and a given name (example in this instance) we call our shell script.  The bash is there just in case someone forgot to make the script executable.

The output in this case is just to show that the data was returned in an attribute called dir as you'll note in the shell script code where we output JSON data which has a key called dir that contains the value of the operating system variable called $PWD.  We could just as easily have written Python code, or have ran another shell command that outputs the data we desire.

To retrieve the value output from our script we use the normal Terraform object attribute reference, but because it is a data source we prepend the word data to the resource - data.external.example.result.dir, with the result element being part of the data source, but the dir is our JSON data key.  This reference can be used anywhere within your code when you need to use the runtime value.

Goodbye to the TF_VAR_ potential error prone values if you need to make use of attributes from the operating system.

Monday, November 25, 2019

SSH config

A simple user based configuration file with lots of possible combinations is the $HOME/.ssh/config file.
This file is located in the user home directory, if the user has created one. If not you can create your own and start to define the SSH keys required to log on to particular hosts, the user you use to log on and lots more.

Example of defining a key and user to a specific host;

Host jenkins.tps.co.uk
  User ec2-user
  IdentityFile ~/.ssh/steve-jenkins.pem
  StrictHostKeyChecking no

The above file would log you on as ec2-user using the steve-jenkins.pem key located in the users .ssh directory inside their home directory. It also ignores the fingerprint prompt through the StrictHostKeyChecking.


Example of using a bastion/jump host;

Host bastion.tps.co.uk
  User admin
  StrictHostKeyChecking no
  ControlPersist 5m
  IdentityFile ~/.ssh/bastion.pem
Host 172.31.10.20
  User admin
  StrictHostKeyChecking no
  ProxyJump bastion.tps.co.uk

This will set the ability to SSH to the 172.31.10.20 host in the cloud through the host called bastion.tps.co.uk, logging on as admin with the bastion.pem file in the users .ssh directory. The ControlPersist sets a time out of 5 minutes where you will be logged out if no activity occurs for 5 minutes.

Using SSH command line through bastion to another host;
ssh -i ${privatesshkeyfile} -A user@${bastionnameorip} ssh ${farsidehost}

Thursday, August 17, 2017

RHEL Alternatives

When changing environments for certain applications, Java being a key one which has many different files and variables to be configured you want to be able to gt the full environment changed in one go, rather than having to set different paths, etc.

The RHEL alternatives command does just that, allowing you to change your Java environment so that java, javac, javadoc all get set to the correct location without having to change your PATH and other environment variables.

Here's how you can set Java to the Oracle version after you have installed it on to your RHEL system.  You'll note that if you use the alternatives command you will get the JRE rather than the JDK, and if you're developing or require the JDK you will want to change the settings.

Before we can add a new option, we need to find out the next available number to use;

sudo alternatives --config java

There are 3 programs which provide 'java'.

  Selection    Command
-----------------------------------------------
   1           java-1.7.0-openjdk.x86_64 (/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.141-2.6.10.1.el7_3.x86_64/jre/bin/java)
 + 2           java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64/jre/bin/java)

   3           /usr/java/jdk1.8.0_131/jre/bin/java


Enter to keep the current selection[+], or type selection number: 

From the above output we can see that our next available slot is 4, so we need to add Oracle JDK to this location;

sudo alternatives --install /usr/bin/java java /usr/java/jdk1.8.0_131/bin/java 4

The above command will also identify the javac and javadoc commands too, so no other changes are required.

Now we can set Oracle Java as our default JDK;

sudo alternatives --config java

There are 4 programs which provide 'java'.

  Selection    Command
-----------------------------------------------
   1           java-1.7.0-openjdk.x86_64 (/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.141-2.6.10.1.el7_3.x86_64/jre/bin/java)
 + 2           java-1.8.0-openjdk.x86_64 (/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64/jre/bin/java)
   3           /usr/java/jdk1.8.0_131/jre/bin/java
   4           /usr/java/jdk1.8.0_131/bin/java

Enter to keep the current selection[+], or type selection number:

Select 4 and you will have Oracle JDK default system wide.

Thursday, May 4, 2017

Puppet Rake serverspec testing

Using Rake to ServerSpec test

If you don't like kitchen, or your team is using Rake you may choose to use the rake spec command for serverspec testing.  The directory layout is simpler than Kitchen, but requires more configuration as you need to create the following files;
  • Rakefile
  • spec_helper.rb
  • Vagrantfile
  • Provisioning script for the Vagrant VMs
These files need further configuration to make serverspec work using Rake.  The Rakefile and the spec_helper.rb files will live within the test directory, but spec_helper.rb will live in a spec subdirectory.
You'll notice the key difference compared to RSpec is the localhost directory. Each host will need it's spec test included. For example;
 Vagrantfile
 test
 ├── coretests
 │   ├── nginx_spec.rb
 │   └── dns_spec.rb
 ├── Rakefile
 └── spec
     └── web
     │   ├── nginx_spec.rb -> ../../coretests/nginx_spec.rb
     └── webserver1.al.local
     │   └── nginx_spec.rb -> ../../coretests/nginx_spec.rb 
     └── spec_helper.rb
      
Example diagramatic view of a project and where the directory structure fits;

Above we now have an SSH host for web and weberver1.al.local. The difference between the localhost and the SSH version is in the spec_helper.rb file, so be sure to check your configurations.
In the above we symlink each host with the required tests in the coretests folder.

Rakefile

Here is an example Rake file to perform build, config and run;
require 'rake'
require 'rspec/core/rake_task'
 
hosts = %w(
  web
)
 
task :spec => 'spec:all'
 
namespace :spec do
  task :all => hosts.map {|h| 'spec:' + h.split('.')[0] }
  hosts.each do |host|
    short_name = host.split('.')[0]
    role       = short_name.match(/.+/)[0]
 
    desc "Run serverspec to #{host}"
    RSpec::Core::RakeTask.new(short_name) do |t|
      ENV['TARGET_HOST'] = host
      t.pattern = "spec/{base,#{role}}/*_spec.rb"
      t.verbose = true
    end
  end
end
The important elements about the above file are;
  • hosts = %w (
    • Each line in this list will be a host in your Vagrantfile
    • The name specified here will be the same as the name of the define :web for example, which is the beginning of the definition of the VM
  • task :spec => 'spec:all'
    • This line defines what tasks to run within the file.  This file will run the all task.
  • task :all
    • This is the main task that tells rake to use the hosts variable to loop through
    • The hosts.each starts the real work
    • If you have more than one host you will need to add them to the hosts array
  • t.pattern
    • This line defines the name of the directory that will be used to target the tests to run for the current host in the loop

spec/spec_helper.rb

This file defines how the VM will be created and the connection to it, as well as other environmental information to be used within the VM.
Example;
require 'serverspec'
require 'net/ssh'
require 'tempfile'
 
set :backend, :ssh
 
if ENV['ASK_SUDO_PASSWORD']
  begin
    require 'highline/import'
  rescue LoadError
    fail "highline is not available. Try installing it."
  end
  set :sudo_password, ask("Enter sudo password: ") { |q| q.echo = false }
else
  set :sudo_password, ENV['SUDO_PASSWORD']
end
 
host = ENV['TARGET_HOST']
`vagrant up #{host}`
 
config = Tempfile.new('', Dir.tmpdir)
config.write(`vagrant ssh-config #{host}`)
config.close
 
options = Net::SSH::Config.for(host, [config.path])
 
options[:user] ||= Etc.getlogin
 
set :host,        options[:host_name] || host
set :ssh_options, options
 
# Disable sudo
# set :disable_sudo, true
 
# Set environment variables
# set :env, :LANG => 'C', :LC_MESSAGES => 'C'
 
# Set PATH
# set :path, '/sbin:/usr/local/sbin:$PATH'
#

Vagrantfile

This will live above the test directory.  For our example so far we would need a VM with the name of web;
Vagrant.configure("2"do |config|
  config.vm.define :web do | web |
    web.vm.provider "virtualbox" do | vb |
        vb.memory = 1024
        vb.cpus = 1
        vb.name = "WebNode"
    end
    web.vm.hostname = "web.al.local"
    web.vm.network "private_network", ip: "192.168.100.23"
    web.vm.provision "shell", path: "bin/web.sh"
    web.vm.box = "bento/centos-7.2"
  end
end
The define :web matches the web in the Rakefile.  The bin/web.sh will call the necessary commands to provision the VM, prior to serverspec running.

Running the test

To run a serverspec test using rake simply run the following;
rake spec
Unlike kitchen, rake does not show the build and configuration of the VM, so you will be waiting some time until it gets to the test phase, which means the output is cleaner, since it is only the tests.