Sep 212022
 

This post is a bit of an after the event one!

podman run -it ubi8/ubi

yum update

yum group install “Development Tools”

yum clean all

cat <<EOF > /etc/yum.repos.d/conda.repo 
[conda]
name=Conda 
baseurl=https://repo.anaconda.com/pkgs/misc/rpmrepo/conda
enabled=1 
gpgcheck=1 
gpgkey=https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc 
EOF 

yum update

yum install conda

yum clean all

source /opt/conda/etc/profile.d/conda.sh
conda -V 

# conda update -n base -c defaults conda <not working with rhel8.1 yet, suspect python/gcc issue

conda config --add channels defaults
conda config --add channels bioconda
conda config --add channels conda-forge
conda create -n <environment name>

My default image is called conda-4-7-10 and available at quay.io

Ok updated to 4-8-4 and we have rid of the dodgy asc gpg issue, image should be up later. Should be good to pull packages after this into the container

podman push – move the container image to quay.io

podman login quay.io

username

password

podman push quay.io/<user>/<repo>/<name>

 Posted by at 2:10 pm
Nov 302020
 

Install Fail2Ban and configure some default settings:

sudo yum update
sudo yum install epel-release
sudo yum install fail2ban
sudo nano /etc/fail2ban/jail.local

A typical template for jail.local looks like this:

[DEFAULT]
bantime = 43200
findtime = 600
maxretry = 3
banaction = iptables-multiport
backend = systemd

[sshd]
enabled = true

ignore = a list of I.P address that you do not wish to ban

bantime (seconds) – typical values are:
3600 (1 hour); 10800 (3 hours); 21600 (6 hours)
43200 (12 hours); 86400 (24 hours)

maxretry is the number of failures before a host is banned. Default value is 3.

findtime (seconds) and maxretry are related. Combined, they dictate the number of attempts (maxretry) within a given timeframe (findtime) which results in a ban. Default value of findtime is 600, which means that when fail2ban counts 3 failed attempts, within the last 10 minutes, it will ban the querying IP address.

Good values for findtime are:
300 or 600 (300 seconds = 5 minutes, 600 = 10 minutes)
3 for maxretry

Enable Fail2Ban at reboot, start the service, and check status:

systemctl start fail2ban
systemctl enable fail2ban
systemctl status fail2ban
fail2ban-client status
fail2ban-client status sshd

Nov 302020
 

Render spaces as dots – useful for sanity-checking files.
Go to settings; type “white space”
In the first option – “Render Whitespace” – change this from “selection” to “all”

The default colour scheme can make it difficult to see the dots. The colour can be changed. The following is from https://thenetworktransit.com/whitespace-in-vs-code/
who did all the heavy lifting here i.e. scouring the correct VS setting that is required.

Go to settings; type “Color Customization”
3rd option down – Workbench: Color Customization – click “edit in settings.jason”
Add the line:

"editorWhitespace.foreground": "ffa500"

E.g. it should look like this:

{
"editor.renderWhitespace": "all",
"workbench.colorCustomizations": {
"editorWhitespace.foreground": "ffa500"
}
}

Quit and relaunch Visual Studio

__________________________________

Aug 282019
 

This document is the beginning of a training document to describe the process from data curation from the novaseq, bcl2fastq conversion, indexing for metadata, archival to tape, validation, labeling, retrieval.

Data from a novaseq run is controlled via the novaseq machine – novaseqdata has a samba share which the machine can see from the front end program. This is selected and created at the start of a run giving the run it’s name and directory. Data written here is as sbsuser and that also translates into sbsuser on novaseqdata

After the run is complete the data is run through a run through a program to extract fastq data

bcl2fastq

ssh into novaseqdata as sbsuser

run the bcl2fastq command  – run from within the /mnt/novaseqdata/training/180525_A00471_0028_AHCKH2DMXX

Note: ignore \ in the following listings – type it all in on one line.

cd /mnt/novaseqdata/training/180525_A00471_0028_AHCKH2DMXX
nohup /usr/local/bin/bcl2fastq -R \
/mnt/novaseqdata/training/180525_A00471_0028_AHCKH2DMXX -o \
/mnt/novaseqdata/training/180525_A00471_0028_AHCKH2DMXX --sample-sheet \
/mnt/novaseqdata/training/180525_A00471_0028_AHCKH2DMXX/SampleSheet.csv

The process can take a couple of hours from start to finish depending on the run type – S2 or S4

CellRanger

Note: ignore \ in the following listings – type it all in on one line.

cd /mnt/novaseqdata/training/190823_A00471_0062_BHF7JJDRXX/
/opt/cellranger-3.0.2/cellranger mkfastq --run=/mnt/novaseqdata/training/ \
190823_A00471_0062_BHF7JJDRXX/ \
--samplesheet=/mnt/novaseqdata/training/ \
190823_A00471_0062_BHF7JJDRXX/sample_sheet_dan_williamson.csv

The process can take an hour or so to run – creates HF7JJDRXX as a subdirectory  which has the fastq files written within it

index the directory data with tree

From within the /mnt/novaseqdata/training/180525_A00471_0028_AHCKH2DMXX run directory issue this command

tree -a -T '180525_A00471_0028_AHCKH2DMXX' -H \
/mnt/novaseqdata/training/180525_A00471_0028_AHCKH2DMXX -o index.html \
/mnt/novaseqdata/training/180525_A00471_0028_AHCKH2DMXX

tape backup and restore * see later entry for tape manipulation in the library

From within the /mnt/novaseqdata/training directory issue this command

cd /mnt/novaseqdata/training/
tar -cMvf /dev/st0 180525_A00471_0028_AHCKH2DMXX/*

To restore the data from tape: –

cd /tmp
tar -xMvf /dev/st0 180525_A00471_0028_AHCKH2DMXX/*
or
tar -xMvf /dev/st0

rsync from novaseqdata to rocket

rsync -av \
/mnt/novaseqdata/training/180525_A00471_0028_AHCKH2DMXX 
nbh23@rocket.hpc.ncl.ac.uk:/nobackup/proj/scbsu/

Trailing slashes are important – if there is no upstream directory 180525_A00471_0028_AHCKHSDMXX rsync will create it and write data  into it accordingly. (NB can’t have more than one / separation upstream I discovered)

Once done it needs to be chmod’d recursively to 775 for other people from the group to access it: –

chmod -R 775 /nobackup/proj/scbsu/180525_A00471_0028_AHCKH2DMXX/
 Posted by at 8:37 am
Jun 222018
 

Anything that doesn’t warrant a War & Peace explanation goes in here…

The command and a brief what it does and why would you want to.

First up querying what a package brings to the party – excess baggage that may require further exploration…

repoquery
repoquery --requires --resolve

Good for rpmfusion or other repo’s that may add in things that change base packages. It does require that the repo is in the yum configuration for it to query. The yum settings can be switched on and off so that future yum updates wont check that particular repo and install things without your knowledge. This requires that yum-utils is installed.

yum-config-manager

The –enablerepo option overides the permanent option set in the .repo file for only the current command. –disablerepo does the opposite for enabled repos.

If you install (or have installed) the yum-utils package then you can use yum-config-manager to permanently enable/disable repos without editing the file:

yum-config-manager --enable remi

will permanently enable the remi repo.

yum repolist all

will list all the repo id which you use with the –enable or –disable option.

one-off install from disabled repositories

By default yum installs only from the enabled repositories. For some reason if you like to install a package from a disabled repositories, use –enablerepo option in the ‘yum install’ as shown below.

yum --enablerepo=fedora-source install vim-X11.x86_64
RedHat Centos 7 firewall quickies
firewall-cmd --get-active-zones - Shows which zones you have and which interfaces

firewall-cmd --zone=public --add-port=2049/tcp --permanent - Adds nfs4 port 2049 to the firewall permanently

firewall-cmd --reload - Reload the firewall for the changes to take effect
firewall-cmd --zone=public --list-all - shows all the current rules, interfaces, ports etc
firewall-cmd --permanent --zone=public --add-port=80/tcp - add a port permanently web and tcp in this instance
firewall-cmd --zone=public --remove-port=80/tcp - remove a port 'The firewall giveth and the firewall taketh away!'
firewall-cmd --panic-on - exactly as advertised, shuts up shop
firewall-cmd --query-panic - what state is panic in? on or off
firewall-cmd --panic-off - open for business again

 

RedHat Centos 7 services quickies
systemctl start name.service - Starts a service
systemctl stop name.service - Stops a service
systemctl restart name.service - Restarts a service
systemctl reload name.service - Reloads configuration
systemctl status name.service - Checks if a service is running
systemctl is-active name.service - Checks if a service is running
systemctl list-units --type service --all - Displays the status of all services

systemctl enable name.service - Enables a service.
systemctl disable name.service - Disables a service.
systemctl status name.service - Checks if a service is enabled.
systemctl is-enabled name.service - Checks if a service is enabled.
systemctl list-unit-files --type service - Lists all services and checks if they are enabled.
systemctl list-dependencies --after - Lists services that are ordered to start before the specified unit.
chkconfig --list systemctl list-dependencies --before - Lists services that are ordered to start after

RedHat Centos 7 xrdp
yum install epel-release
yum install xrdp
systemctl enable xrdp.service
systemctl start xrdp.service
firewall-cmd --zone=public --add-port=3389/tcp --permanent
firewall-cmd --reload
restorecon -v /usr/sbin/xrdp
restorecon -v /usr/sbin/xrdp-sesman
reboot

Screen command
Delete a rogue screen session
screen -ls
screen -S <pid> -p 0 -X quit
rpm install by date query

rpm -qa --queryformat '%{installtime} (%{installtime:date}) %{name}\n' | sort -n | tail -5

miss off the tail to get a complete list 🙂

samba firewall

firewall-cmd –permanent –add-port=137/tcp
firewall-cmd –permanent –add-port=138/tcp
firewall-cmd –permanent –add-port=139/tcp
firewall-cmd –permanent –add-port=445/tcp
firewall-cmd –permanent –add-port=901/tcp

firewall-cmd –reload

Man page to nicely formatted PDF

For when you absolutely have to have a nicely printed Man page to scribble all over at the machine in question..

man -t <man page you want in here> | ps2pdf - man-page-name.pdf

 

List disk usage/folder sizes in order
du -xh * | sort -rn | head -20
fsck check

The dreaded ‘non shall pass’ when a system hasn’t been shutdown correctly or a drive is failing.

Run fsck without -a or -P

Look at the file system it is complaining about – probably one that is in device-mapper and write down the path. Then: –

fsck /dev/mapper/device/in/the/warning/message

and follow the prompts as they come up on the screen to repair the affected system.

run process as other user to see ‘their’ issues

as root: –
su <uid>

ls and see different date format

ls -l –time-style=full-iso

Find your external I.P address

curl ifconfig.me

 Posted by at 10:16 am
Jun 072018
 

These instructions are for University provisioned Windows 10 laptops.

Please note that University Common Desktop PCs (both Windows 7 and 10) will automatically add the photocopiers as a printer called ‘IGM Copier’.

If you are using a private/personal laptop (Windows 7, Windows 10, Apple, or Linux), please come and see us in room B245, 2nd floor, BioMedicine West building, for further advice.

Ensure that you have an internet connection
Click on Windows logo
Click settings icon (cog)
Click Devices
Click Printers & Scanners
Click +Add a printer
Wait, while Windows lists every possible printer it can see. This is not helpful, in this case!
Scroll down the list and click ‘The printer that I want isn’t listed’
Click ‘Select a shared printer by name’
In the box underneath, carefully type \\medson\
When you press that final \ key, a drop-down menu will appear. Scroll down the list and choose ‘IGM Copier’
The address/name of printer should now read as \\medson\IGM Copier
Click Next
Wait, while Windows configures the printer
Click Next
Click Finish

Feb 212018
 

* Confirm NIC id/link speed info:

dmesg | grep -i duplex

______________________________________

* ** To add – how to configure network settings from command line/minimum install only * ***

______________________________________

* Use iPERF for ‘real world’ connection stats:

yum install iperf

Place one PC in server mode:

iperf -s

On client PC:

iperf -c <ip.address.of.server>

______________________________________

 

Dec 062017
 

*** Install the Windows Subsystem for Linux:

https://msdn.microsoft.com/en-us/commandline/wsl/install-win10
(Ubuntu, Ope*S*se, and SL*S are currently available. Fedora is due at some stage)

It is possible that the WLS should mount any pre-existing Windows drives automatically. This was not the case for me, so below are workarounds. It is possible that, as WLS matures, these steps are not required.

*** Add your *existing* Windows drives e.g. your University research folders

In this example, our Windows 10 PC automatically adds the Institute Z: and O: drives at login. We can use these pre-existing Windows drives and mount them.

IGM example – O: and Z: drives:

$ sudo mkdir /mnt/o
$ sudo mount -t drvfs O: /mnt/o

$ sudo mkdir /mnt/z
$ sudo mount -t drvfs Z: /mnt/z

OR directly mount University drives. In this example, we will mount the Flow Cytometry drives (amend your drive letters; these are what we use here at the IGM). S: drive represents the live/instrument data; R: drive represents archived FC data.

Example 1:

$ sudo mkdir /mnt/s
$ sudo mkdir /mnt/r

$ sudo mount -t drvfs '\\fsarchive.ncl.ac.uk\IHGScratch\FLOWCF' /mnt/s
$ sudo mount -t drvfs '\\fsresearch2-dbx.ncl.ac.uk\IGM_FLOWCF' /mnt/r

Example 2:

$ sudo mkdir /mnt/s
$ sudo mkdir /mnt/r

$ sudo mount -t drvfs '\\campus.ncl.ac.uk\scratch\IHGSCRATCH\FLOWCF' /mnt/s
$ sudo mount -t drvfs '\\campus.ncl.ac.uk\IGM\FLOWCF' /mnt/r