Nov 302020
 

Install Fail2Ban and configure some default settings:

sudo yum update
sudo yum install epel-release
sudo yum install fail2ban
sudo nano /etc/fail2ban/jail.local

A typical template for jail.local looks like this:

[DEFAULT]
bantime = 43200
findtime = 600
maxretry = 3
banaction = iptables-multiport
backend = systemd

[sshd]
enabled = true

ignore = a list of I.P address that you do not wish to ban

bantime (seconds) – typical values are:
3600 (1 hour); 10800 (3 hours); 21600 (6 hours)
43200 (12 hours); 86400 (24 hours)

maxretry is the number of failures before a host is banned. Default value is 3.

findtime (seconds) and maxretry are related. Combined, they dictate the number of attempts (maxretry) within a given timeframe (findtime) which results in a ban. Default value of findtime is 600, which means that when fail2ban counts 3 failed attempts, within the last 10 minutes, it will ban the querying IP address.

Good values for findtime are:
300 or 600 (300 seconds = 5 minutes, 600 = 10 minutes)
3 for maxretry

Enable Fail2Ban at reboot, start the service, and check status:

systemctl start fail2ban
systemctl enable fail2ban
systemctl status fail2ban
fail2ban-client status
fail2ban-client status sshd

Feb 212018
 

* Confirm NIC id/link speed info:

dmesg | grep -i duplex

______________________________________

* ** To add – how to configure network settings from command line/minimum install only * ***

______________________________________

* Use iPERF for ‘real world’ connection stats:

yum install iperf

Place one PC in server mode:

iperf -s

On client PC:

iperf -c <ip.address.of.server>

______________________________________

 

Dec 062017
 

*** Install the Windows Subsystem for Linux:

https://msdn.microsoft.com/en-us/commandline/wsl/install-win10
(Ubuntu, Ope*S*se, and SL*S are currently available. Fedora is due at some stage)

It is possible that the WLS should mount any pre-existing Windows drives automatically. This was not the case for me, so below are workarounds. It is possible that, as WLS matures, these steps are not required.

*** Add your *existing* Windows drives e.g. your University research folders

In this example, our Windows 10 PC automatically adds the Institute Z: and O: drives at login. We can use these pre-existing Windows drives and mount them.

IGM example – O: and Z: drives:

$ sudo mkdir /mnt/o
$ sudo mount -t drvfs O: /mnt/o

$ sudo mkdir /mnt/z
$ sudo mount -t drvfs Z: /mnt/z

OR directly mount University drives. In this example, we will mount the Flow Cytometry drives (amend your drive letters; these are what we use here at the IGM). S: drive represents the live/instrument data; R: drive represents archived FC data.

Example 1:

$ sudo mkdir /mnt/s
$ sudo mkdir /mnt/r

$ sudo mount -t drvfs '\\fsarchive.ncl.ac.uk\IHGScratch\FLOWCF' /mnt/s
$ sudo mount -t drvfs '\\fsresearch2-dbx.ncl.ac.uk\IGM_FLOWCF' /mnt/r

Example 2:

$ sudo mkdir /mnt/s
$ sudo mkdir /mnt/r

$ sudo mount -t drvfs '\\campus.ncl.ac.uk\scratch\IHGSCRATCH\FLOWCF' /mnt/s
$ sudo mount -t drvfs '\\campus.ncl.ac.uk\IGM\FLOWCF' /mnt/r

 

Nov 062017
 

Etcher website  < https://etcher.io/ > allows you to quickly and easily create bootable USB flash drives, based on ISOs. There are installers for Mac OS X (native app and homebrew); Windows; Linux (AppImage, deb, and rpm). There is also a command line version available for Mac/Linux and Windows.

We have used Etcher for creating CentOS installers and have found it to be very reliable. A superb, no fuss, cross-platform solution.

Etcher Mac OS X version

May 262017
 
CentOS 7:

https://bashtheshell.github.io/guide/ssd-trim-on-centos-7/

 

Windows 7:

Enabled by default but can be confirmed via elevated command prompt and:

fsutil behavior query disabledeletenotify

DisableDeleteNotify = 1 = TRIM support disabled
DisableDeleteNotify = 0 = TRIM support enabled

To enable trim, issue:

fsutil behavior set disabledeletenotify 0

 

Mac OS X:

Enabled by default on Apple provided SSDs.
Requires enabling on third-party provided SSDs.

To confirm via the terminal, issue:

system_profiler SPSerialATADataType | grep 'TRIM'

Enable TRIM via terminal, with this command:

sudo trimforce enable

Follow the on-screen advice.

For reference, Third-party solutions include Cindori TrimEnabler, which were
a workaround when Apple temporarily disabled default trim support
for non-Apple SSDs.
Oct 312013
 

MuTect 1.1.4 has been added to the cluster (31/10/2013)

MuTect is a method developed at the Broad Institute for the reliable and accurate identification of somatic point mutations in next generation sequencing data of cancer genomes.

——————————————————————-
$ module load apps/mutect
apps/mutect/1.1.4/bin
|
OK

$ ls $MUTECTDIR
bin doc java

$ ls $MUTECTDIR/*
/opt/gridware/pkg/apps/mutect/1.1.4/bin/bin:
mutect muTect

/opt/gridware/pkg/apps/mutect/1.1.4/bin/doc:
LICENSE.TXT version.txt

/opt/gridware/pkg/apps/mutect/1.1.4/bin/java:
muTect.jar

$ mutect
---------------------------------------------------------------------------------
The Genome Analysis Toolkit (GATK) v2.2-25-g2a68eab, Compiled 2012/11/08 10:30:02
Copyright (c) 2010 The Broad Institute
For support and documentation go to http://www.broadinstitute.org/gatk
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------
usage: java -jar muTect.jar -T [-args ] [-I ] [-rbs ] [-et
] [-K ] [-tag ] [-rf ] [-L ] [-XL ] [-isr
] [-im ] [-ip ] [-R ] [-ndrs]
[--disableRandomization] [-maxRuntime ] [-maxRuntimeUnits ] [-dt ]
[-dfrac ] [-dcov ] [-baq ] [-baqGOP ] [-PF
] [-OQ] [-BQSR ] [-DIQ] [-EOQ] [-preserveQ ] [-DBQ
] [-S ] [-rpr] [-kpr] [-U ] [-nt ] [-nct
] [-mte] [-bfh ] [-rgbl ] [-ped
] [-pedString ] [-pedValidationType ] [-l ] [-log
] [-h]

-T,--analysis_type Type of analysis to run

[snip]

##### ERROR ------------------------------------------------------------------------------------------
##### ERROR A USER ERROR has occurred (version 2.2-25-g2a68eab):
##### ERROR The invalid arguments or inputs must be corrected before the GATK can proceed
##### ERROR Please do not post this error to the GATK forum
##### ERROR
##### ERROR See the documentation (rerun with -h) for this tool to view allowable command-line arguments.
##### ERROR Visit our website and forum for extensive documentation and answers to
##### ERROR commonly asked questions http://www.broadinstitute.org/gatk
##### ERROR
##### ERROR MESSAGE: Argument with name '--analysis_type' (-T) is missing.
##### ERROR ------------------------------------------------------------------------------------------

——————————————————————-

$ module whatis apps/mutect
apps/mutect :

Title: MuTect
Summary: Reliable and accurate identification of somatic point mutations
License: NONFREE - Free for academic non-commercial research purposes only; see http://j.mp/10q7Mmv
Group: Bioinformatics
URL: http://www.broadinstitute.org/cancer/cga/mutect

Name: mutect
Version: 1.1.4
Module: apps/mutect/1.1.4/bin
Module path: /opt/gridware/etc/modules/apps/mutect/1.1.4/bin
Package path: /opt/gridware/pkg/apps/mutect/1.1.4/bin

Repository: git+http://download.alces-software.com/git/packager-base@8cca844
Package: ext/mutect/@3956ddde
Last update: 2013-04-03

Builder: root@headnode1.lampredi.cluster.local
Build date: 2013-10-31T12:42:07
Build parameters: zipfile=/opt/gridware/ext/muTect-1.1.4-bin.zip
Compiler: N/A

For further information, execute:

module help apps/mutect/1.1.4/bin

——————————————————————-

$ module help apps/mutect/1.1.4/bin

----------- Module Specific Help for 'apps/mutect/1.1.4/bin' ---------------------------

======== MuTect ========
Reliable and accurate identification of somatic point
mutations
=======================

This module sets up your environment for the use of the 'mutect'
application. This module sets up version '1.1.4' of the
application.

>> SYNOPSIS <> LICENSING <> FURTHER INFORMATION <> GET STARTED <<

Please refer to the website for further details on usage of this
package.

——————————————————————-

$ module show apps/mutect/1.1.4/bin
-------------------------------------------------------------------
/opt/gridware/etc/modules/apps/mutect/1.1.4/bin

module-whatis

Title: MuTect
Summary: Reliable and accurate identification of somatic point mutations
License: NONFREE - Free for academic non-commercial research purposes only; see http://j.mp/10q7Mmv
Group: Bioinformatics
URL: http://www.broadinstitute.org/cancer/cga/mutect

Name: mutect
Version: 1.1.4
Module: apps/mutect/1.1.4/bin
Module path: /opt/gridware/etc/modules/apps/mutect/1.1.4/bin
Package path: /opt/gridware/pkg/apps/mutect/1.1.4/bin

Repository: git+http://download.alces-software.com/git/packager-base@8cca844
Package: ext/mutect/@3956ddde
Last update: 2013-04-03

Builder: root@headnode1.lampredi.cluster.local
Build date: 2013-10-31T12:42:07
Build parameters: zipfile=/opt/gridware/ext/muTect-1.1.4-bin.zip
Compiler: N/A

For further information, execute:

module help apps/mutect/1.1.4/bin

module-conflict apps/mutect
setenv MUTECTDIR /opt/gridware/pkg/apps/mutect/1.1.4/bin
setenv MUTECTBIN /opt/gridware/pkg/apps/mutect/1.1.4/bin/bin
setenv MUTECTDOC /opt/gridware/pkg/apps/mutect/1.1.4/bin/doc
setenv MUTECTJAVA /opt/gridware/pkg/apps/mutect/1.1.4/bin/java
setenv MUTECT_JAVA_MEM_MX 2g
setenv MUTECT_JAVA_TMPDIR /tmp
setenv MUTECT_JAVA_OPTS -Xmx$MUTECT_JAVA_MEM_MX -Djava.io.tmpdir=$MUTECT_JAVA_TMPDIR
prepend-path PATH /opt/gridware/pkg/apps/mutect/1.1.4/bin/bin

——————————————————————-

Lampredi HPC
igm-lampredi-users@ncl.ac.uk

IGM IT
igmithelp@ncl.ac.uk
x8625

Oct 312013
 

Strelka 1.0.12 has been added to the cluster (31/10/2013)

This is an analysis package designed to detect somatic SNVs and small indels from the aligned sequencing reads of matched tumor-normal samples.

——————————————————————-

$ module load apps/strelka
apps/strelka/1.0.12/gcc-4.4.6
| -- libs/gcc/system
| * --> OK
|
OK

$ ls $STRELKADIR/
bin etc lib libexec opt

$ ls $STRELKADIR/*
/opt/gridware/pkg/apps/strelka/1.0.12/gcc-4.4.6/bin:
configureStrelkaWorkflow.pl demo

/opt/gridware/pkg/apps/strelka/1.0.12/gcc-4.4.6/etc:
strelka_config_bwa_default.ini strelka_config_eland_default.ini strelka_config_isaac_default.ini

/opt/gridware/pkg/apps/strelka/1.0.12/gcc-4.4.6/lib:
Utils.pm

/opt/gridware/pkg/apps/strelka/1.0.12/gcc-4.4.6/libexec:
callSomaticVariants.pl consolidateResults.pl countFastaBases filterSomaticVariants.pl strelka2

/opt/gridware/pkg/apps/strelka/1.0.12/gcc-4.4.6/opt:
Makefile samtools-0.1.18_no_tview vcftools vcftools-r837.tar.gz
samtools samtools-0.1.18_no_tview.tar.bz2 vcftools-r837

$ $STRELKABIN/demo/run_demo.bash

**** Starting demo configuration.
**** Configuration cmd: '/opt/gridware/pkg/apps/strelka/1.0.12/gcc-4.4.6/bin/demo/../configureStrelkaWorkflow.pl --tumor=/opt/gridware/pkg/apps/strelka/1.0.12/gcc-4.4.6/bin/demo/data/NA12891_dupmark_chr20_region.bam --normal=/opt/gridware/pkg/apps/strelka/1.0.12/gcc-4.4.6/bin/demo/data/NA12892_dupmark_chr20_region.bam --ref=/opt/gridware/pkg/apps/strelka/1.0.12/gcc-4.4.6/bin/demo/data/chr20_860k_only.fa --config=/opt/gridware/pkg/apps/strelka/1.0.12/gcc-4.4.6/bin/demo/strelka_demo_config.ini --output-dir=./strelkaDemoAnalysis'

INFO: Scanning reference genome
INFO: Scanning reference genome complete

Successfully configured analysis and created makefile '/users/alces-cluster/strelkaDemoAnalysis/Makefile'.

To run the analysis locally using make, run:

make -C /users/alces-cluster/strelkaDemoAnalysis

...or:

cd /users/alces-cluster/strelkaDemoAnalysis
make

**** Completed demo configuration.

**** Starting demo workflow execution.
**** Workflow cmd: 'make -C ./strelkaDemoAnalysis'

make: Entering directory `/users/alces-cluster/strelkaDemoAnalysis'
/opt/gridware/pkg/apps/strelka/1.0.12/gcc-4.4.6/libexec/callSomaticVariants.pl --config=/users/alces-cluster/strelkaDemoAnalysis/config/run.config.ini --chrom=chr20 --bin=0001 && touch /users/alces-cluster/strelkaDemoAnalysis/chromosomes/chr20/bins/0001/task.complete
/opt/gridware/pkg/apps/strelka/1.0.12/gcc-4.4.6/libexec/callSomaticVariants.pl --config=/users/alces-cluster/strelkaDemoAnalysis/config/run.config.ini --chrom=chr20 --bin=0002 && touch /users/alces-cluster/strelkaDemoAnalysis/chromosomes/chr20/bins/0002/task.complete
/opt/gridware/pkg/apps/strelka/1.0.12/gcc-4.4.6/libexec/callSomaticVariants.pl --config=/users/alces-cluster/strelkaDemoAnalysis/config/run.config.ini --chrom=chr20 --bin=0000 && touch /users/alces-cluster/strelkaDemoAnalysis/chromosomes/chr20/bins/0000/task.complete
/opt/gridware/pkg/apps/strelka/1.0.12/gcc-4.4.6/libexec/filterSomaticVariants.pl --config=/users/alces-cluster/strelkaDemoAnalysis/config/run.config.ini --chrom=chr20 && touch /users/alces-cluster/strelkaDemoAnalysis/chromosomes/chr20/task.complete
/opt/gridware/pkg/apps/strelka/1.0.12/gcc-4.4.6/libexec/consolidateResults.pl --config=/users/alces-cluster/strelkaDemoAnalysis/config/run.config.ini && touch /users/alces-cluster/strelkaDemoAnalysis/task.complete

Analysis complete. Final somatic calls can be found in /users/alces-cluster/strelkaDemoAnalysis/results

make: Leaving directory `/users/alces-cluster/strelkaDemoAnalysis'

**** Completed demo workflow execution.

**** Starting comparison to expected results.
**** Expected results dir: /opt/gridware/pkg/apps/strelka/1.0.12/gcc-4.4.6/bin/demo/expected_results
**** Demo results dir: ./strelkaDemoAnalysis/results

**** No differences between expected and computed results.

**** Demo/verification successfully completed

——————————————————————-

$ module whatis apps/strelka
apps/strelka :

Title: Strelka
Summary: Somatic variant calling workflow for matched tumor-normal samples
License: NONFREE - Illumina Open Source Source License
Group: Bioinformatics
URL: https://sites.google.com/site/strelkasomaticvariantcaller/

Name: strelka
Version: 1.0.12
Module: apps/strelka/1.0.12/gcc-4.4.6
Module path: /opt/gridware/etc/modules/apps/strelka/1.0.12/gcc-4.4.6
Package path: /opt/gridware/pkg/apps/strelka/1.0.12/gcc-4.4.6

Repository: git+http://download.alces-software.com/git/packager-base@8cca844
Package: ext/strelka/@f8d32f14
Last update: 2013-10-31

Builder: root@headnode1.lampredi.cluster.local
Build date: 2013-10-31T12:46:53
Build parameters: tarball=/opt/gridware/ext/strelka_workflow-1.0.12.tar.gz
Compiler: compilers/gcc/system
System: Linux 2.6.32-279.2.1.el6_lustre.gc46c389.x86_64 x86_64
Arch: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 2x4 (b6b89932)
Dependencies: libs/gcc/system (using: libs/gcc/system)

For further information, execute:

module help apps/strelka/1.0.12/gcc-4.4.6

——————————————————————-

$ module help apps/strelka/1.0.12/gcc-4.4.6

----------- Module Specific Help for 'apps/strelka/1.0.12/gcc-4.4.6' ---------------------------

======== Strelka ========
Somatic variant calling workflow for matched tumor-normal
samples
========================

This module sets up your environment for the use of the 'strelka'
application. This module sets up version '1.0.12' of the
application.

>> SYNOPSIS <> LICENSING <> FURTHER INFORMATION <> GET STARTED <<

Please refer to the website for further details on usage of this
package.

——————————————————————-

$ module show apps/strelka/1.0.12/gcc-4.4.6
-------------------------------------------------------------------
/opt/gridware/etc/modules/apps/strelka/1.0.12/gcc-4.4.6:

module-whatis

Title: Strelka
Summary: Somatic variant calling workflow for matched tumor-normal samples
License: NONFREE - Illumina Open Source Source License
Group: Bioinformatics
URL: https://sites.google.com/site/strelkasomaticvariantcaller/

Name: strelka
Version: 1.0.12
Module: apps/strelka/1.0.12/gcc-4.4.6
Module path: /opt/gridware/etc/modules/apps/strelka/1.0.12/gcc-4.4.6
Package path: /opt/gridware/pkg/apps/strelka/1.0.12/gcc-4.4.6

Repository: git+http://download.alces-software.com/git/packager-base@8cca844
Package: ext/strelka/@f8d32f14
Last update: 2013-10-31

Builder: root@headnode1.lampredi.cluster.local
Build date: 2013-10-31T12:46:53
Build parameters: tarball=/opt/gridware/ext/strelka_workflow-1.0.12.tar.gz
Compiler: compilers/gcc/system
System: Linux 2.6.32-279.2.1.el6_lustre.gc46c389.x86_64 x86_64
Arch: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 2x4 (b6b89932)
Dependencies: libs/gcc/system (using: libs/gcc/system)

For further information, execute:

module help apps/strelka/1.0.12/gcc-4.4.6

module-prereq libs/gcc/system
module-conflict apps/strelka
setenv STRELKADIR /opt/gridware/pkg/apps/strelka/1.0.12/gcc-4.4.6
setenv STRELKABIN /opt/gridware/pkg/apps/strelka/1.0.12/gcc-4.4.6/bin
prepend-path PATH /opt/gridware/pkg/apps/strelka/1.0.12/gcc-4.4.6/bin


——————————————————————-

Lampredi HPC
igm-lampredi-users@ncl.ac.uk

IGM IT
igmithelp@ncl.ac.uk
x8625

Oct 282013
 

FamSeq 1.0.0-2 has been added to lampredi2. Details are:

$ module load apps/famseq
apps/famseq/1.0.0-2/gcc-4.4.6
| -- libs/gcc/system
| * --> OK
|
OK

[...@node05(lampredi) ~]$ ls $FAMSEQDIR
bin doc sample

[...@node05(lampredi) ~]$ ls $FAMSEQDIR/doc
manual.pdf

[...@node05(lampredi) ~]$ FamSeq

Program: FamSeq (Sequence calling using pedigree information)
Version: 0.0.1

Usage: FamSeq [options]

Input type: vcf input vcf file
LK input likelihood file

——————————————————————-

$ module whatis apps/famseq
apps/famseq :

Title: FamSeq
Summary: A computational tool for calculating probability of variants in family-based sequencing data
License: Public Domain
Group: Bioinformatics
URL: http://odin.mdacc.tmc.edu/~wwang7/FamSeqIndex.html

Name: famseq
Version: 1.0.0-2
Module: apps/famseq/1.0.0-2/gcc-4.4.6
Module path: /opt/gridware/etc/modules/apps/famseq/1.0.0-2/gcc-4.4.6
Package path: /opt/gridware/pkg/apps/famseq/1.0.0-2/gcc-4.4.6

Repository: git+http://download.alces-software.com/git/packager-base@ea64008
Package: apps/famseq/1.0.0-2@61f6fd9a
Last update: 2013-10-28

Builder: root@headnode1.lampredi.cluster.local
Build date: 2013-10-28T14:52:15
Compiler: compilers/gcc/system
System: Linux 2.6.32-279.2.1.el6_lustre.gc46c389.x86_64 x86_64
Arch: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 2x4 (b6b89932)
Dependencies: libs/gcc/system (using: libs/gcc/system)

For further information, execute:

module help apps/famseq/1.0.0-2/gcc-4.4.6

——————————————————————-

$ module help apps/famseq/1.0.0-2/gcc-4.4.6

----------- Module Specific Help for 'apps/famseq/1.0.0-2/gcc-4.4.6' ---------------------------

======== FamSeq ========
A computational tool for calculating probability of
variants in family-based sequencing data
=======================

This module sets up your environment for the use of the 'famseq'
application. This module sets up version '1.0.0-2' of the
application.

>> SYNOPSIS <> LICENSING <> FURTHER INFORMATION <> GET STARTED <<

Please refer to the website for further details on usage of this
package.

——————————————————————-

$ module show apps/famseq/1.0.0-2/gcc-4.4.6
-------------------------------------------------------------------
/opt/gridware/etc/modules/apps/famseq/1.0.0-2/gcc-4.4.6:

module-whatis

Title: FamSeq
Summary: A computational tool for calculating probability of variants in family-based sequencing data
License: Public Domain
Group: Bioinformatics
URL: http://odin.mdacc.tmc.edu/~wwang7/FamSeqIndex.html

Name: famseq
Version: 1.0.0-2
Module: apps/famseq/1.0.0-2/gcc-4.4.6
Module path: /opt/gridware/etc/modules/apps/famseq/1.0.0-2/gcc-4.4.6
Package path: /opt/gridware/pkg/apps/famseq/1.0.0-2/gcc-4.4.6

Repository: git+http://download.alces-software.com/git/packager-base@ea64008
Package: apps/famseq/1.0.0-2@61f6fd9a
Last update: 2013-10-28

Builder: root@headnode1.lampredi.cluster.local
Build date: 2013-10-28T14:52:15
Compiler: compilers/gcc/system
System: Linux 2.6.32-279.2.1.el6_lustre.gc46c389.x86_64 x86_64
Arch: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 2x4 (b6b89932)
Dependencies: libs/gcc/system (using: libs/gcc/system)

For further information, execute:

module help apps/famseq/1.0.0-2/gcc-4.4.6

module-prereq libs/gcc/system
module-conflict apps/famseq
setenv FAMSEQDIR /opt/gridware/pkg/apps/famseq/1.0.0-2/gcc-4.4.6
setenv FAMSEQBIN /opt/gridware/pkg/apps/famseq/1.0.0-2/gcc-4.4.6/bin
setenv FAMSEQDOC /opt/gridware/pkg/apps/famseq/1.0.0-2/gcc-4.4.6/doc
setenv FAMSEQSAMPLE /opt/gridware/pkg/apps/famseq/1.0.0-2/gcc-4.4.6/sample
prepend-path PATH /opt/gridware/pkg/apps/famseq/1.0.0-2/gcc-4.4.6/bin

——————————————————————-

Lampredi HPC
igm-lampredi-users@ncl.ac.uk

IGM IT
igmithelp@ncl.ac.uk
x8625