This page lists all the tools available, for working with strand updates, pre- and post-imputation checking, plus a few others

If you have any issues with these programs please contact me, Will Rayner
william dot rayner at helmholtz-munich dot de
and/or
will dot rayner at strand dot org dot uk

A simple to use cluster plotting program.
The ASHG 2013 poster describing the use of this program can be downloaded here:

Detailed instructions and usage examples can be found in the zip download.

Download (13Mb)

Previous Versions

ScatterShot.2013DEC09.zip (13Mb)
ScatterShot.2013NOV25.r1.zip (12.9Mb)
ScatterShot_2013OCT25_r1.zip (16.4Mb)
ScatterShot_2013OCT24_r2.zip (15.5Mb)
ScatterShot.zip (12.9Mb)

Program to derive Kcal values from questionnaire data
See the accompanying paper here for details

Download here: PhysicalActivityProgram.zip

Program to check a QC'd plink .bim file against the HRC, 1000G or CAAPA reference SNP list in advance of imputation

All versions below 4.3.0 are designed for use in an interactive session which does not work well on a high performance cluster. This version is designed to run in a non-interactive manner or use the more recent versions 4.3.0 and above (recommended).

Download here:HRC-1000G-check-bim-v4.2.11-NoReadKey.zip

V4.2.2

Added two new options for allele frequency thresholds (-t <difference>, -n)

-t 0.3 sets the allele difference threshold to 0.3, the default if not set is 0.2. Use this to change the allele frequency difference used to exclude SNPs in the final file, range 0 - 1, the larger the difference the fewer SNPs that will be excluded.

-n flag to specify that you do not wish to exclude any SNPs based on allele frequency difference, if -n is used -t has no effect.

V4.2.3

Fixed bug whereby SNPs that were incorrectly mapped in the bim file were updated only for position and not chromosome.

V4.2.4

Added support for X chromosome in HRC release v1.1

V4.2.5

Fixed bug with Chromosome X being left out of plink command file

V4.2.6

Added the ability to use gzipped reference panels

Implemented a check to ensure the same number of variants are present in the .bim and .frq files

V4.2.7

Minor update to the usage information display

V4.2.8

Added new flag -c to specify checking individual chromosome(s) rather than assuming genome wide

V4.2.9

Update to allow reading of bim files with allele codes 1,2,3,4

Fixed a minor bug that resulted in a warning of a null entry if there were no differences between the bim file and the reference

V4.2.10

Update to check path and name of bim and frequency file are correct and report meaningfully if not

V4.2.11

Changed the plink commands to update and retain the Ref/Alt alleles in the plink conversion commands

V4.2.12

Added -a flag to disable the automatic removal of palindromic SNPs with MAF > 0.4

V4.2.13

Added ability to read the bgzipped reference panels, as well as plain gzipped files

Added -l flag to set path to preferred plink executable

Added better support for the paths to the plink files in the shell script

Added -o flag to allow the final output path for all the plink and VCF files to be specified

V4.3.0

Added TOPMed and rebuilt code to use less memory

Removed the interactive terminal size check, this version will work both interactively and on a cluster

V4.3.1

Added flag to allow variants not found in the reference panel to be kept in the output, if used in conjunction with -n it will keep all the input variants

V4.3.2

Changed the help messages to be clearer and work on more systems

V4.3.3

Added new plink 2 run script with updated options, this is now created alongside the plink v1.9 run script, both work with the latest development versions of plink

tested with plink 1.9 (beta 7.11) and plink 2 (v2.0.0-a.7LM)

Use the -l command to specify the plink executable, with or without path, otherwise the assumption is the commands plink and plink2 are in the path and will be used by default.

If the plink and/or plink2 executable is in the same directory as the bim file it will be used in the output.

Checks:

Strand, alleles, position, Ref/Alt assignments and frequency differences. In addition to the reference file v4 and above require the plink .bim and (from the plink --freq command) .frq files.

Produces:

A set of plink, and from v4.3.3 plink2, commands to update or remove SNPs based on checks against the reference file specified as well as the possibility to create a file (FreqPlot) of cohort allele frequency vs reference panel allele frequency.

Updates:

Strand, position, ref/alt assignment

Removes:

A/T & G/C SNPs if MAF > 0.4, SNPs with differing alleles, SNPs with > 0.2 allele frequency difference, SNPs not in reference panel.

All of these removal steps can be adapted or turned off in the latest versions >v4.3.1

Requires the unzipped (or V4.2.13 onwards, gzipped) tab delimited HRC reference (currently v1.1 HRC.r1-1.GRCh37.wgs.mac5.sites.tab). This appears to no longer be available from the Haplotype Reference Consortium Website but is available from on this site, link here: HRC.r1-1.GRCh37.wgs.mac5.sites.tab.gz

Usage: perl HRC-1000G-check-bim.pl -b <bim file> -f <Frequency file> -r <Reference panel> -h

Requires the unzipped 1000G legend file (instructions to create this are below) or (recommended) download here (1.4G in size): 1000GP_Phase3_combined.legend.gz for autosomes only or here with X chromosome included 1000GP_Phase3_combined.legend.incX.gz

Usage: perl HRC-1000G-check-bim.pl -b <bim file> -f <Frequency file> -r <Reference panel> -g -p <population>
1000G population will default to ALL if not specified.

The CAAPA reference panel can be downloaded here (513MB): all.caapa.sorted.zip
Many thanks to Kathleen Barnes and Michelle Daya of CAAPA for sharing this and also to Margaret Parker and Michael Cho for the initial reformatting

Usage is the same as for the HRC panel, namely using the flags -h -r all.caapa.sorted.txt

Usage: perl HRC-1000G-check-bim.pl -b <bim file> -f <Frequency file> -r all.caapa.sorted.txt -h

This section is included for the sake of completeness but is no longer needed as the 1000G reference panel file, with and without X chromosome, can be downloaded on the links above or below.

The reference file for 1000G can be created from the legend files on the impute website or downloaded from the link below.
To create the file you will need to extract the legend files from the 1000GP_phase3.tgz file: https://mathgen.stats.ox.ac.uk/impute/1000GP_Phase3.html

Download the .tgz and extract all .legend files, place them in a directory together with the following script: concatenate-1000GP.zip and run the script (perl concatenate-1000GP.pl). This will create a file (1000GP_Phase3_combined.legend) suitable for use with the checking program.

Alternatively the file can be downloaded from this site here: 1000GP_Phase3_combined.legend with X chromosome 1000GP_Phase3_combined.legend.incX.gz

Perl script to extract a subset of data from the UKBioBank phenotype data download.

Requires the Stata .dct and R .tab files to run. Run without any arguments to get a description of usage.

Downloads:

Version 1.0 extract.zip

Version 1.2.2 extract-v1.2.2.zip

This program was developed to give an overview of phenotypes (columns) in a tab delimited text file, to do this it attempts to determine the type of each column (numeric, text, mixed text/numeric or coded variables) and summarise it appropriately.

v3.1

phenotype_qc.zip

v4.1

Updated to better accommodate the missing column names in the dct file.

phenotype_qc.v4.1.zip

Version 3.1

Usage: perl phenotype_qc.pl -f <phenotypeFile> [-m <missingValueIdentifier> -i <sampleIDcolumn> -c <column> -s <columnFile> -l <headerLookup> -h]

Version 4.1

Usage: perl phenotype_qc.pl -f <phenotypeFile> [-m <missingValueIdentifier> -i <sampleIDcolumn> -c <column> -s <columnFile> -l <headerLookup> -h -p -k <#columns per chunk>]

Version 4.1 introduces a new method for extracting the data from the phenotype file, this is necessary as for large data sets (>5GB) where v3.1 would not perform well or at all. This version works on data sets of any size however it will only work in a Unix/Linux environment (Cygwin possible but not tested) as it requires the UNIX cut command to extract the data. The option -k specifies how many columns at a time to extract using cut, the default is 25 which is a trade off between the number of calls to cut vs the amount of memory required. If you have problems with memory usage set this a smaller number. Higher numbers are possible but on test sets to date the time to complete a run increases, probably due to memory allocation.

The second new option (-p) is to plot all the numeric phenotypes, this allows a quick visual check of the distribution of values. Running this version, with or without plotting will still require GD::Graph to be installed, for instructions on how to do this see here: GD::Graph

Phenotype QC program command line options:

The categories reported by the program are:

Numeric: this category reports:

- Total number of non-missing variables.

- Total number of missing entries.

- Mean.

- Median.

- Standard deviation.

- Number of entries that are either greater or less than three standard deviations from the mean.

Text, Mixed, and Coded

These categories all report:

- Total number of non-missing variables.

- Total number of missing entries.

- Total number of unique entries, and if the number of unique entries is below 50 these are displayed with counts for each entry.

Coded:

Entries in this category can be text coded, numeric coded or mixed coded.

A column is described as containing a coded set of variables if the total number of unique values is less than 0.1% of the total number of entries, or 20, whichever is the greater.

The latest versions (v3.1 & v4.1) also produce a "fingerprint" summary of each row (assumed sample) and column (assumed phenotype) these are designed to verify whether a sample or phenotype has changed between data releases.
The fingerprint files (prefixed with FP, fingerprint phenotype, and FPS, fingerprint Sample) both contain the following columns:

A Perl wrapper using Spreadsheet::Read to convert various spreadsheet formats (xls, xlsx, csv, ods) to tab delimited text files.

Download

Usage:

The Chipendium tool to identify the array and strand orientation from the SNP list in a plink format bim file is currently unavailable but will be returning soon.