View source for SFS Estimation

This method will estimate the site frequency spectrum, the method is described in [[Nielsen2012]]. The theory behind the model is briefly described [[realSFSmethod|here]]

This is a 2 step procedure first generate a ".saf" file (site allele frequency likelihood), followed by an optimization of the .saf file which will estimate the Site frequency spectrum (SFS).

For the optimization we have implemented 2 different approaches both found in the misc folder. The diagram below shows the how the method goes from raw bam files to the SFS. 

You can also estimate a [[2d SFS Estimation| 2dsfs]].
<pre>
* NB the ancestral state needs to be supplied for the full SFS, but you can use the -fold 1 to estimate the folded SFS and then use the reference as ancestral.
</pre>
<classdiagram type="dir:LR">
 [sequence data{bg:orange}]->GL[genotype likelihoods|SAMtools;GATK;SOAPsnp;Kim et.al]
[genotype likelihoods|SAMtools;GATK;SOAPsnp;Kim et.al]->doSaf[.saf file{bg:blue}]
[.saf file{bg:blue}]->optimize('emOptim2')[.saf.ml file{bg:red}]
 </classdiagram>

=Brief Overview=
<pre>
./angsd -doSaf 
	-> angsd version: 0.569	 build(Dec 17 2013 12:40:00)
	-> Analysis helpbox/synopsis information:
--------------
angsd_realSFS.cpp:
	-doSaf		0
	1: perform multisample GL estimation
	2: use an inbreeding version
	3: calculate genotype probabilities (beta)
	4: Assume genotype posteriors as input (still beta) 
	-doThetas		0 (calculate thetas)
	-underFlowProtect	0
	-fold			0 (deprecated)
	-anc			(null) (ancestral fasta)
	-noTrans		0 (remove transitions)
	-pest			(null) (prior SFS)

</pre>

<pre>
misc/emOptim2
./emOptim2 afile.sfs nChr [-start FNAME -P nThreads -tole tole -maxIter  -nSites ]
nChr is the number of chromosomes. (twice the number of diploid invididuals)
</pre>
See more here: [[EmOptim2]]

=Options=
;-doSaf 1: Calculate the Site allele frequency likelihood based on individual genotype likelihoods assuming HWE

;-doSaf 2:(version above 0.503) Calculate per site posterior probabilities of the  site allele frequencies based on individual genotype likelihoods while taking into account individual inbreeding coefficients. This is implemented by Filipe G. Vieira.  You need to supply a file containing all the inbreeding coefficients. -indF

;-doSaf 3: Calculate the posterior probabilities of the sample allele frequency distribution for each site. This needs a prior distribution of the SFS (which can be obtained from -doSaf 1/emOptim2)

;-doSaf 4: Calculate the posterior probabilities of the sample allele frequency distribution for each site based on genotype probabilities. The genotype probabilities should be provided by the using using the -beagle options. Often the genotype probabilities will be obtained by haplotype imputation. 

;-underFlowProtect [INT] 
0: (default) no underflow protection. 1: use underflow protection. For large data sets (large number of individuals) underflow projection is needed. 


==Example==
A full example is shown below where we use the test data that can be found on the [[quick start]] page. In this example we use GATK genotype likelihoods.

first generate .saf file  with 4 threads
<pre>
./angsd -bam bam.filelist -doSaf 1 -out small -anc  chimpHg19.fa -GL 2 -P 4
</pre>
We always recommend that you filter out the bad qscore bases and meaningless mapQ reads. eg '''-minMapQ 1 -minQ 20'''. So the above analysis with these filters can be written as:
<pre>
./angsd -bam bam.filelist -doSaf 1 -out small -anc chimpHg19.fa -GL 2 -P 4 -minMapQ 1 -minQ 20
</pre>
Obtain a maximum likelihood estimate of the SFS using EM algorithm 
<pre>
misc/emOptim2 small.saf 20 -maxIter 100 -P 4 >small.sfs
</pre>

What does the 20 mean in the emOptim2 command?
This is the number of chromosomes
* If you have say 10 diploid samples, then you should choose 20
* if you have say 12 diploid samples, then you should choose 24.
* If you have say 14 diploid samples, and you fold the spectrum then you should choose 28

=Interpretation of the emOptim2 output file=

The output from emOptim2  (small.sfs) is the SFS in logscale.
This is to be interpreted as:

column1 := probabilty of sampling zero derived alleles (on logscale)

column2 := probabilty of sampling one derived allele  (on logscale)

column3 := probabilty of sampling two derived allele  (on logscale)

column4 := probabilty of sampling three derived allele  (on logscale)

etc


each row is a region of the genome (see below).
==NB==
The generation of the .saf file contains a saf for each site, whereas the optimization requires information for a region of the genome. The optimization will therefore use large amounts of memory.
The program defaults to 50megabase regions, and will loop over the genome using 50 megebase chunks. You can increase this adding -nSites 500000000. Which will then use 500megabase regions.

=Folded spectra=
If you don't have the ancestral state, you can instead estimate the folded SFS. This is done by supplying the -anc with the reference genome and also supply -fold 1.
Then you should remember to change the second parameter to the emOptim2 function as the number of individuals instead of the number of chromosomes.

The above example would then be

<pre>
#first generate .saf file
./angsd -bam bam.filelist -doSaf 1 -out small -anc  hg19.fa -GL 2 -fold 1
#now try the EM optimization with 4 threads
misc/emOptim2 small.saf 10 -maxIter 100 -P 4 >small.sfs
</pre>

=Posterior of the per-site distributions of the sample allele frequency=
If you supply a prior for the SFS (which can be obtained from the -doSaf/emOptim2 analysis), the output of the .saf file will no longer be  site allele frequency likelihoods but instead will be the log posterior probability of the sample allele frequency for each site.

=Format specification of binary .saf file=
The information in this section is only usefull for people who wants to work with the "multisample GL"/"sample allele frequencies" for the individual sites.

Assuming 'n' individuals we have (2n+1) categories for each site, each value is encoded as ctype double which on 'all known' platforms occupies 8 bytes. The (2n+1) values are log scaled like ratios, which means that the most likely category will have a value of 0.

The information for the first site is the first (2n+1)sizeof(double) bytes. The information for the next site is the next (2n+1) bytes etc. The positions are given by the ascii file called '.saf.pos'


<pre>
#sample code to read .saf in c/c++ assuming 10 individuals
FILE *fp = fopen("mySFS.saf","rb");
int nInd = 10;
double persite[2*nInd+1];
fread(persite,sizeof(double)*(2*nInd+1),1,fp);
for(int i=0;i<2*nind+1;i++)
   fprintf(stderr,"cat: %d = %f\n",i,persite[i]);
</pre>

* If the -fold 1 has been set, then the dimension is no longer 2*nInd+1 but nInd+1
* If the -pest parameter has been supplied the output is no longer likelihoods but log posterior site allele frequencies

=How to plot=
Assuming the we have obtained a single global sfs(only one line in the output) from emOptim2 program, and this is located in '''file.saf.sfs''', then we can plot the results simply like:
<pre>
sfs<-exp(scan("small.sfs")) #read in the log sfs
barplot(sfs[-c(1,length(sfs))]) #plot variable sites 
</pre>
[[File:Sfs1.png|thumb]]
We can make it more fancy like below:

<pre>
#function to normalize
norm <- function(x) x/sum(x)
#read data
sfs <- exp(scan("small.sfs"))
#the variability as percentile
pvar<- (1-sfs[1]-sfs[length(sfs)])*100
#the variable categories of the sfs
sfs<-norm(sfs[-c(1,length(sfs))]) 
barplot(sfs,legend=paste("Variability:= ",round(pvar,3),"%"),xlab="Chromosomes",
names=1:length(sfs),ylab="Proportions",main="mySFS plot",col='blue')
</pre>
[[File:Sfs2.png|thumb]]

If your output from emOptim2 contains more than one line, it is because you have estimated multiple local SFS's. Then you can't use the above commands directly but should first pick a specific row.

<pre>
sfs<-exp(as.numeric(read.table("multiple.sfs")[1,])) #first region.
#do the above
sfs<-exp(as.numeric(read.table("multiple.sfs")[2,])) #second region.
</pre>