文章目录介绍环境搭建软件下载结果展示基因数据下载流程基因数据处理利用GEO分析绘制拟火山图注意，本系列有连贯性，每一步都很详细，每一步都很重要，请耐心读完！！

PyVCF是一个用于处理VCF(VariantCallFormat)文件的python库。它提供了许多功能来读取，过滤和修改VCF文件中的变异PyVCF是一个用于读取和写入VCF格式文件的Python库。主要功能包括:1、读取VCF文件：PyVCF提供了一个vcf.Reader()函数，可以用来打开并读取VCF文件。读取后的文件可以进行遍历，每个元素是一个vcf.model._Record的实例2

从计算机辅助药物设计、疫苗研发，到基因数据处理与分析，再到提供精准医疗服务于肿瘤治疗、产前筛查等医疗技术，高性能计算HPC在生命科学研究中扮演着十分重要的角色。

从计算机辅助药物设计、疫苗研发，到基因数据处理与分析，再到提供精准医疗服务于肿瘤治疗、产前筛查等医疗技术，高性能计算HPC在生命科学研究中扮演着十分重要的角色。

从计算机辅助药物设计、疫苗研发，到基因数据处理与分析，再到提供精准医疗服务于肿瘤治疗、产前筛查等医疗技术，高性能计算HPC在生命科学研究中扮演着十分重要的角色。

但现有的基因数据处理方式大多数是只利用一种基因数据，例如只使用DNA或者是只使用mRNA，不能综合所有

1.代码：/***@authorxubo*morecode:https://github.com/xubo245/SparkLearning*moreblog:http://blog.csdn.net/xubo245*/packageorg.gcdss.cli.diseaseimportjava.text.SimpleDateFormatimportjava.util.Dateimportorg.

1.介绍：cs-bwamem依赖的是adam-0.14.0里面的adamSave存储之后无法用adam-0.18.2的结果读取：2.adam-0.18.2adamsave可以用loadParquetAlignments读取。但无法用loadParquetAlignments读取adam-0.14.23.代码：packageorg.gcdss.cli.testimportjava.nio.file.

1.GRCH38chr14：hadoop@Master:~/xubo/project/load$./load.shstart:1SLF4J:Failedtoloadclass"org.slf4j.impl.StaticLoggerBinder".SLF4J:Defaultingtono-operation(NOP)loggerimplementationSLF4J:Seehttp://www.sl

1.分两组(1)kmer长度为：5to21(2)kmer长度为：5to55by102.代码：packageorg.gcdss.cliimportjava.text.SimpleDateFormatimportjava.util._importorg.apache.spark._importorg.bdgenomics.adam.projections.{AlignmentRecordField,P

```hadoop@Master:~/cloud/adam/xubo/data/GRCH38Sub/cs-bwamem$bwamemGRCH38BWAindex/GRCH38chr1L3556522.fastag38L100c10000000Nhs20.fq>g38L100c10000000Nhs20.bwamem.sam[M::bwa_idx_load_from_disk]read0ALTcon

1.参考：packageorg.bdgenomics.adam.cliclassFlattenSuiteextendsADAMFunSuite{valloader=Thread.currentThread().getContextClassLoadervalinputPath=loader.getResource("small.vcf").getPathvaloutputFile=File.cre

1.small.vcf:没记录2.读取：5loadtime:3287ms{"variant":{"variantErrorProbability":139,"contig":{"contigName":"1","contigLength":null,"contigMD5":null,"referenceURL":null,"assembly":null,"species":null,"refere

基因数据处理2之ftp数据快速查找linux下可以用grep:curl-s"ftp://ftp.1000genomes.ebi.ac.uk/vol1/ftp/current.tree"|grepNA12878

1.从cloud/adam移出到xubo/ref:hadoop@Master:~/cloud/adam/xubo/data/test20160310$mkdir-p~/xubo/ref/GRCH38Index/hadoop@Master:~/cloud/adam/xubo/data/test20160310$mvGCA_000001405.15_GRCh38/*~/xubo/ref/GRCH38I

指令：```hadoop@Master:~/cloud/adam/xubo/data/GRCH38Sub/cs-bwamem$bwamemGRCH38BWAindex/GRCH38chr1L3556522.fastag38L100c10000000Nhs20Paired1.fqg38L100c10000000Nhs20Paired2.fq>g38L100c10000000Nhs20Paired12

基因数据处理1之mapping_to_cram参考资料：AWorkedExampleObtainsomepublicdataWewillusethefirst100,000read-pairsfromayeastdataset.curlftp

1.下载：gitclonehttps://github.com/broadgsa/gatk-protected.git2.安装：gitcheckout3.5mvncleanpackage-DskipTests3.安装成功：[INFO]ReactorSummary:[INFO][INFO]GATKRoot..........................................SUCCES

1.代码：/***@authorxubo*/packageorg.gcdss.cli.vcfimportorg.apache.spark.{SparkConf,SparkContext}/***Createdbyxuboon2016/5/23.*/objectextractGRCH38chr20vcf{defmain(args:Array[String]){valconf=newSparkConf

1.指令：（1）samtoolsmpileup-vfHomo_sapiens_assembly19chr20.fastaNA12878_snp_A2G_chr20_225058.sorted.bam>NA12878_snp_A2G_chr20_225058.variants或者：samtoolsmpileup-vfHomo_sapiens_assembly19chr20.fastaNA12878_

更多代码请见：https://github.com/xubo245基因数据处理系列1.解释很久没运行SparkBWA了，系统文件有点多，重新运行。

更多代码请见：https://github.com/xubo245基因数据处理系列1.解释重新运行，跟换了文件地址后来终端，需要运行其他的2.代码：hadoop@Master:~/disk2/xubo/

前面一片已经讲了遇到的问题，之前也遇到过：基因数据处理31之avocado运行avocado-cli中的avocado问题3-变异识别找不到RecordGroupSample（null）http://blog.csdn.net

1.总的脚本hadoop@Master:~/xubo/project/alignment$catrunSparkBWAAndCsbwamem.shecho"startsparkBWA"./sparkBWA/sparkBWA.shecho"startCsbwamemalgin"./cs-bwamem/csbwamemAlign.shecho"startCsbwamemmerge"./cs-bwame

脚本：hadoop@Master:~/xubo/project/alignment/sparkBWA$catg38L100c10000000Nhs20Paired12Bwamem.shecho"start"startTime4=`date+"%s.%N"`time4=`date+"%Y%m%d%H%M%S"`#spark-submit--classorg.apache.spark.examples

（一）问题问题1：avocado数据读取：avocado存到磁盘是：RDD[Genotype]valprocessedGenotypes:RDD[Genotype]=postProcessVariants(calledVariants,stats).flatMap(variantContext=>variantContext.genotypes)//savevariantstooutputfile

数据来自：http://219.219.220.149:18080/history/application_1466866953605_0001/executors/http://219.219.220.248:8042/node/containerlogs/container_1466866953605_0001_01_000003/hadoop/stderr/?start=0SLF4J:Cla

更多代码请见：https://github.com/xubo245基因数据处理系列之SparkBWA1.解释1.1简述当partitions超过节点数量的时候Lostexecutor的问题，已经提交到SparkBWA

脚本：spark-submit--classSparkBWA\--masterlocal\--archivesbwa.zip\SparkBWA.jar\-algorithmmem-readspaired\-index/home/hadoop/xubo/ref/GRCH38L1Index/GRCH38chr1L3556522.fasta\-partitions3\/xubo/alignment/sp

1.修改Makefile.common：将LIBBWA_LIBS=-lrt改为LIBBWA_LIBS=-lrt-lz不然会报错误【5】2.make之后修改java.library.path步骤：vi/etc/profile加入exportLD_LIBRARY_PATH=/home/hadoop/xubo/tools/SparkBWA/build:$LD_LIBRARY_PATH使生效：source

运行记录：hadoop@Master:~/xubo/ref/GRCH38L1Index/pe$bwamem../GRCH38chr1L3556522.fastaGRCH38chr1L3556522N1000000L100paired1.fastqGRCH38chr1L3556522N1000000L100paired2.fastq>GRCH38chr1L3556522N1000000L100pai

脚本1：spark-submit--classSparkBWA\--masterspark://219.219.220.149:7077\--conf"spark.executor.extraJavaOptions=-Djava.library.path=/home/hadoop/xubo/tools/SparkBWA/build"\--driver-java-options"-Djava.lib

xubo@xubo:~/xubo/data/alignment/cs-bwamem$bwamembwaindex/GRCH38chr1L3556522.fastag38l500N10000.fq>g38l500N10000.bwamem.sam[M::bwa_idx_load_from_disk]read0ALTcontigs[M::process]read10000sequences(50000

脚本：hadoop@Mcnode1:~/xubo/tools/SparkBWA/build$catpaired.shspark-submit--classSparkBWA\--masteryarn-client\--conf"spark.executor.extraJavaOptions=-XX:MaxPermSize=1024M"--driver-java-options"-Djava.libr

hadoop@Master:~/xubo/project/alignment/sparkBWA$./paired.shUsingpropertiesfile:/home/hadoop/cloud/spark-1.5.2/conf/spark-defaults.confAddingdefaultproperty:spark.executor.extraJavaOptions=-Djava.libra

1.安装gitclonehttps://github.com/citiususc/SparkBWA.gitcdSparkBWAmake2.使用：报错：没有成功hadoop@Master:~/xubo/project/alignment/sparkBWA$./run.shError:Mustspecifyaprimaryresource(JARorPythonorRfile)Runwith--hel

记录：hadoop@Master:~/cloud/adam/xubo/data/GRCH38Sub/cs-bwamem$snap-alignersinglesnap/snapindexg38L100c10000000Nhs20.fq-osnap/g38L100c10000000Nhs20.snap.samWelcometoSNAPversion1.0beta.23.Loadingindexfrom

core用不少了，只用了4个，实际可以14个。1.由于GRCH过大，及其内存小，运行不了全基因组匹配2.上传：spark-submit--classcs.ucla.edu.bwaspark.BWAMEMSpark--masterspark://219.219.220.149:7077/home/hadoop/xubo/tools/cloud-scale-bwamem-0.2.1/target/cl

脚本：hadoop@Master:~/xubo/project/alignment/sparkBWA$catpairedERR.shspark-submit--classSparkBWA\--masterlocal[4]\--driver-memory1500m\--executor-memory1500m\--executor-cores1\--archivesbwa.zip\--verbose

1.构建索引：bwaindexref.fa或者从ftp下载，请参考【1】2.BWA：bwa.sh为脚本文件hadoop@Master:~/cloud/adam/xubo/data/test20160310/bwa3sh$catbwa.sh#!/bin/bashbwaaln../GCA_000001405.15_GRCh38/GCA_000001405.15_GRCh38_full_analysis

（1）pair1.fq》saibwaalnGRCH38BWAindex/GRCH38chr1L3556522.fastag38L100c10000000Nhs20Paired1.fq>g38L100c10000000Nhs20Paired1.saipair1记录：hadoop@Master:~/cloud/adam/xubo/data/GRCH38Sub/cs-bwamem$bwaalnGRCH3

基因数据处理8之BWA_MEM小数据集处理环境：ubuntu14.046G内存参考基因：GRCH38来源请参考【1】1.fastq数据：SRR003161.fastq的头20行，即5条reads操作记录

版本：https://github.com/ytchen0323/cloud-scale-bwamem/releases/tag/v0.2.11.需要设置spark路径：/cluster/spark/spark-1.3.1-bin-hadoop2.4/lib/spark-assembly-1.3.1-hadoop2.4.0.jar-->/home/hadoop/cloud/spark-1.5.2/

基因数据处理7之BWA_MEM运行太长hadoop@Master:~/cloud/adam/xubo/data/test20160310$bwamemGCA_000001405.15_GRCh38/GCA

1.下载：https://github.com/lh3/wgsim可以git或者zip2.安装：gcc-g-O2-Wall-owgsimwgsim.c-lz-lm3.数据下载：可以使用bwakit下载：https://github.com/lh3/bwa/tree/master/bwakit下载：bwa.kit/run-gen-refhs38DH4.使用方法和默认配置：hadoop@Master:

1.概念：ART基因序列数据生成器详细请见论文：【1】和官网【2】2.下载：ART-bin-GreatSmokyMountains-04.17.16-Linux64.tgzhttp://www.niehs.nih.gov/research/resources/assets/docs/artbingreatsmokymountains041716linux64tgz.tgz3.配置sudocp到用户

1.先使用art生成数据：请看前一篇2.上传fastq到hdfs：hadoop@Master:~/cloud/adam/xubo/data/GRCH38Sub/cs-bwamem$spark-submit--classcs.ucla.edu.bwaspark.BWAMEMSpark--masterlocal[2]/home/hadoop/xubo/tools/cloud-scale-bwamem-

1.art生成模拟序列：art_illumina-ssHS20-iGRCH38BWAindex/GRCH38chr1L3556522.fna-l100-c10000000-og38L100c10000000Nhs202.上传到hdfs，制定partition数spark-submit--classcs.ucla.edu.bwaspark.BWAMEMSpark--masterspark://mas

1.使用的是之前的数据：spark-submit–classcs.ucla.edu.bwaspark.BWAMEMSpark–masterlocal[2]/home/hadoop/xubo/tools/cloud-scale-bwamem-0.2.1/target/cloud-scale-bwamem-0.2.0-assembly.jarupload-fastq01fastq/g38L100c50

hadoop@Master:~/cloud/adam/xubo/data/GRCH38Sub/cs-bwamem$snap-alignerindexGRCH38BWAindex/GRCH38chr1L3556522.fastasnapindexWelcometoSNAPversion1.0beta.23.Hashtableslack0.300000LoadingFASTAfile'GRCH38BW