SAM 3.2.1 buildmodel - create a new model from a family of sequences, or refine an existing model R. Hughey & A. Krogh R. Hughey and A. Krogh., SAM: Sequence alignment and modeling software system. Technical Report UCSC-CRL-96-22, University of California, Santa Cruz, CA, September 1996. http://www.cse.ucsc.edu/research/compbio/ismb99.tutorial.html http://www.cse.ucsc.edu/research/compbio/sam.html buildmodel buildmodel seqlab buildmodel perl "buildmodel" 0 run Run name perl " $value" test 1 train Train sequences perl " -train $value" 2 seqsfile perl 1 sam_model_file $run.mod sam_model perl 1 outfiles *.freq *.stat *.weightoutput input Input options 2 alphabet Alphabet (-alphabet) perl ($value)? " -alphabet $value":"" RNA DNA protein train_test_sets Training and test sets train2 Train sequences, second set perl ($value)? " -train2 $value": "" test Test sequences perl ($value)? " -test $value" : "" test2 Test sequences, second set perl ($value)? " -test2 $value": "" nseq Chose at most this number of sequences from each set (-nseq) perl ($value)? " -nseq $value" : "" ntrain Pick randomly this number of sequences from all files specified as training set (-ntrain) perl (defined $value) ? " -ntrain $value" : "" If this option is set, the system partition randominto the training and the test set. It will randomly pick 'ntrain' sequences from all files specified (training and testing) using the random seed ''trainseed' and reserve the rest for the test set. trainseed Random seed for partitioning the sequences into the test set and the training set (-trainseed) perl (defined $value)? " -trainseed $value" : "" By default, the seed is set to the process ID number, which is printed on the output file so that the partition can be reproduced. init_models_alignments Initial models and alignments nmodels How many initial models (-nmodels) perl ($value)? " -nmodels $value" : "" alignfile Initial alignment for initial model (-alignfile) perl ($value)? " -alignfile $value" : "" 8 One of the best ways to train a hidden Markov model is to use an existing rough alignment to get the process started. In this case, any initial models are ignored in favor of this starting alignment. The 'alignment_weights' parameter can be used to specify a file of weights, or 'aweight_method' can be used to internally calculate sequence weights based on the initial alignment. readseq_ok_alig perl 1 sequence_models When greater than zero, build initial model from randomly selected sequences in the training set (-sequence_models) perl (defined $value && $value != $vdef) ? " -sequence_models $value" : "" 0.0 perl ! $model_file && $alignfile If greater than zero, models from randomly chosen single sequences in the training set are created. As a subcase of an initial alignment, For each of the initial models required by buildmodel, a random sequence will be chosen and a model created based on that sequence regularized with a weight equal to the value of sequence_models. As long as fewer models are created than sequences in the training set, a different sequence will be chosen for each model. Noise will be reduced according to retrain_noise_scale. model_file Initial model (-model_file) perl ($value)? " -model_file $value" : "" sam_model perl 1 This is ignored if an initial alignment is done. perl ! $alignfile nullmodel_file Null model (-nullmodel_file) perl ($value)? " -nullmodel_file $value" : "" control Control options 2 seed Seed number for noise generation and for selection of initial model lengths (-seed) perl (defined $value)? " -seed $value" : "" rerun Rerun the second best model = this number (-rerun) perl ($value)? " -rerun $value" : "" dpstyle Flavor of internal dynamic programming for scoring and training (-dpstyle) perl ($value) ? " -dpstyle $value" : "" 0 0 1 4 5 , 0 or 1: For increased speed and performance (and possible worse results), Viterbi (1) training can be used rather than the default EM training (0). In some cases this is not robust. . 4 or 5: training based on the posterior-decoded alignments. These options will more than double computation time and are not recommended. SW Sequence scoring for Viterbi (-SW) perl ($value)? " -SW $value":"" 2 0 1 2 perl $viterbi model_length Model length parameters 2 modellength Models length (-modellength) perl ($value)? " -modellength $value" : "" minmodlen Min models length (-minmodlen) perl ($value)? " -minmodlen $value" : "" maxmodlen Max models length (-maxmodlen) perl ($value)? " -maxmodlen $value" : "" model_abort_length Models abort length (-model_abort_length) perl ($value && $value != $vdef)? " -model_abort_length $value" : "" 10000 regul Regularizers and mixtures parameters 2 regularizerfile Regularizer (-regularizerfile) perl ($value)? " -regularizerfile /local/gensoft/lib/sam/$value":"" 2 long_match.regularizer trained.regularizer weak-gap.regularizer sam1.3.regularizer trained.regularizer: Regularizer optimized for unweighted transition counts on some set of re-estimated HSSP alignments cheap_gap.regularizer: Makes gap opening and closing very cheap allowing exploration of many different alignments, but giving too high a cost to long matches long_match.regularizer: Assigns somewhat reasonable gap costs for unweighted data, useful for sweeping away 'chatter' into big matches and big gaps, by making gap opening expensive but gap extension fairly cheap. reglength Length of the regularizer (-reglength) perl ($value)? " -reglength $value" : "" 2 priorlibrary Dirichlet mixture prior (-priorlibrary) perl ($value && $value ne $vdef)? " -priorlibrary /local/gensoft/lib/sam/$value":"" recode1.20comp mall-opt.9comp opt-weight1.9comp uprior.9comp null.1comp recode1.20comp uprior9.plib: The 9-component library discussed in the aforementioned paper. Optimized for unweighted blocks data. mall-opt.9comp: Library re-optimized for unweighted data from an HSSP subset. opt-weight1.9comp: Library reoptimized for weighted version of same HSSP subset. recode1.20comp: A 20-component Dirichlet mixture trained on (realigned) HSSP alignments that have a diverse set of sequences. Intended for use in recoding inputs to neural net, but also useful as a standard regularizer. null.1comp: A one-component regularizer with tiny alpha, to get effectively no regularization. prior_weight Weight of the prior library (-prior_weight) perl ($value && $value != $vdef)? " -prior_weight $value" : "" 1.0 2 perl $priorlibrary del_jump_conf Confidence in the regularizer for transitions leaving a delete state (-del_jump_conf) perl ($value && $value != $vdef)? " -del_jump_conf $value" : "" 1.0 The regularizer's transition values are multiplied by this number. ins_jump_conf Confidence in the regularizer for transitions leaving an insert state (-ins_jump_conf) perl ($value && $value != $vdef)? " -ins_jump_conf $value" : "" 1.0 insconf Confidence in the regularizer for character probabilities in an insert state (-insconf) perl ($value && $value != $vdef)? " -insconf $value" : "" 10000 The high default means that the regularizer will overpower the actual counts determined by aligning sequences to the model. The regularizer's character insert values are multiplied by this number. match_jump_conf Confidence in the regularizer for transitions leaving a match state (-match_jump_conf) perl ($value && $value != $vdef)? " -match_jump_conf $value" : "" 1.0 2 matchconf Confidence in the regularizer for character probabilities in a match state (-matchconf) perl ($value && $value != $vdef)? " -matchconf $value" : "" 1.0 2 regul_struct Regularizers with structural information parameters 2 trans_priors Structure-specific transition prior library (-trans_priors) perl ($value)? " -trans_priors /local/gensoft/lib/sam/$value":"" 2 TransFromRev15.plib transweight A multiplier that affects the influence of the pseudocounts generated by the structure-specific transition priors (-transweight) perl ($value && $value != $vdef)? " -transweight $value" : "" 1.0 2 template Template three- column file (amino acid sequence, secondary structure, accessibility) (-template) perl ($value)? " -template $value" : "" 2 noise Noise and annealing parameters 2 initial_noise Amount of noise to add for the first iteration, when greater than zero (-initial_noise) perl ($value && $value != $vdef)? " -initial_noise $value" : "" -1.0 It serves the important purpose to make models differ, if the program runs many models simultaneously. anneal_noise Amount of noise to add to the model (-anneal_noise) perl ($value && $value != $vdef)? " -anneal_noise $value" : "" 5.0 If greater than 0, annealing is performed. If 'initial_noise' is also given, that will determine the noise for the first iteration, and 'anneal_noise' the noise in the following iterations. During the estimation process the annealing noise is decreased by a speed determined by 'anneal_length'. anneal_length Speed with which noise should be decreased to zero (-anneal_length) perl ($value && $value != $vdef)? " -anneal_length $value" : "" 0.8 When greater than or equal to 1, the noise is decreased linearly. When less than 1, the noise is decreased exponentially by multiplying the noise with this value in each iteration. In this case noise injection is halted when the amount of noise reaches 10% of its initial value. randomize Determines how noise is added to the model (-randomize) perl ($value && $value != $vdef)? " -randomize $value" : "" 50 . greater than 0 - a set of 'randomize' random paths are calculated through the model according the the regularizer probabilities. Each of these sequences is weighted by the amount of noise. These sequences are added to the counts generated by the real frequencies, thus the noise setting is somewhat dependent on the number of sequences being trained. . less than 0 - a set of '-randomize' random paths are calculated through the model. With a weight of -noise/randomize, these counts are added to the normalized (probability, rather frequency) model, and than the model is renormalized. Thus, the noise generation is similar to that of the first case, but total noise added is independent of both the number of sequences and the randomize setting. . equal to zero - for each probability parameter in the model, a random number between 0 and the corresponding parameter in the normalized regularizer is found. This number is scaled by the level of the noise, given for instance by 'initial_noise', and added to the probability in the model. After doing this for the whole model, all the probabilities are normalized. This is the fastest means of noise generation. retrain_noise_scale Scale by this multiplier initial_noise (-retrain_noise_scale) perl ($value && $value != $vdef)? " -retrain_noise_scale $value" : "" 0.1 After a model has been created, adding too much noise to the model may eliminate all the trained information. Therefore, if an initial model or an initial alignment is specified, noise ('initial_noise' or 'anneal_noise') is reduced from the default setting by a factor of this value. surgery_noise_scale After the first surgery, anneal_noise is scaled by this multiplier, which must be between 0.0 and 1.0 (-surgery_noise_scale) perl ($value && $value != $vdef)? " -surgery_noise_scale $value" : "" 0.1 In the case of a surgery (see below), the starting noise of the re-estimation process after a surgery, whether or not an initial model is specified, is the 'anneal_noise' scaled by this parameter. surgery Surgery parameters 3 It is often the case that during the course of learning, some match states in the model are used by few sequences, while some insertion states are used by many sequences. Model Surgery is a means of dynamically adjusting the model during training. The basic operation of surgery is to delete unused match states and to insert match states in place of over-used insert states In the default case, any match state used by less than one half of the sequences is removed, forcing those sequences to use an insert state or to significantly change their alignment to the model. Similarly, any insert state used by more than half sequences is replaced with a number of match states approximating the average number of characters inserted by that insert state. nsurgery Maximum number of surgeries to perform (-nsurgery) perl (defined $value && $value != $vdef)? " -nsurgery $value" : "" 3 Surgery will be 'nsurgery' times: a full re-estimation process is performed including 'reestimates' re-estimations, or until the 'stopcriterion' is reached. By default surgery is performed up to two times. stopcriterion The reestimation loop will stop whenever the improvement in the NLL score is less than this number (-stopcriterion) perl ($value && $value != $vdef)? " -stopcriterion $value" : "" 0.1 reestimates Maximum number of reestimates to perform after a surgery (-reestimates) perl ($value && $value != $vdef)? " -reestimates $value":"" 40 mainline_cutoff Confidence in the regularizer for transitions leaving a match state (-mainline_cutoff) perl ($value && $value != $vdef)? " -mainline_cutoff $value" : "" 0.5 Setting this option to a number other than the default will indicate how much non-match, or main line, activity will be accepted. For example, a setting of 0.25 indicates that any match state used by less than one quarter of the sequences should be removed, while any insert state used by more than one quarter of the sequences should be expanded into a number of match states approximately equal to the average number of characters emitted by that state. For finer tuning of the surgery process, the parameters 'cutmatch', 'cutinsert', and 'fracinsert', can be used. cutmatch Remove any match state with a smaller portion of sequences than this (-cutmatch) perl ($value && $value != $vdef)? " -cutmatch $value" : "" 0.5 perl $mainline_cutoff != 0.5 cutinsert Replace any insert state with a higher portion of sequences than this (-cutinsert) perl ($value && $value != $vdef)? " -cutinsert $value" : "" 0.5 perl $mainline_cutoff != 0.5 fracinsert When inserted states are to replace, replace it with the average number or characters generated by the insert state multiplied by this number (-fracinsert) perl ($value && $value != $vdef)? " -fracinsert $value" : "" 1.0 perl $mainline_cutoff != 0.5 maxinserts Maximum number of states inserted after any node by the surgery (-maxinserts) perl ($value && $value != $vdef)? " -maxinserts $value" : "" 100 weight Weight parameters 4 weight_filename Use external parameters for sequence weigthing specified in this file (-weight_filename) perl ($value)? " -weight_filename $value" : "" In this file, any line starting with a percent sign is ignored as a comment. The first non-comment line is presumed to be a description of the weights file. The next non-comment line contains two integers, the number of weighted sequences and the number of weighted subfamilies. Remaining uncommented lines consist of a sequence identifier, white-space, and floating-point sequence weights, one per family. Weights can be positive, negative, or zero, and need not sum to one. If a sequence does not have a corresponding weight, its weight is set to 1.0 and a message is printed. Sequences and weights do not need to be in the same order within their respective files. alignment_weights Use external parameters specified in this file to weight alignments used to form initial models (-alignment_weights) perl ($value)? " -alignment_weights $value" : "" perl $alignfile See at 'weight_filename' just above for the syntax. align_weight_method Internal weighting method of alignments (-aweight_method) perl ($value != $vdef) ? "-aweight_method $value" : "" 2 0 1 2 3 perl $alignfile aweight_bits Target bits per column to save in determining alignment sequence weighting (-aweight_bits) perl (defined $value && $value != $vdef) ? " -aweight_bits $value)" : "" 0.5 perl $alignfile For the Henikoff and flat weighting schemes, if this parameter is set negative, the initial weighting is used, without adjusting the total weights to get a specified number of bits saved. The initial weighting in these schemes has a total weight of: number_sequences^'aweight_exponent'. aweight_exp Exponent in determing alignment sequence weighting (-aweight_exponent) perl (defined $value && $value != $vdef) ? " -aweight_bits $value)" : "" 0.5 perl $alignfile weight_length An annealing schedule for the sequence weight multiplier (-weight_length) perl ($value)? " -weight_length $value" : "" When greater than or equal to 1, the weight multiplier is increased linearly to 'weight_final' in 'anneal_length' iterations. When less than 1, the multiplier is increased exponentially until the multiplier reaches 90% of its final value. weight_final The final (steady-state) multiplier of sequence weights (-weight_final) perl ($value && $value != $vdef)? " -weight_final $value" : "" 1.0 internal_weight Use internal maximum discrimination sequence weighting during training (-internal_weight perl ($value != $vdef) ? " -internal_weight $value" : "" 0 0 1 2 When using method 1, sequences with very poor scores may get excessively large weights. Method 2 modifies the weights of such outlier sequences. If a sequence scores so badly that it exceeds the median score by three standard deviations, it is weighted with a decreasing linear weight function, reaching a minimum of 1.0 for the sequence with the worst score. perl (! $weight_filename) && (! $alignment_weights) iweight User controlled parameter for internal weighting (-iweight) perl (defined $value && $value != $vdef) ? " -iweight $value)" : "" 0.1 perl $internal_weight The worst scoring sequence will have a weight of 1, while the best scoring sequence will have a weight of this parameter, typically in the 0.01 to 0.1 range. constraint_opt Constraint options 5 SAM HMM constraints provide a mechanism for associating position in a sequence with a specific node in a model. During model training, a constrained position remains associated with its node, allowing the remaining portions of the sequence to align naturally to the model. This serves as a method of incorporating prior knowledge about the training sequence, such as structurally similar regions. Currently, there is an inconsistency in scoring constraints, in that for EM-style scoring, the constrained score is reported, which will generally provide a very strong signal. For Viterbi or posterior-alignment scoring, constraints are used to find the best path, but the unconstrained score of that path is reported, and thus the score of a constrained sequence is similar to that of an unconstrained sequence. constraints Constraints File (-constraints) perl (defined $value) ? " -constraints $value" : "" The node labels in the constraints file are assigned to nodes with the same node number. If necessary, the length of the model will be increased to allow for all labels. If 'surgery' is used, an unlabeled node may be added or removed, resulting in the labels and node numbers differing in the final model. When retraining a model using constraints, the model must already contain labels. Constraints work with both EM and Viterbi. . Constraints File Format: Lines starting with '#' are comments and ignored. The file is divided into sections starting with a line containing only the section name followed by a colon. Currently there is only one type of section containing sequence constraints, starting with : constraints: Multiple constraints sections may appear, allowing for the easy combining of files. A constraint entry defines the model nodes to which residues of a sequence are constrained. Multiple entries may appear for a sequence, however a given residue of a sequence may be constrained only once. Constraints entries consists of lines in the form: seqid constrdef, constrdef, ... Where the 'seqid' is any sequence identifier consisting of non-whitespace characters and 'constrdef' defines a set of model node labels for a range of positions in the sequence in any combination of the following formats: startpos-endpos: startlabel Specifies that consecutive residues starting at startpos and ending at endpos are constrained to consecutive node labels starting at startlabel. seqpos: label Constrains a single position to a node label. startpos-endpos Specifies that these residues are to be constrained, but doesn't actually associate a label with the positions. This is used when the labels are to be assigned from an alignment by modelfromalign. seqpos Specifies a single residue to be constrained from an alignment. empty An entry may be empty (zero or more white space) and is ignored. This can be useful when generating constraint files. Positions in sequences are identified by one-based indices and node labels are positive numbers. A simple example of a constraint file is shown here. constraints: PROT1 10-30: 8, 80-82: 100 PROT2 12-28: 10, 62: 100 PROT3 16-32: 12 const_out_file new_constraints.crt perl defined $constraints constraints_out Create a new constraints definition file (-constraints_out) 0 perl ($value) ? " -constraints_out $const_out_file" : "" constraints_from_align Constraints from align (-constraints_from_align) perl ($value) ? " -constraints_from_align 1" : "" 0 perl defined $constraints fim FIM parameters 6 FIM_method_train Table probabilities for FIMs (-FIM_method_train) perl ($value && $value != $vdef)? " -FIM_method_train $value":"" -1 2 -1 0 1 2 3 5 6 FIM_method_score Table probabilities for FIMs (insert prob) (-FIM_method_score) perl ($value && $value != $vdef)? " -FIM_method_score $value":"" -6 2 -6 0 1 2 3 4 5 6 Insert_method_train Table probabilities for nodes that are not FIMs (-Insert_method_train) perl ($value && $value != $vdef)? " -Insert_method_train $value":"" -1 2 -1 0 1 2 3 5 6 Insert_method_score Table probabilities for nodes that are not FIMs (insert prob) (-Insert_method_score) perl ($value && $value != $vdef)? " -Insert_method_score $value":"" 0 2 0 1 2 3 4 5 6 train_reset_inserts Set insert and FIM character tables according to this variabl (-train_reset_inserts) perl ($value && $value ne $vdef)? " -train_reset_inserts $value":"" 6 -1 0 1 2 3 5 6 fimstrength Factor by which to multiply the FIM letter emission probabilities (-fimstrength) perl ($value && $value != $vdef)? " -fimstrength $value" : "" 1.0 If sequences are not using the FIM enough, it can be adjusted with this parameter. Changing this parameter its default from 1.0 to 2.0, for example, make use of the FIM twice as likely as before. The value of this parameter is also applied to simple null models, and if less than zero, its absolute value is applied to both FIMs and normal insertion states. fimtrans Factor to adjust the cost of the insert to insert loop within the FIM (-fimtrans) perl ($value && $value != $vdef)? " -fimtrans $value" : "" 0.0 If set to a non-zero value, this option becomes a multiplier of the geometric average of the match to match transitions in the model. Thus, if set to 1.0 (the default), the FIM insert-to-insert transition will cost the same as the average match to match transition. When set to a negative number, non-FIM insert-to-insert transitions are set to p-(1-f)p^2, where p is the regularized and normalized frequency counts for the transition and fis the FIM insert-to-insert transition. The effect of this formula is to ensure that the FIMs remain cheap in comparison to any insert-to-insert transitions, something that can be critical in using buildmodel to train HMMs. align_fim Add FIMs to the ends of the initial model (-align_fim) perl ($value)? " -align_fim 1":"" 0 perl $alignfile output Output options 2 dump_parameters Parameter output (-dump_parameters) perl ($value && $value ne $vdef)? " -dump_parameters $value":"" 0 0 1 2 all_stats Print individual model statistics (-all_stats) perl ($value)? " -all_stats 1":"" 0 2 many_files Print the probability model, frequency model, and run statistics to different files (-many_files) perl ($value)? " -many_files 1":"" 0 print_frequencies Print frequencies (-print_frequencies) perl ($value)? " -print_frequencies 1":"" 0 print_all_models Print models after each iteration of the forward-backward procedure (-print_all_models) perl ($value)? " -print_all_models 1":"" 0 print_surg_models Print models after each surgery (-print_surg_models) perl ($value)? " -print_surg_models 1":"" 0 2 print_all_weights Generates a weight output file is generated after each iteration of the forward-backward procedure (-print_all_weights) perl ($value)? " -print_all_weights 1":"" 0 binary_output Write models in binary format (-binary_output) perl ($value)? " -binary_output 1":"" 0