Pyteomics documentation v4.6a1

History of changes

«  Example 4: Spectrum Annotation   ::   Contents

History of changes

4.6a0

  • When passing an existing file (by name) to pyteomics.mgf.write() or pyteomics.fasta.write and other writing functions, the file will be opened for writing by default. Previously, it would be opened for appending with a warning about the upcoming change. Please be aware that existing files will be overwritten if passed by name. The rationale for this is better reproducibility if the same code is run multiple times. You can use the file_mode argument of the writing functions to override this behaviour, or pass your own file objects.
  • Fix issue #98 (#99 and
#101 by Joshua Klein).

4.5.6

4.5.5

  • Fix issue #77.

4.5.4

  • Fix issue #74.
  • In pyteomics.auxiliary.fdr(), raise PyteomicsError instead of ZeroDivisionError when using formula 1 on input without any target PSMs.
  • Provide more accurate amino acid masses in mass.std_aa_mass.
  • Fix SyntaxError in pyteomics.pylab_aux on Python 2.7.

4.5.3

  • Fix ThreadPool shutdown and add new parameter ephemeral_pool in pyteomics.usi.PROXIAggregator (#67 by Joshua Klein).
  • Bugfix in pyteomics.proforma.GenericModificationResolver (#68 by Joshua Klein).
  • New helper function pyteomics.fasta.decoy_entries().
  • New arguments charge_carrier, absolute in mass.calculate_mass() and mass.Composition.mass() (#61). Charge is now only handled in Composition.mass() and not Composition.__init__().
  • Bugfix in pyteomics.tandem (#71 by @superrino130).

4.5.2

  • Support Python 3.10.

4.5.1

4.5

4.4.2

  • Add cleavage rules from MS ontology as pyteomics.parser.psims_rules. pyteomics.parser.cleave() now understands keys and accessions from psims_rules as rules.
  • Improve mzIdentML parser performance (and possibly others in some cases) by relying more on offset indexes (#34 by Joshua Klein).
  • Extend the pyteomics.mztab.MzTab parser with auto-generated properties. Almost all metadata entities are now exposed as properties on the parser object (#23 by Joshua Klein).
  • Fix the version parsing in pyteomics.mztab to support shorter vMzTab version strings (#24 by Donavan See).
  • Tweak the pyteomics.pepxml.PepXML parser to present some values that were previously reported as None.
  • Fix compatibility with SQLAlchemy 1.4 (#32 by Joshua Klein).

4.4.1

  • Further tweaked behavior of pyteomics.auxiliary.file_helpers._check_use_index(), which is responsible for handling of use_index in read() functions in parser modules.
  • Fix indexing when element identifiers contain XML-escaped characters (#20 by Joshua Klein).
  • Add support for MzTab 2.0 (#22 by @annalefarova).
  • Also, check out the Pyteomics Discussions page! You can use it to share your thoughts, ask questions, discuss coding practices, etc.

4.4

4.3.3

  • Add pyteomics.electrochem.gravy() (#9 by Vladimir Gorshkov).

  • Fixes and improvements in pyteomics.pepxml.roc_curve() (#10 by Andrey Rozenberg).

  • Changes in guessing behavior of read() functions.

    In modules that implement indexing parsers for non-XML formats (MGF, FASTA, PEFF, ms1/ms2), when a parser is instantiated using read(), the parser class to instantiate is guessed based on the mode of the file object passed to read() (text or binary).

    With some file-like objects, mode cannot be easily deduced without consuming some of the data. You will now see more warnings in case use_index is not explicitly passed to read() and reading mode is not obvious. There will also be warnings if use_index is specified but the file is opened in the wrong mode. To avoid all of this, you are encouraged to instantiate parser classes directly, or explicitly specify use_index to read() in all corner cases.

4.3.2

Fix #7.

4.3.1

Technical release.

4.3

First release after the move to Github. Issue and PR numbers from now on refer to the Github repo. Archive of the Bibucket issues and PRs is stored here.

Changes in this release:

4.2

  • Changes in XML XPath implementation. For standard XML parser classes, this only means a minor change in performance (should be a slight improvement, most noticeable for TandemXML).

    • For custom classes: the implementation of xpath evaluation in pyteomics.xml.XML.iterfind() has changed. Pseudo-conditions are now not supported. Instead, an attempt is made to support full XPath. The main difference is that the XPath is evaluated on XML elements, whereas pseudo-conditions used to be evaluated for complete Python dictionaries. To reproduce old behavior, you can just write an explicit if statement at an appropriate place. New implementation allows actually skipping the elements that do not satisfy the XPath predicate. When writing classes which by default iterate over elements based on a complex XPath, set _default_iter_path instead of _default_iter_tag.

      Warning

      Beware that if _default_iter_path differs from _default_iter_tag and you use indexing, all elements corresponding to _default_iter_tag will be indexed. This is a limitation of the index building procedure. This discrepancy will lead to confusing behavior (length checks, membership tests and other things based on index will not correspond to items returned by iteration). map() calls will also operate on the full index.

  • New keyword arguments queue_size, queue_timeout and processes for indexed parsers with support for map().

  • New method mass.Unimod.by_id(). Also, mass.Unimod now supports dict-like queries with record IDs.

  • Reduce memory footprint for unit primitives (PR #35 by Joshua Klein).

  • New functions pyteomics.auxiliary.sigma_T() and pyteomics.auxiliary.sigma_fdr().

  • Fix issues #44, #46, #47, #48.

4.1.2

Bugfix: fix the standard mass value for pyrrolysine (issue #42).

4.1.1

  • Add numpress support for mzML and mzXML files. To read files compressed with Numpress, install pynumpress (PyPI, GitHub).
  • Bugfixes.

API changes

  • In ms1.read() and ms2.read(), the default value for use_index is now False. Using the indexed parsers may result in incorrect behavior if the “first” scan number in S-lines is not unique.

4.1

4.0.1

Fix issue #35 (incorrect order of deserialized offset indexes on older Python versions).

4.0

See also

Pyteomics 4.0: five years of development of a Python proteomics framework

3.5.1

Technical release to update the package metadata on PyPI. Project documentation on pythonhosted.org has been deleted. Latest documentation is available at: https://pyteomics.readthedocs.io/.

3.5

  • Preserve accession information on cvParam elements in mzML parser. Dictionaries produced by the parser can now be queried by accession using pyteomics.auxiliary.cvquery(). (Contributed by J. Klein)

  • Add optional decode_binary argument in pyteomics.mzml.MzML and pyteomics.mzxml.MzXML. When set to False, the parsers provide binary records suitable for decoding on demand. (Contributed by J. Klein)

  • Add method write_byte_offsets() in pyteomics.mzml.MzML, pyteomics.mzxml.MzXML and pyteomics.mzid.MzIdentML. Byte offsets can be loaded later to speed up random access. (Contributed by J. Klein)

  • Random access to MGF spectrum entries.

    This functionality will be changed in upcoming versions.

  • New module pyteomics.protxml for parsing of ProteinProphet output files.

  • Add PeptideProphet and iProphet analysis information to the output of pyteomics.pepxml.DataFrame().

  • New parameter huge_tree in XML parser constructors and read() functions. It is passed to the underlying lxml calls. Default value is False. Set to True to overcome errors such as: XMLSyntaxError: xmlSAX2Characters: huge text node.

  • New parameter skip_empty_cvparam_values in XML parser constructors. It instructs the parser to treat the empty “value” attributes in cvParam elements as if they were not there. This is helpful in cases when such empty “values” are present in one vendor’s file and absent in another: enabling the parameter will result in more unified output. Default value is False.

  • Change the default value for read_schema to False in XML parsing modules.

  • Change the default value for retrieve_refs to True in MzIdentML constructor.

  • Implement retrieve_refs for pyteomics.mzml.MzML. (Contributed by J. Klein)

  • New parameter keep_cterm in decoy generation functions in pyteomics.fasta.

  • New parameters decoy_prefix and decoy_suffix in all format-specific FDR filtering functions. If the standard is_decoy() function works for your files, you can use these parameters to specify either the prefix or the suffix appended to the protein names in decoy entries.

  • New ion types in pyteomics.mass.std_ion_comp.

  • Bugfixes.

3.4.2

  • New module pyteomics.ms1 for parsing of MS1 files.
  • mass.Composition constructor now accepts ion_type and charge parameters.
  • New functions pyteomics.mzid.DataFrame() and pyteomics.mzid.filter_df(). Their behavior may be refined later on.
  • Changes in behavior of pyteomics.auxiliary.filter() and pyteomics.auxiliary.qvalues():
    • both functions now always return DataFrames with pandas.DataFrame input and full_output=True.
    • string values of key, is_decoy and pep are substituted with simple itemgetter functions for non-pandas, non-numpy input;
    • additional parameters score_label, decoy_label, pep_label, and q_label for output control.
  • Performance optimizations in XML parsing code.

3.4.1

3.4

3.3.1

New submodule pyteomics.featurexml with a parser for OpenMS featureXML files.

3.3

  • mzML and mzIdentML parsers can now create an index of element offsets. This allows quick random access to elements by unique ID.
  • mzML parsers now come in two flavors: pyteomics.mzml.MzML and pyteomics.mzml.PreIndexedMzML. The latter uses the byte offsets listed at the end of the file.
  • New parameters convert_arrays and read_charges in mgf.read() allow using it without numpy and possibly improve performance. The default behavior is retained.
  • Performance optimizations in mgf.read() and parser.cleave().
  • New decoy generation mode called “fused decoy”, described in the paper accepted to JASMS.

API changes

  • pyteomics.parser.cleave() no longer accepts the labels argument. It is emphasized that the input sequences are expected to be in plain one-letter notation, but no checks are performed.
  • DataFrame() functions in pepxml and tandem now extract more protein-related information. The list-like protein-related values can be reported as lists or packed into strings, depending on the optional paramter sep. Some column names have changed as a result.
  • Call signatures of pyteomics.fasta.decoy_sequence() and the functions using it are slightly changed. Standard modes are now also exposed as individual functions.

3.2

New submodule pyteomics.mass.unimod contains rewritten machinery for handling of Unimod relational databases (contributed by Joshua Klein). This is a substitution and extension for the old mass.Unimod class. pyteomics.mass.unimod requires SQLAlchemy.

Other changes:

  • New function pyteomics.auxiliary.linear_regression_perpendicular() provides a linear fit minimizing distances from data points to the fit line (as opposed to pyteomics.auxiliary.linear_regression(), which minimizes vertical distances).
  • Both new and old linear regression functions now accept a single array of shape (N, 2).
  • pyteomics.pylab_aux.scatter_trend() now has an optional parameter regression which can be a callable performing the regression. Also, the regression equation is now the label of the regression line, not the scatter plot.
  • Another two new parameters for pyteomics.pylab_aux.scatter_trend() are sigma_kwargs and sigma_values.
  • pyteomics.pylab_aux functions plot_line() and scatter_trend() now return the objects they create.
  • Writer functions (pyteomics.mgf.write(), pyteomics.fasta.write(), pyteomics.fasta.write_decoy_db()) now accept a file_mode argument that overrides the mode in which the file is opened.
  • In pyteomics.mgf.write() one can now override the format spec for fragment m/z, intensity and charge values using the optinal fragment_format argument. Key order and key-value parameter formatters are now also handled via optional arguments.
  • pyteomics.fasta.decoy_db() now supports ignore_comments and parser arguments.

3.1.1

3.1

This release offers integration with the great pandas library. Working with qvalues() and filter() functions is now much easier if you have your PSMs in a DataFrame. Many search engines use CSV as their output format, allowing direct creation of DataFrame objects. New functions pyteomics.tandem.DataFrame() and pyteomics.pepxml.DataFrame() faciliatate creation of DataFrames from corresponding formats.

Also, qvalues(), filter() and fdr() functions can now use posterior error probabilities (PEPs) instead of using decoys for q-value calculation.

  • In qvalues() and filter() functions, key and is_decoy can now be array-like objects or strings (as well as functions and iterators). If a string is given, it is used as a field name in the PSM array or DataFrame. fdr() functions also support strings and iterables as arguments.
  • New parameter pep in qvalues(), filter() and fdr() functions. It can be callable, array-like, or iterator. Conflicts with decoy-related parameters. Compatible with key, but makes it optional.
  • Fixed the behavior of filter.chain() functions. They now treat the full_output argument the same way as filter() functions.
  • Fixed the issue that caused exceptions when calling fasta.decoy_db() and fasta.write_decoy_db() with explicitly given mode (signature for creation of pyteomics.auxiliary.FileReader objects slightly changed).
  • Pyteomics now uses setuptools and is a namespace package.
  • Minor fixes.

API changes

  • Default value of remove_decoy in qvalues() is now False.

3.0.1

3.0.0

  • XML parsers are now implemented as objects, each format has its own class. Those classes can be instantiated using the same arguments as read() functions accepted, and support direct iteration and the with syntax. The read() functions are now simple aliases to the corresponding constructors.
  • As a result, functions iterfind(), version_info() and get_by_id() functions are now deprecated in favor of methods iterfind() and get_by_id() and attribute version_info of corresponding instances.
  • In pyteomics.mgf.write(), the order of keys and the format of values are now controlled via module-level variables.
  • In pyteomics.electrochem, correction for pK of terminal groups depending on the terminal residue is implemented; example set of pK and corrected pK added.
  • Imports of external dependencies are delayed where possible, so that unnecessary ImportErrors do not occur.
  • local_fdr() renamed to qvalues() in pepxml, mzid, tandem and auxiliary. local_fdr() did not reflect the semantics of the function. The algorithm has been also corrected so that the array of q-values is always sorted (as it should be by definition).
  • qvalues() now also accepts a parameter full_output which keeps the PSMs alongside their scores and associated q-values.
  • All fdr(), qvalues(), and filter() functions now accept a new parameter correction. It is used for more accurate estimation of the number of false positives using TDA (paper with explanation).
  • filter() functions now support both iterator protocol and context manager protocol. They now also accept the full_output parameter, which has the following meaning: if True (default), then an array of PSMs is directly returned by the function. Otherwise, an iterator is returned, as before. The array takes some memory, but this way is usually around 2x faster.
  • New function pyteomics.pylab_aux.plot_qvalue_curve().
  • pyteomics.mass.Composition objects now have a mass() method (equivalent to pyteomics.mass.calculate_mass().
  • Also, Composition and objects returned by pyteomics.parser.amino_acid_composition() now inherit from collections.defaultdict and collections.Counter.
  • Decoy-related functions in pyteomics.fasta now accept a new parameter keep_nterm that preserves the N-terminal residue in the generated decoy sequences.
  • Minor fixes.

API changes

2.5.5

Fix for a memory leak in pyteomics.mzid.get_by_id(), which affects pyteomics.mzid.read() with retrieve_refs=True.

2.5.4

  • New functions local_fdr() in pepxml, mzid, and tandem. The function returns a NumPy array with PSM scores and corresponding values of local FDR.
  • New parameter iterative in read() functions of XML parsing modules. Parsing of mzIdentML files with retrieve_refs=True got significantly faster.

2.5.3

  • Universally applicable modifications are now allowed in pyteomics.parser.isoforms().
  • It is now also possible to specify non-terminal modifications which are only applicable to terminal residues.
  • Fix in pyteomics.parser.parse(): if the labels argument is provided, it needs to contain standard terminal groups if they are present in the sequence or if show_unmodified_termini is set to True.
  • pyteomics.mass.Composition instances are now pickleable.
  • Performance improvements.

2.5.2

  • New parameter reverse in all filter() functions.
  • New function pyteomics.mass.fast_mass2(), which is analogous to pyteomicsmass.fast_mass(), but supports full modX notation and is several times slower.
  • Fix in pyteomics.pepxml.read() for compatibility with files produced with Mascot2XML utility.
  • Unknown labels now allowed in pyteomics.electrochem and pyteomics.achrom functions in accordance with new general policy.

2.5.1

2.5.0

API changes

  • The boolean overlap parameter in pyteomics.parser.cleave() is replaced with an integer min_length. Since min_length uses pyteomics.parser.length(), the labels keyword argument is now accepted by cleave() and num_sites(), if needed. With carefully designed cleavage rules, all cleavage functions work with modX sequences.
  • The labels argument in pyteomics.parser.parse() and related functions has changed its meaning. parse() won’t raise an exception for non-standard labels in sequences if the labels keyword argument is not given.
  • The modX notation specification is now more strict to avoid ambiguity: only zero or two terminal groups can be present in a modX sequence. Sequences with one terminal group specified will be supported where possible, but be advised that sequences such as “H-OH” are intrinsically ambiguous.

2.4.3

  • Added the ratio keyword argument for FDR calculation.
  • Minor changes in iterfind() functions of file parsers.
  • Bugfix in pyteomics.mgf.write() (duplication of pepmass key).
  • Removed non-functional parameter read_schema for pyteomics.tandem.read().

2.4.2

  • Bugfix in pyteomics.mass.most_probable_isotopic_composition(). The bug manifested itself after version 2.4.0, when pyteomics.mass.nist_mass was expanded. Also, the format of the returned value is now in accordance with the documentation.

2.4.1

  • New function pyteomics.auxiliary.filter() for filtering lists of PSMs not coming directly from files in supported formats.
  • Also, a format-agnostic helper function pyteomics.auxiliary.fdr().

2.4.0

  • New functions for filtering to a certain FDR level based on target-decoy strategy, as well as for FDR estimation, in pyteomics.tandem, pyteomics.pepxml and pyteomics.mzid. The functions are called filter() (beware of shadowing the built-in function) and fdr() (in each of the modules). Chained versions filter.chain() and filter.chain.from_iterable() are also available. See Data Access for more info.

  • New function pyteomics.parser.coverage() for sequence coverage calculation.

  • New function pyteomics.fasta.decoy_chain(), a chained version of pyteomics.fasta.decoy_db().

  • New elements in pyteomics.mass.nist_mass. Pretty much all elements are there now.

  • Fix in pyteomics.parser.parse() to cover some fancy corner cases.

  • Bugfix in pyteomics.tandem: modification info is now fully extracted.

  • pyteomics.mass.isotopic_composition_abundance() is now able to calculate abundances for larger molecules.

    Note

    Rounding errors may be significant in this case.

2.3.0

  • New parameter “read_schema” in read() functions of XML parsing modules. When set to False, disables the attempts to fetch an auxiliary file and obtain structure information about the file being parsed.
  • New function chain() in all modules that have a read() function, for convenient chaining of multiple files. chain() only works as a context manager. Use itertools.chain() in other cases. The chain.from_iterable form is also available as a context manager.
  • New function pyteomics.auxiliary.print_tree() for exploration of complex nested dicts produced by XML parsers.
  • New sets of retention coefficients in pyteomics.achrom.
  • Bugfix in pyteomics.pepxml. The bug caused an exception when parsing some pepXML files.
  • The output of pyteomics.mgf.read() now always contains a masked array of charges.
  • Other minor fixes.

API change

  • In pyteomics.mgf.read() the precursor charge is now always represented by a list of ints (a ChargeList object).

2.2.2

2.2.1

  • Update parsers for FASTA headers.
  • NamedTuple for FASTA entries is now defined globally, which should solve pickling problems.

2.2.0

  • New module pyteomics.tandem for reading output files of X!Tandem search engine.

2.1.6

  • Fix in pyteomics.pepxml. pepXML files generated by TPP are now processed without errors.

2.1.5

  • Fix in pyteomics.pepxml. ‘modified_peptide’ is now always available.
  • Fix in pyteomics.mass (issue #2 in the bug tracker).
  • Improved arithmetics for Composition objects.

2.1.4

  • In fasta, decoy_db() now doesn’t write to file, but returns an iterator over FASTA records. The old decoy_db() is now called write_decoy_db(), which is equivalent to decoy_db() combined with write().

Bugfixes:

  • In pyteomics.mgf.read(), the charges, if present, are returned as a masked array now. Previously, an exception occurred if charges were missing for some of the fragments.
  • Values in mass.nist_mass corrected.
  • Other minor corrections.

2.1.3

  • Adjust the behavior affected by the bug fixed in 2.1.2. name attributes of <cvParam> elements in the absence of value attributes are now collected in a list under the ‘name’ key.
  • Add support for overlapping matches in parser.cleave().

2.1.2

  • Bugfix in XML parsers. The bug caused the mzML parser to break on some files. The fix can slightly change the format of the output.

2.1.1

  • Rename keys in the dicts returned by mgf.read() to facilitate writing code working with both MGF and mzML.
  • The items yielded by fasta.read() now have attributes description and sequence.

2.1.0

  • New sets of retention coefficients in achrom.
  • mass.Composition now only stores non-zero ints.
  • fasta now has tools for parsing of FASTA headers.
  • File parsers now implement the context manager protocol. We recommend using with statements to avoid resource leaks.

API changes

  • ‘pepmass’ is now a tuple in the output of mgf.read() (to allow reading precursor intensities).
  • new function fasta.parse() for convenient parsing of FASTA headers.
  • fasta.std_parsers stores parsers for common UniProt header formats.
  • new parameter parser in fasta.read() allows to apply parsing while reading a FASTA file.
  • close parameter removed in all functions that do file I/O. The unified behavior is: if the parameter is a file object, it won’t be closed by the function. If a file path is given, the file object will be created and closed inside the corresponding function.

2.0.3

  • Added new class pyteomics.mass.Unimod. The interface is experimental and may change.
  • Improved iterfind() function in XML-reading modules.
  • pyteomics.mass.Composition objects now support multiplication by int.
  • Bugfix in auxiliary.linear_regression().

2.0.2

2.0.1

API changes

2.0.0

  • Added mzid module for parsing of mzIdentML files.
  • Fixed bugs, improved tests.

API changes

  • top-module functions in fasta, mgf, mzml, pepxml, as well as mzid, are now called read().
  • in parser, parse_sequence() renamed to parse(). It now accepts an optional parameter allow_unknown_modifications.
  • mgf.write_mgf() and fasta.write_fasta() renamed to write().
  • the output format of all read() functions has changed.

1.2.5

1.2.4

  • Changes in pyteomics.mass.

API changes

  • Composition objects can be created using positional first argument, which will be treated as a sequence or (upon failure) as a formula. This means that all functions relying on Composition (calculate_mass(), most_probable_isotopic_composition(), isotopic_composition_abundance()) allow that as well. However, it’s of no use for the latter.
  • Composition entries for modifications can be added to aa_comp and used in composition and mass calculations. This way the specified group will be added to any residue bearing this modification.
  • That being said, the add_modifications() function is not needed anymore and has been removed.
  • Addition and subtraction of Composition objects now produces a Composition object, allowing addition/subtraction of multiple objects.
  • Composition is now a subclass of collections.defaultdict so one can safely retrieve values without checking if a key exists.

1.2.3

API changes

1.2.2

  • Bugfix in pyteomics.pepxml: modification info is now extracted.
  • New optional boolean argument ‘split’ in pyteomics.parser.parse_sequence() allows to generate a list of tuples where modifications are separated from the residues instead of a regular list of labels. In labels not only modX labels are now allowed, but also separate mod prefixes. Such modifications are assumed to be applicable to any residue.

1.2.1

  • Memory usage significantly decreased when parsing large mzML and pepXML files.

1.2.0

  • Added support for Python 3. Python 2.7 is still supported, Python 2.6 is not.

1.1.1

  • New function called add_modifications() added in pyteomics.mass. It updates aa_comp.
  • Also, pyteomics.parser.isoforms() is a new function to get all possible modified sequences of a peptide.

1.1.0

  • New module added - pyteomics.mgf. It is intended for reading and writing files in Mascot Generic Format.

1.0.2

  • In pyteomics.pepxml module, now all search hits are read from file (not only the top hit).

API changes:

  • pyteomics.pepxml.read(): information specific to search hits is now stored in a list under the 'search_hits' key. The list is sorted by hit rank.

1.0.1

1.0.0

  • The first public release of Pyteomics.

API changes:

«  Example 4: Spectrum Annotation   ::   Contents