Pyteomics documentation v4.1.3dev0

xml - utilities for XML parsing

«  pylab_aux - auxiliary functions for plotting with pylab   ::   Contents

xml - utilities for XML parsing

This module is not intended for end users. It implements the abstract classes for all XML parsers, XML and IndexedXML, and some utility functions.

Dependencies

This module requres lxml and numpy.


class pyteomics.xml.ArrayConversionMixin(*args, **kwargs)[source]

Bases: pyteomics.auxiliary.utils.BinaryDataArrayTransformer

Methods

binary_array_record Hold all of the information about a base64 encoded array needed to decode the array.
decode_data_array(self, source[, …]) Decode a base64-encoded, compressed bytestring into a numerical array.
__init__(self, *args, **kwargs)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

class binary_array_record

Bases: pyteomics.auxiliary.utils.binary_array_record

Hold all of the information about a base64 encoded array needed to decode the array.

Attributes:
compression

Alias for field number 1

data

Alias for field number 0

dtype

Alias for field number 2

key

Alias for field number 4

source

Alias for field number 3

Methods

count()
decode(self) Decode data into a numerical array
index() Raises ValueError if the value is not present.
__init__

x.__init__(…) initializes x; see help(type(x)) for signature

compression

Alias for field number 1

count()
data

Alias for field number 0

decode(self)

Decode data into a numerical array

Returns:
np.ndarray
dtype

Alias for field number 2

index()

Raises ValueError if the value is not present.

key

Alias for field number 4

source

Alias for field number 3

decode_data_array(self, source, compression_type=None, dtype=<type 'numpy.float64'>)

Decode a base64-encoded, compressed bytestring into a numerical array.

Parameters:
source : bytes

A base64 string encoding a potentially compressed numerical array.

compression_type : str, optional

The name of the compression method used before encoding the array into base64.

dtype : type, optional

The data type to use to decode the binary array from the decompressed bytes.

Returns:
np.ndarray
class pyteomics.xml.ByteCountingXMLScanner(source, indexed_tags, block_size=1000000)[source]

Bases: pyteomics.auxiliary.file_helpers._file_obj

Carry out the construction of a byte offset index for source XML file for each type of tag in indexed_tags.

Inheris from pyteomics.auxiliary._file_obj to support the object-oriented _keep_state() interface.

Methods

build_byte_index(self, \*args, \*\*kwargs) Builds a byte offset index for one or more types of tags.
scan  
__init__(self, source, indexed_tags, block_size=1000000)[source]
Parameters:
indexed_tags : iterable of bytes

The XML tags (without namespaces) to build indices for.

block_size : int, optional

The size of the each chunk or “block” of the file to hold in memory as a partitioned string at any given time. Defaults to 1000000.

build_byte_index(self, *args, **kwargs)[source]

Builds a byte offset index for one or more types of tags.

Parameters:
lookup_id_key_mapping : Mapping, optional

A mapping from tag name to the attribute to look up the identity for each entity of that type to be extracted. Defaults to ‘id’ for each type of tag.

Returns:
defaultdict(dict)

Mapping from tag type to dict from identifier to byte offset

class pyteomics.xml.IndexSavingXML(source, read_schema=False, iterative=True, build_id_cache=False, use_index=True, *args, **kwargs)[source]

Bases: pyteomics.auxiliary.file_helpers.IndexSavingMixin, pyteomics.xml.IndexedXML

An extension to the IndexedXML type which adds facilities to read and write the byte offset index externally.

Attributes:
default_index
index

Methods

build_id_cache(self, \*args, \*\*kwargs) Construct a cache for each element in the document, indexed by id attribute
build_tree(self, \*args, \*\*kwargs) Build and store the ElementTree instance for the underlying file
clear_id_cache(self) Clear the element ID cache
clear_tree(self) Remove the saved ElementTree.
get_by_id(self, \*args, \*\*kwargs) Retrieve the requested entity by its id.
iterfind(self, path, \*\*kwargs) Parse the XML and yield info on elements with specified local name or by specified “XPath”.
prebuild_byte_offset_file(cls, path) Construct a new XML reader, build its byte offset index and write it to file
write_byte_offsets(self) Write the byte offsets in _offset_index to the file at _byte_offset_filename
get_by_ids  
get_by_index  
get_by_index_slice  
get_by_indexes  
get_by_key_slice  
next  
reset  
__init__(self, source, read_schema=False, iterative=True, build_id_cache=False, use_index=True, *args, **kwargs)

Create an XML parser object.

Parameters:
source : str or file

File name or file-like object corresponding to an XML file.

read_schema : bool, optional

Defines whether schema file referenced in the file header should be used to extract information about value conversion. Default is False.

iterative : bool, optional

Defines whether an ElementTree object should be constructed and stored on the instance or if iterative parsing should be used instead. Iterative parsing keeps the memory usage low for large XML files. Default is True.

use_index : bool, optional

Defines whether an index of byte offsets needs to be created for elements listed in indexed_tags. This is useful for random access to spectra in mzML or elements of mzIdentML files, or for iterative parsing of mzIdentML with retrieve_refs=True. If True, build_id_cache is ignored. If False, the object acts exactly like XML. Default is True.

indexed_tags : container of bytes, optional

If use_index is True, elements listed in this parameter will be indexed. Empty set by default.

build_id_cache(self, *args, **kwargs)

Construct a cache for each element in the document, indexed by id attribute

build_tree(self, *args, **kwargs)

Build and store the ElementTree instance for the underlying file

clear_id_cache(self)

Clear the element ID cache

clear_tree(self)

Remove the saved ElementTree.

get_by_id(self, *args, **kwargs)

Retrieve the requested entity by its id. If the entity is a spectrum described in the offset index, it will be retrieved by immediately seeking to the starting position of the entry, otherwise falling back to parsing from the start of the file.

Parameters:
elem_id : str

The id value of the entity to retrieve.

id_key : str, optional

The name of the XML attribute to use for lookup. Defaults to self._default_id_attr.

Returns:
dict
iterfind(self, path, **kwargs)

Parse the XML and yield info on elements with specified local name or by specified “XPath”.

Parameters:
path : str

Element name or XPath-like expression. Only local names separated with slashes are accepted. An asterisk (*) means any element. You can specify a single condition in the end, such as: "/path/to/element[some_value>1.5]" Note: you can do much more powerful filtering using plain Python. The path can be absolute or “free”. Please don’t specify namespaces.

**kwargs : passed to self._get_info_smart().
Returns:
out : iterator
classmethod prebuild_byte_offset_file(cls, path)

Construct a new XML reader, build its byte offset index and write it to file

Parameters:
path : str

The path to the file to parse

reset(self)

Resets the iterator to its initial state.

write_byte_offsets(self)

Write the byte offsets in _offset_index to the file at _byte_offset_filename

class pyteomics.xml.IndexedXML(source, read_schema=False, iterative=True, build_id_cache=False, use_index=True, *args, **kwargs)[source]

Bases: pyteomics.auxiliary.file_helpers.IndexedReaderMixin, pyteomics.xml.XML

Subclass of XML which uses an index of byte offsets for some elements for quick random access.

Attributes:
default_index
index

Methods

build_id_cache(self, \*args, \*\*kwargs) Construct a cache for each element in the document, indexed by id attribute
build_tree(self, \*args, \*\*kwargs) Build and store the ElementTree instance for the underlying file
clear_id_cache(self) Clear the element ID cache
clear_tree(self) Remove the saved ElementTree.
get_by_id(self, \*args, \*\*kwargs) Retrieve the requested entity by its id.
iterfind(self, path, \*\*kwargs) Parse the XML and yield info on elements with specified local name or by specified “XPath”.
get_by_ids  
get_by_index  
get_by_index_slice  
get_by_indexes  
get_by_key_slice  
next  
reset  
__init__(self, source, read_schema=False, iterative=True, build_id_cache=False, use_index=True, *args, **kwargs)[source]

Create an XML parser object.

Parameters:
source : str or file

File name or file-like object corresponding to an XML file.

read_schema : bool, optional

Defines whether schema file referenced in the file header should be used to extract information about value conversion. Default is False.

iterative : bool, optional

Defines whether an ElementTree object should be constructed and stored on the instance or if iterative parsing should be used instead. Iterative parsing keeps the memory usage low for large XML files. Default is True.

use_index : bool, optional

Defines whether an index of byte offsets needs to be created for elements listed in indexed_tags. This is useful for random access to spectra in mzML or elements of mzIdentML files, or for iterative parsing of mzIdentML with retrieve_refs=True. If True, build_id_cache is ignored. If False, the object acts exactly like XML. Default is True.

indexed_tags : container of bytes, optional

If use_index is True, elements listed in this parameter will be indexed. Empty set by default.

build_id_cache(self, *args, **kwargs)

Construct a cache for each element in the document, indexed by id attribute

build_tree(self, *args, **kwargs)

Build and store the ElementTree instance for the underlying file

clear_id_cache(self)

Clear the element ID cache

clear_tree(self)

Remove the saved ElementTree.

get_by_id(self, *args, **kwargs)[source]

Retrieve the requested entity by its id. If the entity is a spectrum described in the offset index, it will be retrieved by immediately seeking to the starting position of the entry, otherwise falling back to parsing from the start of the file.

Parameters:
elem_id : str

The id value of the entity to retrieve.

id_key : str, optional

The name of the XML attribute to use for lookup. Defaults to self._default_id_attr.

Returns:
dict
iterfind(self, path, **kwargs)

Parse the XML and yield info on elements with specified local name or by specified “XPath”.

Parameters:
path : str

Element name or XPath-like expression. Only local names separated with slashes are accepted. An asterisk (*) means any element. You can specify a single condition in the end, such as: "/path/to/element[some_value>1.5]" Note: you can do much more powerful filtering using plain Python. The path can be absolute or “free”. Please don’t specify namespaces.

**kwargs : passed to self._get_info_smart().
Returns:
out : iterator
reset(self)

Resets the iterator to its initial state.

class pyteomics.xml.MultiProcessingXML(source, read_schema=False, iterative=True, build_id_cache=False, use_index=True, *args, **kwargs)[source]

Bases: pyteomics.xml.IndexedXML, pyteomics.auxiliary.file_helpers.TaskMappingMixin

XML reader that feeds indexes to external processes for parallel parsing and analysis of XML entries.

Attributes:
default_index
index

Methods

build_id_cache(self, \*args, \*\*kwargs) Construct a cache for each element in the document, indexed by id attribute
build_tree(self, \*args, \*\*kwargs) Build and store the ElementTree instance for the underlying file
clear_id_cache(self) Clear the element ID cache
clear_tree(self) Remove the saved ElementTree.
get_by_id(self, \*args, \*\*kwargs) Retrieve the requested entity by its id.
iterfind(self, path, \*\*kwargs) Parse the XML and yield info on elements with specified local name or by specified “XPath”.
map(self[, target, processes, …]) Execute the target function over entries of this object across up to processes processes.
get_by_ids  
get_by_index  
get_by_index_slice  
get_by_indexes  
get_by_key_slice  
next  
reset  
__init__(self, source, read_schema=False, iterative=True, build_id_cache=False, use_index=True, *args, **kwargs)

Create an XML parser object.

Parameters:
source : str or file

File name or file-like object corresponding to an XML file.

read_schema : bool, optional

Defines whether schema file referenced in the file header should be used to extract information about value conversion. Default is False.

iterative : bool, optional

Defines whether an ElementTree object should be constructed and stored on the instance or if iterative parsing should be used instead. Iterative parsing keeps the memory usage low for large XML files. Default is True.

use_index : bool, optional

Defines whether an index of byte offsets needs to be created for elements listed in indexed_tags. This is useful for random access to spectra in mzML or elements of mzIdentML files, or for iterative parsing of mzIdentML with retrieve_refs=True. If True, build_id_cache is ignored. If False, the object acts exactly like XML. Default is True.

indexed_tags : container of bytes, optional

If use_index is True, elements listed in this parameter will be indexed. Empty set by default.

build_id_cache(self, *args, **kwargs)

Construct a cache for each element in the document, indexed by id attribute

build_tree(self, *args, **kwargs)

Build and store the ElementTree instance for the underlying file

clear_id_cache(self)

Clear the element ID cache

clear_tree(self)

Remove the saved ElementTree.

get_by_id(self, *args, **kwargs)

Retrieve the requested entity by its id. If the entity is a spectrum described in the offset index, it will be retrieved by immediately seeking to the starting position of the entry, otherwise falling back to parsing from the start of the file.

Parameters:
elem_id : str

The id value of the entity to retrieve.

id_key : str, optional

The name of the XML attribute to use for lookup. Defaults to self._default_id_attr.

Returns:
dict
iterfind(self, path, **kwargs)

Parse the XML and yield info on elements with specified local name or by specified “XPath”.

Parameters:
path : str

Element name or XPath-like expression. Only local names separated with slashes are accepted. An asterisk (*) means any element. You can specify a single condition in the end, such as: "/path/to/element[some_value>1.5]" Note: you can do much more powerful filtering using plain Python. The path can be absolute or “free”. Please don’t specify namespaces.

**kwargs : passed to self._get_info_smart().
Returns:
out : iterator
map(self, target=None, processes=-1, queue_timeout=4, args=None, kwargs=None, **_kwargs)

Execute the target function over entries of this object across up to processes processes.

Results will be returned out of order.

Parameters:
target : Callable, optional

The function to execute over each entry. It will be given a single object yielded by the wrapped iterator as well as all of the values in args and kwargs

processes : int, optional

The number of worker processes to use. If negative, the number of processes will match the number of available CPUs.

queue_timeout : float, optional

The number of seconds to block, waiting for a result before checking to see if all workers are done.

args : Sequence, optional

Additional positional arguments to be passed to the target function

kwargs : Mapping, optional

Additional keyword arguments to be passed to the target function

**_kwargs

Additional keyword arguments to be passed to the target function

Yields:
object

The work item returned by the target function.

reset(self)

Resets the iterator to its initial state.

class pyteomics.xml.TagSpecificXMLByteIndex(source, indexed_tags=None, keys=None)[source]

Bases: object

Encapsulates the construction and querying of a byte offset index for a set of XML tags.

This type mimics an immutable Mapping.

Parameters:
index_tags: iterable of bytes

The tag names to include in the index

Attributes:
indexed_tags : iterable of bytes

The tag names to index, not including a namespace

offsets : defaultdict(OrderedDict(str, int))

The hierarchy of byte offsets organized {"tag_type": {"id": byte_offset}}

indexed_tag_keys: dict(str, str)

A mapping from tag name to unique identifier attribute

Methods

build_index(self) Perform the byte offset index building for py:attr:source.
build  
items  
keys  
__init__(self, source, indexed_tags=None, keys=None)[source]

x.__init__(…) initializes x; see help(type(x)) for signature

build_index(self)[source]

Perform the byte offset index building for py:attr:source.

Returns:
offsets: defaultdict

The hierarchical offset, stored in offsets

class pyteomics.xml.XML(source, read_schema=False, iterative=True, build_id_cache=False, **kwargs)[source]

Bases: pyteomics.auxiliary.file_helpers.FileReader

Base class for all format-specific XML parsers. The instances can be used as context managers and as iterators.

Methods

build_id_cache(self, \*args, \*\*kwargs) Construct a cache for each element in the document, indexed by id attribute
build_tree(self, \*args, \*\*kwargs) Build and store the ElementTree instance for the underlying file
clear_id_cache(self) Clear the element ID cache
clear_tree(self) Remove the saved ElementTree.
get_by_id(self, \*args, \*\*kwargs) Parse the file and return the element with id attribute equal to elem_id.
iterfind(self, path, \*\*kwargs) Parse the XML and yield info on elements with specified local name or by specified “XPath”.
next  
reset  
__init__(self, source, read_schema=False, iterative=True, build_id_cache=False, **kwargs)[source]

Create an XML parser object.

Parameters:
source : str or file

File name or file-like object corresponding to an XML file.

read_schema : bool, optional

Defines whether schema file referenced in the file header should be used to extract information about value conversion. Default is False.

iterative : bool, optional

Defines whether an ElementTree object should be constructed and stored on the instance or if iterative parsing should be used instead. Iterative parsing keeps the memory usage low for large XML files. Default is True.

build_id_cache : bool, optional

Defines whether a dictionary mapping IDs to XML tree elements should be built and stored on the instance. It is used in XML.get_by_id(), e.g. when using pyteomics.mzid.MzIdentML with retrieve_refs=True.

huge_tree : bool, optional

This option is passed to the lxml parser and defines whether security checks for XML tree depth and node size should be disabled. Default is False. Enable this option for trusted files to avoid XMLSyntaxError exceptions (e.g. XMLSyntaxError: xmlSAX2Characters: huge text node).

build_id_cache(self, *args, **kwargs)[source]

Construct a cache for each element in the document, indexed by id attribute

build_tree(self, *args, **kwargs)[source]

Build and store the ElementTree instance for the underlying file

clear_id_cache(self)[source]

Clear the element ID cache

clear_tree(self)[source]

Remove the saved ElementTree.

get_by_id(self, *args, **kwargs)[source]

Parse the file and return the element with id attribute equal to elem_id. Returns None if no such element is found.

Parameters:
elem_id : str

The value of the id attribute to match.

Returns:
out : dict or None
iterfind(self, path, **kwargs)[source]

Parse the XML and yield info on elements with specified local name or by specified “XPath”.

Parameters:
path : str

Element name or XPath-like expression. Only local names separated with slashes are accepted. An asterisk (*) means any element. You can specify a single condition in the end, such as: "/path/to/element[some_value>1.5]" Note: you can do much more powerful filtering using plain Python. The path can be absolute or “free”. Please don’t specify namespaces.

**kwargs : passed to self._get_info_smart().
Returns:
out : iterator
reset(self)

Resets the iterator to its initial state.

pyteomics.xml.xpath(tree, path, ns=None)[source]

Return the results of XPath query with added namespaces. Assumes the ns declaration is on the root element or absent.

Parameters:
tree : ElementTree
path : str
ns : str or None, optional
pyteomics.xml.xsd_parser(schema_url)[source]

Parse an XSD file from the specified URL into a schema dictionary that can be used by XML parsers to automatically cast data to the appropriate type.

Parameters:
schema_url : str

The URL to retrieve the schema from

Returns:
dict

«  pylab_aux - auxiliary functions for plotting with pylab   ::   Contents