pycommons.ds package¶
Some common and shared data structures. A factory for functions checking whether argument values are new. a type variable for the representation-base cache alias of TypeVar(‘T’) Create a function returning True when seeing new values. Creates a function which returns True only the first time it receives a given argument and False all subsequent times. This is based on https://stackoverflow.com/questions/27427067 a function is_new(xx) that will return True the first time it encounters any value xx and False for all values it has already seen Create a cache based on the string representation of an object. In this type of cache is that the repr-representations of objects are used as keys. The first time an object with a given representation is encountered, it is stored in the cache and returned. The next time an object with the same representation is put into this method, the original object with that representation is returned instead. This can be used to still cache and canonically retrieve objects which by themselves are not hashable, like numpy arrays. While the cache itself is not memory-friendly, it can be used to build data structures that re-use the same objects again and again. If these data structures are heavily used, then this can improve the hardware-cache-friendliness of the corresponding code. the cache function TypeError – if the type of a cached object is incompatible with the type of a requested object An immutable version of the the type variable for mapping keys alias of TypeVar(‘K’) the type variable for mapping values alias of TypeVar(‘V’) Create an immutable view of a Mapping. an immutable view on the Mapping a (the view will change if a is changed, but you cannot change a via the view) Tools for working with sequences. the type of the element of the sequences to process alias of TypeVar(‘T’) Merge sorted sequences of integers and return only unique values. You can provide multiple sequences, all of which must be sorted. This function then merges them into a single sorted sequence which contains each elemenet at most once. A typical use case would be to combine the result of Notice that the elements of the sequence must support the less-than operator, i.e., have a __lt__ dunder method. Otherwise this function will crash. The returned sequence is guaranteed to provide strictly increasing values. seqs ( a merged sequence of integers TypeError – if any of the provided iterators or any of their elements is None, or if any of the elements in seqs`is not an :class:`Iterable. Ensure that an This function will solidify an For such single-use objects, a new This form of re-iterabling is useful if you maybe generate items from a slower sequence or do not plan to use all of them. If you want to use all elements several times anyway, it may be more efficient to just wrap the original source into a This is also true if you actually process the generated sequence in some way that may fail or terminate early. Then, first loading all data into a Of course, this can only work if the The function During re-iteration, we need to cache all already visited elements. Thus, without unpacking, we would necessarily have to cache all the elements of the sequence in memory, even if we only need to see them once. However, what we actually do is that we initialize the columns during re-iteration, which may require no pass over the data (no caching), an incomplete pass (thus only caching the actually visited elements), or a complete pass (caching all elements). Then, we unpack the re-iterable for the pass that actually writes the data. The resulting For enabling unpacking, you must set unpackable to True. Notice that wrapping pure instances of the resulting re-iterable iterator TypeError – if source is neither an ValueError – if source is a re-iterable that has been unpacked via Unpack a re-iterable sequence. See the documentation of the unpacked sequence ValueError – if the source is a re-iterable that was created using Submodules¶
pycommons.ds.cache module¶
pycommons.ds.cache.repr_cache().>>> check1: Callable[[str], bool] = is_new()
>>> print(check1("a"))
True
>>> print(check1("a"))
False
>>> print(check1("b"))
True
>>> print(check1("b"))
False
>>> check2: Callable[[int], bool] = is_new()
>>> print(check2(1))
True
>>> print(check2(1))
False
>>> print(check2(3))
True
>>> print(check2(4))
True
>>> print(check2(3))
False
>>> cache1: Callable[[str], str] = repr_cache()
>>> a = f"{1 * 5}"
>>> b = "5"
>>> a is b
False
>>> cache1(a)
'5'
>>> cache1(b) is a
True
>>> cache2: Callable[[float], float] = repr_cache()
>>> x = 5.78
>>> y = 578 / 100
>>> y is x
False
>>> cache2(y)
5.78
>>> cache2(x) is y
True
>>> cache3: Callable[[dict[int, str]], dict[int, str]] = repr_cache()
>>> a = {1: '1', 3: '3', 7: '7'}
>>> b = {1: '1', 3: '3', 7: '7'}
>>> print(cache3(a))
{1: '1', 3: '3', 7: '7'}
>>> print(cache3(a) is a)
True
>>> print(cache3(b) is a)
True
>>> print(cache3(b) is b)
False
>>> class Dummy:
... def __repr__(self):
... return "22222"
>>> cache4: Callable[[Dummy], Dummy] = repr_cache()
>>> _ = cache4(Dummy())
>>> try:
... cache4(22222)
... except TypeError as te:
... s = str(te)
>>> print(s[:34])
Cache yields element of wrong type
>>> "Dummy" in s
True
>>> "int" in s
True
pycommons.ds.immutable_map module¶
typing.Mapping interface.>>> x = {1: 1, 2: 7, 3: 8}
>>> y: Mapping[int, int] = immutable_mapping(x)
>>> x is y
False
>>> x == y
True
>>> x[1] == y[1]
True
>>> x[2] == y[2]
True
>>> x[3] == y[3]
True
>>> z: Mapping[int, int] = immutable_mapping(x)
>>> x is z
False
>>> x == z
True
>>> y is z
False
>>> z: Mapping[int, int] = immutable_mapping(y)
>>> x is z
False
>>> y is z
True
>>> x == z
True
>>> x[9] = 23
>>> y[9] == x[9]
True
>>> try:
... y[1] = 2
... except TypeError as te:
... print(te)
'mappingproxy' object does not support item assignment
>>> try:
... immutable_mapping(5)
... except TypeError as e:
... print(e)
a should be an instance of typing.Mapping but is int, namely 5.
pycommons.ds.sequences module¶
pycommons.math.primes.primes() with some pre-defined values into a sorted sequence.Iterable[TypeVar(T)]) – the sequences, i.e., some instances of Iterable or Iterator>>> list(merge_sorted_and_return_unique([1, 2, 3,], [2, 2]))
[1, 2, 3]
>>> from pycommons.math.primes import primes
>>> list(merge_sorted_and_return_unique(primes(14), [1, 10]))
[1, 2, 3, 5, 7, 10, 11, 13]
>>> list(merge_sorted_and_return_unique(
... primes(14), primes(17), [1, 2, 10, 100]))
[1, 2, 3, 5, 7, 10, 11, 13, 17, 100]
>>> try:
... for _ in merge_sorted_and_return_unique(1):
... pass
... except TypeError as te:
... print(te)
'int' object is not iterable
>>> try:
... for j in merge_sorted_and_return_unique([3], 1):
... print(j)
... except TypeError as te:
... print(te)
'int' object is not iterable
>>> try:
... for j in merge_sorted_and_return_unique([None], [None]):
... print(j)
... except TypeError as te:
... print(te)
Element must not be None.
>>> try:
... for j in merge_sorted_and_return_unique([None], [1]):
... print(j)
... except TypeError as te:
... print(te)
'<' not supported between instances of 'NoneType' and 'int'
>>> try:
... for j in merge_sorted_and_return_unique(None, [1]):
... print(j)
... except TypeError as te:
... print(te)
'NoneType' object is not iterable
>>> try:
... for j in merge_sorted_and_return_unique([print, len], [repr]):
... print(j)
... except TypeError as te:
... print(te)
'<' not supported between instances of 'builtin_function_or_method' and 'builtin_function_or_method'
Iterable can be iterated over multiple times.Iterator into an Iterable. In Python, Iterator is a sub-class of Iterable. This means that if your function accepts instances of Iterable as input, it may expect to be able to iterate over them multiple times. However, if an Iterator is passed in, which also is an instance of Iterable and thus fulfills the function’s type requirement, this is not the case. A typical example of this would be if a Generator is passed in. A Generator is an instance of Iterator, which, in turn, is an instance of Iterable. However, you can iterate over a Generator only once.Iterable wrapper is created. This wrapper will iterate over the original sequence, but cache all elements in an internal list. When you iterate over the sequence again, the elements in the list will be used. This means that all elements of the original sequence will be stored in memory. However, they are only stored if/when they are actually accessed via the iteration sequence. If you do not iterate over them completely, they are not all stored.tuple. But if, for example, your sequence is the result of iterating over a directory tree on the file system, or maybe if it comes from loading a file, then using reiterable() could be useful.tuple may be annoying if your first processed element after that causes a failure or early termination. The bit of overhead of reiterable() may then well be worth your while.Iterator is not otherwise used after calling this function. If you extract elements from the Iterator by yourself otherwise, maybe via next(), then reiterable() cannot work. However, if you only apply next() or other looping paradigms to the Iterable returned by reiterable(), then you can iterate as often as you want over a Generator, for example.unpack_reiterable() can be used to “unpack” the reiterable, returning a single-shot iterator of all the data. The use case is as follows: Our CSV output facility in pycommons.io.csv allows you to turn a sequence of arbitrary data elements to a CSV text stream. This may require two passes over the data: In the first pass, one may determine the columns that are needed. In some cases, we already know the columns and do not need to pass over the data at all. In some other cases, we may need one potentially incomplete pass over the data, just to check if certain columns are needed. For example, when writing out statistical data, we may first want to check if there is any record where the geometric mean is defined. Once we see the first such record, we know that we need to the corresponding column and do not need to check any subsequent elements. Then we can write the CSV header with the column names and start the actual iteration. We thus have one incomplete pass over the data (to find the columns) and one complete pass (to write the data).Iterable will first iterate over the cached elements and then over the remainder in the original Iterator. It thus requires the minimum amount of caching.Iterable into re-iterables has no effect and neither does unpacking. The reason is that if the generation of arbitrarily many iterators is already supported anyway, then we do not need to add the ability to do this via the caching mechanism offered here.Iterable nor an Iterator or if unpackable is not a bool.unpack_reiterable().>>> ri = reiterable(iter(range(4)))
>>> for i in ri:
... print(i)
0
1
2
3
>>> for i in ri:
... print(i)
0
1
2
3
>>> g = (i ** 2 for i in range(5))
>>> r = reiterable(g)
>>> tuple(r)
(0, 1, 4, 9, 16)
>>> tuple(r)
(0, 1, 4, 9, 16)
>>> tuple(r)
(0, 1, 4, 9, 16)
>>> tuple(r)
(0, 1, 4, 9, 16)
>>> tuple(g)
()
>>> g = (i ** 2 for i in range(5))
>>> r = reiterable(g)
>>> i1 = iter(r)
>>> i2 = iter(r)
>>> next(i1)
0
>>> next(i2)
0
>>> next(i2)
1
>>> next(i1)
1
>>> next(i1)
4
>>> next(i1)
9
>>> next(i2)
4
>>> next(i2)
9
>>> i3 = iter(r)
>>> next(i3)
0
>>> next(i3)
1
>>> next(i3)
4
>>> next(i3)
9
>>> next(i3)
16
>>> next(i2)
16
>>> try:
... next(i2)
... except StopIteration as si:
... print(type(si))
<class 'StopIteration'>
>>> try:
... next(i3)
... except StopIteration as si:
... print(type(si))
<class 'StopIteration'>
>>> next(i1)
16
>>> try:
... next(i1)
... except StopIteration as si:
... print(type(si))
<class 'StopIteration'>
>>> a = [1, 2, 3]
>>> reiterable(a) is a
True
>>> a = (1, 2, 3)
>>> reiterable(a) is a
True
>>> a = {1, 2, 3}
>>> reiterable(a) is a
True
>>> a = {1: 1, 2: 2, 3: 3}
>>> reiterable(a) is a
True
>>> k = a.keys()
>>> reiterable(k) is k
True
>>> k = a.values()
>>> reiterable(k) is k
True
>>> tuple(reiterable((x for x in range(5))))
(0, 1, 2, 3, 4)
>>> try:
... reiterable(None)
... except TypeError as te:
... print(str(te)[:60])
source should be an instance of any in {typing.Iterable, typ
>>> try:
... reiterable(1)
... except TypeError as te:
... print(str(te)[:60])
source should be an instance of any in {typing.Iterable, typ
>>> try:
... reiterable((2, 3), "x")
... except TypeError as te:
... print(str(te)[:59])
unpackable should be an instance of bool but is str, namely
>>> type(merge_sorted_and_return_unique([1, 2, 3,], [2, 2]))
<class 'generator'>
>>> type(reiterable(merge_sorted_and_return_unique([1, 2, 3,], [2, 2])))
<class 'pycommons.ds.sequences.__Reiterator'>
>>> def myfunc():
... if hasattr(myfunc, "a"):
... s = getattr(myfunc, "a")
... else:
... s = 0
... for i in range(s, s + 3):
... yield i
... setattr(myfunc, "a", s + 3)
>>> ri = reiterable(myfunc())
>>> tuple(ri)
(0, 1, 2)
>>> tuple(ri)
(0, 1, 2)
>>> ri = reiterable(myfunc())
>>> tuple(ri)
(3, 4, 5)
>>> tuple(ri)
(3, 4, 5)
reiterable() for more details.reiterable() with unpackable set to False or if the re-iterable has already been unpacked.>>> x = reiterable(iter(range(5)), unpackable=True)
>>> y = iter(x)
>>> next(y)
0
>>> next(y)
1
>>> z = unpack_reiterable(x)
>>> for i in z:
... print(i)
0
1
2
3
4
>>> try:
... unpack_reiterable(x)
... except ValueError as ve:
... print(ve)
Re-iterable has already been unpacked!
>>> try:
... reiterable(x, False)
... except ValueError as ve:
... print(ve)
Unsafe operation: The source is a re-iterable that has been unpacked.
>>> try:
... unpack_reiterable(reiterable(iter(range(5)), False))
... except ValueError as ve:
... print(ve)
This re-iterable cannot be unpacked.