cdxcore.version#

Framework to track code versions of functions, classes, and their members via a simple decorating mechanism implemented with cdxcore.verson.version().

Overview#

A main application is the use in caching results of computational intensive tasks such as data pipelines in machine learning. The version framework allows updating dynamically only those parts of the data dependency graph whose code generation logic has changed. Correspondingly, cdxcore.subdir.SubDir supports fast light-weight file based read/write support for versioned files.

A full caching logic is implemented using cdxcore.subdir.SubDir.cache().

Versioned Functions#

For versioning, basic use is straight forward and self-explanatory:

from cdxbasics.version import version

@version("0.0.1")
def f(x):
    return x

print( f.version.full )   # -> 0.0.1

Dependencies are declared with the dependencies keyword:

@version("0.0.2", dependencies=[f])
def g(x):
    return f(x)

print( g.version.input )  # -> 0.0.2
print( g.version.full )   # -> 0.0.2 { f: 0.0.01 }

You have access to version from within the function:

@version("0.0.2", dependencies=[f])
def g(x):
    print(g.version.full) # -> 0.0.2 { f: 0.0.01 }
    return f(x)
g(1)

You can also use strings to refer to dependencies. This functionality depends on visibility of the referred dependencies by the function in the function’s __global__ scope:

@version("0.0.4", dependencies=['f'])
def r(x):
    return x

print( r.version.full )    # -> 0.0.4 { f: 0.0.01 }

Versioned Classes#

This works with classes, too:

@version("0.0.3", dependencies=[f] )
class A(object):
    def h(self, x):
        return f(x)

print( A.version.input )  # -> 0.0.3
print( A.version.full )   # -> 0.0.3 { f: 0.0.01 }

a = A()
print( a.version.input )  # -> 0.0.3
print( a.version.full )   # -> 0.0.3 { f: 0.0.01 }

Dependencies on base classes are automatic:

@version("0.0.1")
class A(object):
    pass

@version("0.0.2")
class B(A):
    pass

print( B.version.full )   # -> 0.0.2 { A: 0.0.1 }

Member functions are automatically dependent on their defining class:

from cdxcore.version import version

class A(object):
    def __init__(self, x=2):
        self.x = x
    @version(version="0.4.1")
    def h(self, y):
        return self.x*y

@version(version="0.3.0")
def h(x,y):
    return x+y

@version(version="0.0.2", dependencies=[h])
def f(x,y):
    return h(y,x)

@version(version="0.0.1", dependencies=["f", A.h])
def g(x,z):
    a = A()
    return f(x*2,z)+a.h(z)

g(1,2)
print("version", g.version.input)                # -> version 0.0.1
print("full version", g.version.full )           # -> full version 0.0.1 { f: 0.0.2 { h: 0.3.0 }, A.h: 0.4.1 }
print("full version ID",g.version.unique_id48 )  # -> full version ID 0.0.1 { f: 0.0.2 { h: 0.3.0 }, A.h: 0.4.1 }
print("full version ID",g.version.unique_id32 )  # -> full version ID 0.0.1 { f: 0.0.2 { h: 0.3.0 }, A.h: 0.4.1 }
print("depedencies",g.version.dependencies )     # -> depedencies ('0.0.1', {'f': ('0.0.2', {'h': '0.3.0'}), 'A.h': '0.4.1'})

Decorated Function Information#

A decorated function or class has a member version of type cdxcore.version.Version which has the following key properties:

cdxcore.version.Version.input: the input version as defined with cdxcore.version.version().
cdxcore.version.Version.full: a fully qualified version with all dependent functions and classes in human readable form.
cdxcore.version.Version.unique_id48, cdxcore.version.Version.unique_id64: unique hashes of cdxcore.version.Version.full of 48 or 64 characters, respectively. You can use the function cdxcore.version.Version.unique_id() to compute hash IDs of any length.
cdxcore.version.Version.dependencies: a hierarchical list of dependencies for systematic inspection.

Import#

from cdxcore.version import version

Documentation#

Functions

version([version, dependencies, auto_class, ...])

Decorator to "version" a function or class, which may depend on other versioned functions or classes.

Classes

Version(original, version, dependencies, ...)

Class to track version dependencies for a given function or class.

Exceptions

`VersionDefinitionError`(context, message)	Error rasied if an error occured during version definition.
`VersionError`(*args, version_found, ...)	Standardized error type to be raised by applications if a version found did not match an expected version.

class cdxcore.version.Version(original, version, dependencies, auto_class)[source]#

Bases: object

Class to track version dependencies for a given function or class.

This class is used by cdxcore.subdir.version(). Developers will typically access it via a decorated function’s version property.

Key Properties

cdxcore.version.Version.input: input version string as provided by the user.
cdxcore.version.Version.full: qualified full version including versions of dependent functions or classes, as a string.
cdxcore.version.Version.unique_id48: 48 character unique ID. Versions for 60 and 64 characters are also pre-defined.
cdxcore.version.Version.dependencies: hierarchy of version dependencies as a list.

Dependency Resolution

Dependency resolution is lazy to allow creating dependencies on Python elements which are defined later / elsewhere. If an error occurs during dependency resoution an exception of type cdxcore.version.VersionDefinitionError is raised.

Parameters:

originalCallable: Orignal Python element: a function, class, or member function.
versionstr, optional: User version string for original.
dependencieslist[type], optional: List of dependencies as types (preferably) or string names.
auto_classbool, optional: Whether to automatically make classes dependent on their (versioned) base classes, and member functions on their (versioned) containing classes.

Attributes:

dependencies: Returns information on the version of self and all dependent elements.
full: Returns information on the version of self and all dependent elements in human readable form.
input: Returns the input version of this function.
unique_id48: Returns a unique version string for this version of at most 48 characters.
unique_id60: Returns a unique version string for this version of at most 60 characters.
unique_id64: Returns a unique version string for this version of at most 64 characters.

Methods

`is_dependent`(other)	Determines whether the current element is dependent on another element.
`unique_id`([max_len])	Returns a unique version string for this version of at most the specified number of characters.

property dependencies#

Returns information on the version of self and all dependent elements.

For a given function the format is as follows:

If the element has no dependents:
```
"version"
```

If the function has dependencies, return recursively:

( "version", { dependency: dependency.version_full() } )

Example:

from cdxcore.version import version

class A(object):
    def __init__(self, x=2):
        self.x = x
    @version(version="0.4.1")
    def h(self, y):
        return self.x*y

@version(version="0.3.0")
def h(x,y):
    return x+y

@version(version="0.0.2", dependencies=[h])
def f(x,y):
    return h(y,x)

@version(version="0.0.1", dependencies=["f", A.h])
def g(x,z):
    a = A()
    return f(x*2,z)+a.h(z)

g(1,2)
print("version", g.version.input)                # -> version 0.0.1
print("full version", g.version.full )           # -> full version 0.0.1 { f: 0.0.2 { h: 0.3.0 }, A.h: 0.4.1 }
print("depedencies",g.version.dependencies )     # -> depedencies ('0.0.1', {'f': ('0.0.2', {'h': '0.3.0'}), 'A.h': '0.4.1'})

property full: str#

Returns information on the version of self and all dependent elements in human readable form.

Elements are sorted by name, hence this representation can be used to test equality between two versions (cdxcore.version.Version implements == and != based on full).

property input: str#: Returns the input version of this function.

is_dependent(other)[source]#

Determines whether the current element is dependent on another element.

The parameter other can be qualified name, a function, or a class.

Returns:

Versionstr: This function returns None if there is no dependency on other, or the direct user-specified version of the other it is dependent on.

unique_id(max_len=64)[source]#

Returns a unique version string for this version of at most the specified number of characters.

Returns either the simple readable version or the current version plus a unique hash if the simple version exceeds max_len characters.

property unique_id48: str#

Returns a unique version string for this version of at most 48 characters.

Returns either the simple readable version or the current version plus a unique hash if the simple version exceeds 48 characters.

property unique_id60: str#

Returns a unique version string for this version of at most 60 characters.

Returns either the simple readable version or the current version plus a unique hash if the simple version exceeds 60 characters.

property unique_id64: str#

Returns a unique version string for this version of at most 64 characters.

Returns either the simple readable version or the current version plus a unique hash if the simple version exceeds 64 characters.

exception cdxcore.version.VersionDefinitionError(context, message)[source]#

Bases: RuntimeError

Error rasied if an error occured during version definition.

exception cdxcore.version.VersionError(*args, version_found, version_expected)[source]#

Bases: RuntimeError

Standardized error type to be raised by applications if a version found did not match an expected version.

version_expected#: The version expected.

version_found#: The version found.

cdxcore.version.version(version='0.0.1', dependencies=[], *, auto_class=True, raise_if_has_version=True)[source]#

Decorator to “version” a function or class, which may depend on other versioned functions or classes. The point of this decorator is being able to find out the code version of a sequence of function calls, and be able to update cached or otherwise stored results accordingly.

You can cdxcore.version.version() functions, classes, and their member functions.

When a class is versioned it will automatically be dependent on the versions of any versioned base classes. The same is true for versioned member functions: by default they will be dependent on the version of the defining class. Sometimes this behaviour is not helpful. In this case set auto_class to False when defining the cdxcore.version.version() for a member function or derived class.

Simple function example:

from cdxcore.version import version

class A(object):
    def __init__(self, x=2):
        self.x = x
    @version(version="0.4.1")
    def h(self, y):
        return self.x*y

@version(version="0.3.0")
def h(x,y):
    return x+y

@version(version="0.0.2", dependencies=[h])
def f(x,y):
    return h(y,x)

@version(version="0.0.1", dependencies=["f", A.h])
def g(x,z):
    a = A()
    return f(x*2,z)+a.h(z)

g(1,2)
print("version", g.version.input)                # -> version 0.0.1
print("full version", g.version.full )           # -> full version 0.0.1 { f: 0.0.2 { h: 0.3.0 }, A.h: 0.4.1 }
print("full version ID",g.version.unique_id48 )  # -> full version ID 0.0.1 { f: 0.0.2 { h: 0.3.0 }, A.h: 0.4.1 }
print("depedencies",g.version.dependencies )     # -> depedencies ('0.0.1', {'f': ('0.0.2', {'h': '0.3.0'}), 'A.h': '0.4.1'})    

Example for classes:

@version("0.1")
class A(object):
    @version("0.2") # automatically depends on A
    def f(self, x):
        return x
    @version("0.3", auto_class=False ) # does not depend on A
    def g(self, x):
        return x

@version("0.4") # automatically depends on A
class B(A):
    pass

@version("0.4", auto_class=False ) # does not depend on A
class C(A):
    pass

b = B()
c = C()
print( "B", b.version.full )   # -> B 0.4 { A: 0.1 }
print( "C", c.version.full )   # -> C 0.4

See Also

cdxcore.subdir.SubDir.cache() implements a caching mechanism which uses versions to decide whether a cached result can still be used.