cdxcore.version#

Framework to track code versions of functions, classes, and their members via a simple decorating mechanism implemented with cdxcore.verson.version().

Overview#

A main application is the use in caching results of computational intensive tasks such as data pipelines in machine learning. The version framework allows updating dynamically only those parts of the data dependency graph whose code generation logic has changed. Correspondingly, cdxcore.subdir.SubDir supports fast light-weight file based read/write support for versioned files.

A full caching logic is implemented using cdxcore.subdir.SubDir.cache().

Versioned Functions#

For versioning, basic use is straight forward and self-explanatory:

from cdxbasics.version import version

@version("0.0.1")
def f(x):
    return x

print( f.version.full )   # -> 0.0.1

Dependencies are declared with the dependencies keyword:

@version("0.0.2", dependencies=[f])
def g(x):
    return f(x)

print( g.version.input )  # -> 0.0.2
print( g.version.full )   # -> 0.0.2 { f: 0.0.01 }

You have access to version from within the function:

@version("0.0.2", dependencies=[f])
def g(x):
    print(g.version.full) # -> 0.0.2 { f: 0.0.01 }
    return f(x)
g(1)

You can also use strings to refer to dependencies. This functionality depends on visibility of the referred dependencies by the function in the function’s __global__ scope:

@version("0.0.4", dependencies=['f'])
def r(x):
    return x

print( r.version.full )    # -> 0.0.4 { f: 0.0.01 }

Versioned Classes#

This works with classes, too:

@version("0.0.3", dependencies=[f] )
class A(object):
    def h(self, x):
        return f(x)

print( A.version.input )  # -> 0.0.3
print( A.version.full )   # -> 0.0.3 { f: 0.0.01 }

a = A()
print( a.version.input )  # -> 0.0.3
print( a.version.full )   # -> 0.0.3 { f: 0.0.01 }

Dependencies on base classes are automatic:

@version("0.0.1")
class A(object):
    pass

@version("0.0.2")
class B(A):
    pass

print( B.version.full )   # -> 0.0.2 { A: 0.0.1 }

Member functions are automatically dependent on their defining class:

from cdxcore.version import version

class A(object):
    def __init__(self, x=2):
        self.x = x
    @version(version="0.4.1")
    def h(self, y):
        return self.x*y

@version(version="0.3.0")
def h(x,y):
    return x+y

@version(version="0.0.2", dependencies=[h])
def f(x,y):
    return h(y,x)

@version(version="0.0.1", dependencies=["f", A.h])
def g(x,z):
    a = A()
    return f(x*2,z)+a.h(z)

g(1,2)
print("version", g.version.input)                # -> version 0.0.1
print("full version", g.version.full )           # -> full version 0.0.1 { f: 0.0.2 { h: 0.3.0 }, A.h: 0.4.1 }
print("full version ID",g.version.unique_id48 )  # -> full version ID 0.0.1 { f: 0.0.2 { h: 0.3.0 }, A.h: 0.4.1 }
print("full version ID",g.version.unique_id32 )  # -> full version ID 0.0.1 { f: 0.0.2 { h: 0.3.0 }, A.h: 0.4.1 }
print("depedencies",g.version.dependencies )     # -> depedencies ('0.0.1', {'f': ('0.0.2', {'h': '0.3.0'}), 'A.h': '0.4.1'})

Decorated Function Information#

A decorated function or class has a member version of type cdxcore.version.Version which has the following key properties:

Import#

from cdxcore.version import version

Documentation#

Functions

version([version, dependencies, auto_class, ...])

Decorator to "version" a function or class, which may depend on other versioned functions or classes.

Classes

Version(original, version, dependencies, ...)

Class to track version dependencies for a given function or class.

Exceptions

VersionDefinitionError(context, message)

Error raised if an error occurred during version definition.

VersionError(*args, version_found, ...)

Standardized error type to be raised by applications if a version found did not match an expected version.
class cdxcore.version.Version(original, version, dependencies, auto_class)[source]#

Bases: object

Class to track version dependencies for a given function or class.

This class is used by cdxcore.subdir.version(). Developers will typically access it via a decorated function’s version property.

Key Properties

Dependency Resolution

Dependency resolution is lazy to allow creating dependencies on Python elements which are defined later / elsewhere. If an error occurs during dependency resoution an exception of type cdxcore.version.VersionDefinitionError is raised.

Parameters:
originalCallable

Orignal Python element: a function, class, or member function.

versionstr, optional

User version string for original.

dependencieslist[type], optional

List of dependencies as types (preferably) or string names.

auto_classbool, optional

Whether to automatically make classes dependent on their (versioned) base classes, and member functions on their (versioned) containing classes.

property dependencies#

Returns information on the version of self and all dependent elements.

For a given function the format is as follows:

  • If the element has no dependents:

    "version"

  • If the function has dependencies, return recursively:

    ( "version", { dependency: dependency.version_full() } )

Example:

from cdxcore.version import version

class A(object):
    def __init__(self, x=2):
        self.x = x
    @version(version="0.4.1")
    def h(self, y):
        return self.x*y

@version(version="0.3.0")
def h(x,y):
    return x+y

@version(version="0.0.2", dependencies=[h])
def f(x,y):
    return h(y,x)

@version(version="0.0.1", dependencies=["f", A.h])
def g(x,z):
    a = A()
    return f(x*2,z)+a.h(z)

g(1,2)
print("version", g.version.input)                # -> version 0.0.1
print("full version", g.version.full )           # -> full version 0.0.1 { f: 0.0.2 { h: 0.3.0 }, A.h: 0.4.1 }
print("depedencies",g.version.dependencies )     # -> depedencies ('0.0.1', {'f': ('0.0.2', {'h': '0.3.0'}), 'A.h': '0.4.1'})
property full: str#

Returns information on the version of self and all dependent elements in human readable form.

Elements are sorted by name, hence this representation can be used to test equality between two versions (cdxcore.version.Version implements == and != based on full).

property input: str#

Returns the input version of this function.

is_dependent(other)[source]#

Determines whether the current element is dependent on another element.

The parameter other can be qualified name, a function, or a class.

Returns:
Versionstr

This function returns None if there is no dependency on other, or the direct user-specified version of the other it is dependent on.

unique_id(max_len=64)[source]#

Returns a unique version string for this version of at most the specified number of characters.

Returns either the simple readable version or the current version plus a unique hash if the simple version exceeds max_len characters.

property unique_id48: str#

Returns a unique version string for this version of at most 48 characters.

Returns either the simple readable version or the current version plus a unique hash if the simple version exceeds 48 characters.

property unique_id60: str#

Returns a unique version string for this version of at most 60 characters.

Returns either the simple readable version or the current version plus a unique hash if the simple version exceeds 60 characters.

property unique_id64: str#

Returns a unique version string for this version of at most 64 characters.

Returns either the simple readable version or the current version plus a unique hash if the simple version exceeds 64 characters.

exception cdxcore.version.VersionDefinitionError(context, message)[source]#

Bases: RuntimeError

Error raised if an error occurred during version definition.

exception cdxcore.version.VersionError(*args, version_found, version_expected)[source]#

Bases: RuntimeError

Standardized error type to be raised by applications if a version found did not match an expected version.

version_expected#

The version expected.

version_found#

The version found.

cdxcore.version.version(version='0.0.1', dependencies=[], *, auto_class=True, raise_if_has_version=True)[source]#

Decorator to “version” a function or class, which may depend on other versioned functions or classes. The point of this decorator is being able to find out the code version of a sequence of function calls, and be able to update cached or otherwise stored results accordingly.

You can cdxcore.version.version() functions, classes, and their member functions.

When a class is versioned it will automatically be dependent on the versions of any versioned base classes. The same is true for versioned member functions: by default they will be dependent on the version of the defining class. Sometimes this behaviour is not helpful. In this case set auto_class to False when defining the cdxcore.version.version() for a member function or derived class.

Simple function example:

from cdxcore.version import version

class A(object):
    def __init__(self, x=2):
        self.x = x
    @version(version="0.4.1")
    def h(self, y):
        return self.x*y

@version(version="0.3.0")
def h(x,y):
    return x+y

@version(version="0.0.2", dependencies=[h])
def f(x,y):
    return h(y,x)

@version(version="0.0.1", dependencies=["f", A.h])
def g(x,z):
    a = A()
    return f(x*2,z)+a.h(z)

g(1,2)
print("version", g.version.input)                # -> version 0.0.1
print("full version", g.version.full )           # -> full version 0.0.1 { f: 0.0.2 { h: 0.3.0 }, A.h: 0.4.1 }
print("full version ID",g.version.unique_id48 )  # -> full version ID 0.0.1 { f: 0.0.2 { h: 0.3.0 }, A.h: 0.4.1 }
print("depedencies",g.version.dependencies )     # -> depedencies ('0.0.1', {'f': ('0.0.2', {'h': '0.3.0'}), 'A.h': '0.4.1'})

Example for classes:

@version("0.1")
class A(object):
    @version("0.2") # automatically depends on A
    def f(self, x):
        return x
    @version("0.3", auto_class=False ) # does not depend on A
    def g(self, x):
        return x

@version("0.4") # automatically depends on A
class B(A):
    pass

@version("0.4", auto_class=False ) # does not depend on A
class C(A):
    pass

b = B()
c = C()
print( "B", b.version.full )   # -> B 0.4 { A: 0.1 }
print( "C", c.version.full )   # -> C 0.4

See Also

cdxcore.subdir.SubDir.cache() implements a caching mechanism which uses versions to decide whether a cached result can still be used.

Parameters:
versionstr

Version string for this function or class.

dependencieslist[type], optional

List of elements this function depends on. Usually the list contains the actual other element by Python reference. If this is not suitable (for example if the name cannot be resolved in order), a string can be used to identify the dependency. If strings are used, then the function’s global context and, if appliable, the associated self will be searched for the respective element.

auto_classbool, optional

If True, the default, then the version of member function or an inherited class is automatically dependent on the version of the defining/base class. Set to False to turn off. The default is True.

raise_if_has_versionbool, optional

Whether to throw an exception of version are already present. This is usually the desired behaviour except if used in another wrapper, see for example cdxcore.subdir.SubDir.cache(). The default is True.

Returns:
WrapperCallable

The returned decorated function or class will have a version property of type cdxcore.version.Version with the following key properties: