cdxcore.npshm

Shared memory numpy arrays.

Overview

The functions in this module wrap multiprocessing.shared_memory.SharedMemory into a numpy array with garbage collection clean up, depending in the operating system.

In process 1:

from cdxcore.npshm import create_shared_array
import numpy as np

test_name = "Test 2121"    
test = create_shared_array( test_name, shape=(10,3), dtype=np.int32, force=True, full=0 )
test[:,1] = 1

In process 2:

from cdxcore.npshm import create_shared_array
import numpy as np

test_name = "Test 2121"    
test = attach_shared_array( test_name, validate_shape=(10,3), validate_dtype=np.int32 )
assert np.all( test[:,1] == 1)
test[:,2] = 2

Back in process 1:

assert np.all( test[:,2] == 2)

Loading binary files

This module’s’ cdxcore.npshm.read_shared_array() reads numpy arrays stored to disk with cdxcore.npio.to_file() directly into shared memory.

Garbage Collection

The functions here are simplistic wrappers aroud multiprocessing.shared_memory.SharedMemory on Linux and Windows. The returned object will call multiprocessing.shared_memory.SharedMemory.close() upon garbage collection, but not multiprocessing.shared_memory.SharedMemory.unlink().

Linux

Under Linux above setings means that the shared file will reside permanently – and will remain sharable – in /dev/shm/ until it is manually deleted. Call cdxcore.npshm.delete_shared_array() to delete a shared file manually.

The amount of shared memory available on Linux is limited by default. Use `findmnt -o AVAIL,USED /dev/shm to check available size. Modify /etc/fstab to amend.

Windows

Windows keeps track of access to shared memory and will release it automatically upon garbage collection of the last Python object, or upon destruction of all processes with access to the shared memory block. Therefore the object does not persist between independent runs of your software.

Import

from cdxcore.npio import create_shared_array, attach_shared_array, read_shared_array

Documentation

Functions

attach_shared_array(name, *[, ...])	Attach a numpy.ndarray to an existing named shared array.
create_shared_array(name, shape, dtype, *[, ...])	Create a new named shared array.
delete_shared_array(name[, raise_on_error])	Deletes the shared array associated with name by calling multiprocessing.shared_memory.SharedMemory.unlink().
is_shared_array(x)	Whether an array is "shared"
read_shared_array(file, name, *[, ...])	Read a shared array from disk into a new named shared numpy.ndarray in binary format using cdxcore.npio.read_into().

cdxcore.npshm.attach_shared_array(name, *, validate_shape=None, validate_dtype=None, raise_on_error=True, read_only=False)

Attach a numpy.ndarray to an existing named shared array.

This function is a simplistic wrapper aroud creating a numpy.ndarray with an existing multiprocessing.shared_memory.SharedMemory buffer. The returned object will call multiprocessing.shared_memory.SharedMemory.close() upon garbage collection, but not multiprocessing.shared_memory.SharedMemory.unlink().

Linux

Under Linux above setings means that the shared file will reside permanently – and will remain sharable – in /dev/shm/ until it is manually deleted. Call cdxcore.npshm.delete_shared_array() to delete a shared file manually.

The amount of shared memory available on Linux is limited by default. Use `findmnt -o AVAIL,USED /dev/shm to check available size. Modify /etc/fstab to amend.

Windows

Windows keeps track of access to shared memory and will release it automatically upon garbage collection of the last Python object, or upon destruction of all processes with access to the shared memory block. Therefore the object does not persist between independent runs of your software.

Parameters:

namestr

Name of the array. This must be a valid file name. In Linux, shared memory is managed via /dev/shm/.

validate_shapetuple|None, default None

Validate that array has this shape, if not None. If the array has a different shape, raise a ValueError.

validate_dtypedtype|None, default None

Validate that array has this dtype, if not None. If the array has a different dtype, raise a ValueError.

raise_on_errorbool, default True

If an array called name does not exists, this function raises an FileNotFoundError exception if raise_on_error is True; otherwise it will return None.

read_onlyboo, default False

Whether to set numpy’s writeable flag to False.

Returns:

Arraynumpy.ndarray like

The array, or None if no array of name exists and raise_on_error is False.

Raises:

File not foundFileNotFoundError

If an array name does not exist (and raise_on_error is True).

Incorrect geometryValueError

Raised if validate_shape or validate_dtype do not match the array (and raise_on_error is True).

cdxcore.npshm.create_shared_array(name, shape, dtype, *, raise_on_error=True, force=False, full=None)

Create a new named shared array.

This function can force creation of a new array on Linux only.

This function is a simplistic wrapper aroud creating a numpy.ndarray with a newly created multiprocessing.shared_memory.SharedMemory buffer. The returned object will call multiprocessing.shared_memory.SharedMemory.close() upon garbage collection, but not multiprocessing.shared_memory.SharedMemory.unlink().

Linux

Under Linux above setings means that the shared file will reside permanently – and will remain sharable – in /dev/shm/ until it is manually deleted. Call cdxcore.npshm.delete_shared_array() to delete a shared file manually.

The amount of shared memory available on Linux is limited by default. Use `findmnt -o AVAIL,USED /dev/shm to check available size. Modify /etc/fstab to amend.

Windows

Windows keeps track of access to shared memory and will release it automatically upon garbage collection of the last Python object, or upon destruction of all processes with access to the shared memory block. Therefore the object does not persist between independent runs of your software.

Parameters:

namestr

Name of the array. This must be a valid file name. In Linux, shared memory is managed via /dev/shm/.

shapetuple

Shape of the array.

dtypedtype

Numpy dtype of the array.

raise_on_errorbool, default True

If an array called name already exists, this function raises an FileExistsError exception if raise_on_error is True; otherwise it will return None.

fulldefault None

Value to fill array with, or None to not fill the array.

forcebool, default False

Whether to attempt to delete any existing arrays under Linux only. Note that while the file might be get deleted the actual memory is only freed after all references are destroyed.

Returns:

Arraynumpy.ndarray like

Shared numpy array, or None if the named array exists and if raise_on_error is False.

Raises:

File existsFileExistsError

If an array name already exists (and raise_on_error is True).

cdxcore.npshm.delete_shared_array(name, raise_on_error=True)

Deletes the shared array associated with name by calling multiprocessing.shared_memory.SharedMemory.unlink().

Linux

Under Linux, calling unlink() will prevent further attachments to this file, and allows creating a new file in its place. Existing shares remain operational. Note that the file is deleted immediately, not once the last reference was deleted.

Windows

This function does nothing under Windows.

Parameters:

namestr

Name of the array. This must be a valid file name. In Linux, shared memory is managed via /dev/shm/.

raise_on_errorbool, default True

If the file could not be deleted successfully, raise the respective Exception. If the file did not exist, this function will return successfully.

Returns:

Successbool

Whether name can now be used for a new shared memory block.

cdxcore.npshm.is_shared_array(x)

Whether an array is “shared”

cdxcore.npshm.read_shared_array(file, name, *, validate_shape=None, validate_dtype=None, accept_existing=True, buffering=-1, read_only=False, return_status=False)

Read a shared array from disk into a new named shared numpy.ndarray in binary format using cdxcore.npio.read_into().

If accept_existing is True, this function will first attempt to attach to an existing shared array name.

Parameters:

filestr | int

File from open() or file name.

namestr

Name of the array. This must be a valid file name. In Linux, shared memory is managed via /dev/shm/.

validate_shapetuple|None, default None

Validate that array has this shape, if not None. If the array has a different shape, raise a ValueError.

validate_dtypedtype|None, default None

Validate that array has this dtype, if not None. If the array has a different dtype, raise a ValueError.

raise_on_errorbool, default True

If a shared array called name does not exists, this function raises an FileNotFoundError exception if raise_on_error is True; otherwise it will return None.

read_onlyboo, default False

Whether to set numpy’s writeable flag to False.

accept_existingbool, default True

Whether to first try to attach to an existing shared array name. If either validate_shape or validate_dtype is None, then the function will read the array characteristics from the file on disk even if an exsting array exists to ensure its characteristics match those on disk.

bufferingint, default -1

See open(). Use -1 for default behaviour.

return_statusbool default False

Whether to return status as well. See below.

Returns:

Arraynumpy.ndarray like

If return_status is False, return just the array or None if an error occured and raise_on_error is False.

( Array, attached )numpy.ndarray like, bool

If return_status is True, and if no error occured, then the function returns a tuple containing the array and a boolean indicating whether the array was attached to an existing shared array (True), or whether a new shared array was created (False). Useful for status messages.

Raises:

File existsFileExistsError

If an array name already exists, and accept_existing was False (and raise_on_error is True).

Incorrect geometryValueError

Raised if validate_shape or validate_dtype do not match the array (and raise_on_error is True).