cdxcore.util#
Module Attributes
Default map from characters which cannot be used for filenames under either Windows or Linux to valid characters. |
Functions
Inspect a |
|
|
Return a formatted big byte string, e.g. 12.35MB. |
|
Return a formatted big number string, e.g. 12.35M instead of all digits. |
|
Returns string representation for a date of the form "YYYY-MM-DD". |
|
Convert |
|
Return a readable representation of a dictionary. |
|
String representation of an integer with 1000 separators: 10000 becomes "10,000". |
|
Replaces invalid filename characters such as |
|
Returns a formatted string of a list, its elements separated by commas and (by default) a final 'and'. |
|
Returns the |
|
Generate format string for seconds, e.g. "23s"" for |
|
Convers a time to a string with format "HH:MM:SS". |
|
Returns string representation for a time delta in the form "DD:HH:MM:SS,MS". |
|
When called from a function, returns the name of the calling function. |
|
Approximates the size of an object. |
|
Whether an element is atomic. |
|
Tests whether a filename is indeed a valid filename. |
|
Checks whether a type is a |
|
Checks whether |
|
Whether we operate in a jupter session. |
|
Converts a python structure into a simple atomic/list/dictionary collection such that it can be read without the specific imports used inside this program. |
|
Return qualified name including module name of some Python element. |
Returns a set of all |
Classes
|
Format as a string or callable. |
|
Carriage Return ("\r") manager. |
|
Simple object that counts the number of times |
|
Micro utility to measure passage of time. |
|
Track execution time and sub-topic timings. |
- class cdxcore.util.AcvtiveFormat(fmt, label='Format string', name=None, reserved_keywords=None, strict=False)[source]#
Bases:
objectFormat as a string or callable.
This class allows a user to specify a format string either by a Python
str.format()string or aCallable.Example:
from cdxcore.util import AcvtiveFormat fmt = AcvtiveFormat("{x:.2f}", "test format" ) print( fmt(x=1) ) fmt = AcvtiveFormat(lambda x : "{x:.2f}", "test format" ) print( fmt(x=1) )
The advantage of using the
lambda x : {x:.2f}method is that it allows fairly complex formatting and data expressions at the formatting stage.- Parameters:
- fmtstr | Callable
Either a Python
str.format()string containing{}for formatting, or a callable which returns a string.- labelstr, default
Format string A descriptive string for error messages referring the format string, typically in the format
f{which} '{fmt}' cannot have positional arguments...- namestr | None, default
None A name for the formatting string. If not provided, the name will be auto-generated: If
fmtis a string, this string will be used; iffmtis a callable thencdxcore.util.qualified_name()is used.- reserved_keywordsdict | None, default
None Mechanism for defining default keywords which are provided by the environment, not the user. For example:
from cdxcore.util import AcvtiveFormat fmt = AcvtiveFormat("{name} {x:.2f}", "test format", reserved_keywords=dict(name="test") ) print( fmt(x=1) )
- strictbool, default
False If
Falsethis function does not validate that all arguments passed tocdxcore.util.AcvtiveFormat.__call__()have to be understood by the formatting function. This is usally the best solution as the calling entity just passes everything and the formatter selects what it needs.Set to
Trueto validate that the passed arguments match exactly the expected arguments.
- labelstr, default
- __call__(**arguments)[source]#
Execute the format string.
- Parameters:
- argumentsMapping
All arguments to be passed to the format string or function.
If this object was constructed with
strict=Truethen the list of arguments must matchcdxcore.util.AcvtiveFormat.required_argumentsexcept forcdxcore.util.AcvtiveFormat.reserved_keywords.
- Returns:
- textstr
Formatted string.
- class cdxcore.util.CRMan[source]#
Bases:
objectCarriage Return (”\r”) manager.
This class is meant to enable efficient per-line updates using “\r” for text output with a focus on making it work with both Jupyter and the command shell. In particular, Jupyter does not support the ANSI \33[2K ‘clear line’ code. To simulate clearing lines,
CRMankeeps track of the length of the current line, and clears it by appending spaces to a message following “\r” accordingly.This functionality does not quite work accross all terminal types which were tested. Main focus is to make it work for Jupyer for now. Any feedback on how to make this more generically operational is welcome.
crman = CRMan() print( crman("\rmessage 111111"), end='' ) print( crman("\rmessage 2222"), end='' ) print( crman("\rmessage 33"), end='' ) print( crman("\rmessage 1\n"), end='' )
prints:
message 1
While
print( crman("\rmessage 111111"), end='' ) print( crman("\rmessage 2222"), end='' ) print( crman("\rmessage 33"), end='' ) print( crman("\rmessage 1"), end='' ) print( crman("... and more.") )
prints
message 1... and more
- __call__(message)[source]#
Convert message containing “\r” and “\n” into a printable string which ensures that a “\r” string does not lead to printed artifacts. Afterwards, the object will retain any text not terminated by “\n”.
- Parameters:
- messagestr
message containing “\r” and “\n”.
- Returns:
- Message: str
Printable string.
- property current: str#
Return current string.
This is the string that
CRManis currently visible to the user since the last time a new line was printed.
- cdxcore.util.DEF_FILE_NAME_MAP = {'*': '@', '/': '_', ':': ';', '<': '(', '>': ')', '?': '!', '\\': '_', '|': '_'}#
Default map from characters which cannot be used for filenames under either Windows or Linux to valid characters.
- class cdxcore.util.DebugTime(init=0.0)[source]#
Bases:
objectSimple object that counts the number of times
__call__was called. Used for writing tests forcdxcore.util.TrackTime
- class cdxcore.util.Timer[source]#
Bases:
objectMicro utility to measure passage of time.
Example:
from cdxcore.util import Timer with Timer() as t: .... do somthing ... print(f"This took {t}.")
- property fmt_seconds#
Seconds elapsed since construction or
cdxcore.util.Timer.reset(), formatted usingcdxcore.util.fmt_seconds()
- property hours: float#
Hours passed since construction or
cdxcore.util.Timer.reset()
- interval_test(interval)[source]#
Tests if
intervalseconds have passed. If yes, reset timer and return True. Otherwise return False.Usage:
from cdxcore.util import Timer tme = Timer() for i in range(n): if tme.test_dt_seconds(2.): print(f"\\r{i+1}/{n} done. Time taken so far {tme}.", end='', flush=True) print("\\rDone. This took {tme}.")
- property minutes: float#
Minutes passed since construction or
cdxcore.util.Timer.reset()
- property seconds: float#
Seconds elapsed since construction or
cdxcore.util.Timer.reset()
- class cdxcore.util.TrackTime(topic=None, start=None, *, elapsed_seconds=None)[source]#
Bases:
objectTrack execution time and sub-topic timings.
The timer keeps a running total of elapsed seconds and can record laps. Each lap updates the timer’s internal reference time (so subsequent laps measure time since the previous
cdxcore.util.TrackTime.lap_time()call).You can also record lap time under a named sub topic. Sub topics are stored as a dictionary of child
TrackTimeinstances and can be accessed viatimer["topic"].Deterministic testing is supported by overriding the class attribute
cdxcore.util.TrackTime.TIMERwith a callable that returns the “current” time in seconds. In normal usage, leave it at its default (time.time()).Examples
Basic usage with laps:
from cdxcore.util import TrackTime import time t = TrackTime("work") time.sleep(0.1) t.lap_time() time.sleep(0.1) t.lap_time() print(t.seconds) # -> around 0.2
Track two alternating sub topics:
t = TrackTime("work") for _ in range(10): # ... do task 1 ... t.lap_time("t1") # ... do task 2 ... t.lap_time("t2") print(t["t1"].average_lap_seconds) print(t["t2"].average_lap_seconds)
Time a block using a sub topic as a context manager:
with t("io"): # ... do IO ... pass
- Parameters:
- topicstr | TrackTime | None, default
None Name of the timer topic. If
Nonethis function will attempt to determine the name of the calling function and use it as topic name. If aTrackTimeobject is passed, a deep copy will be made.- startbool | None, default
True Whether to start the timer immediately. If
False, the timer will remain stopped untilcdxcore.util.TrackTime.start()is called. UseNoneto keep the same start state as the passedTrackTimeobject. IfNoneand noTrackTimeobject is passed, the timer will start immediately.- elapsed_secondsfloat | None, default
None If not
None, this is the initial number of seconds to start with. This can be used to create a timer which starts with some pre-recorded time. If notNone, this sets the internal lap count to 1, otherwise it is set to 0.
- topicstr | TrackTime | None, default
Notes
cdxcore.util.TrackTime.secondsis the total elapsed seconds for this timer up to current time, if the time is running. Callcdxcore.util.TrackTime.stop()to stop it.cdxcore.util.TrackTime.countis the number of recorded laps/updates for this timer.
- TIMER(/)#
Callable used to compute current time in seconds.
Defaults to
time.time(). For deterministic tests, assign a custom callable tocdxcore.util.TrackTime.TIMERsuch ascdxcore.util.DebugTime.
- __call__(sub_topic, start=True)[source]#
Get or create a timed sub topic.
The returned object is a
cdxcore.util.TrackTimeand can be used as a context manager.Example:
from cdxcore.util import TrackTime tt = TrackTime() with tt("io"): # ... do IO ... pass with tt("processing") as pt: # ... do processing ... with pt("subprocessing") as spt: # ... do sub processing ... pass pass
- Parameters:
- sub_topicstr
Name of the sub topic to get or create. If the sub topic does not exist, it is created and returned.
- startbool, default
True Whether to start the sub topic timer immediately. If
False, the sub topic timer will remain stopped untilcdxcore.util.TrackTime.start()is called on it.
- Returns:
- sub_timer
cdxcore.util.TrackTime The timer for the sub topic. If the sub topic does not exist, it is created and returned. If it already exists, it is returned as is.
- sub_timer
- Raises:
- In use:
KeyError If a sub topic is currently running (i.e. is not
cdxcore.util.TrackTime.is_stopped()), aKeyErroris raised.
- In use:
- add(timer)[source]#
Add the seconds of another timer including its sub topics to this timer. This does not change the other timer.
Returns
selffor chaining.
- add_time(seconds)[source]#
Manual lap count: Add
secondsto this timer’s accumulated time and incrementcount.
- property average_lap_seconds: float | None#
Average lap seconds added to this timer excluding the current running lap if the timer is running.
That means:
from cdxcore.util import TrackTime import time tt = TrackTime() time.sleep(1) tt.lap_time() time.sleep(2) print( tt.average_lap_seconds ) # -> 1 not 1.5
- fmt_average_lap_seconds()[source]#
A human readable string for
cdxcore.util.TrackTime.average_lap_seconds, e.g. “1s” or “3:21” computed usingcdxcore.util.fmt_seconds().
- fmt_lap_seconds()[source]#
A human readable string for
cdxcore.util.TrackTime.lap_seconds, e.g. “1s” or “3:21” computed usingcdxcore.util.fmt_seconds(). This only counts full laps, not any currently running lap/interval.
- fmt_seconds()[source]#
A human readable string for
cdxcore.util.TrackTime.seconds, e.g. “1s” or “3:21” computed usingcdxcore.util.fmt_seconds(). This includes time until now if the timer was not stopped.
- get(sub_topic, default=None, *, clone_default=True)[source]#
Return sub topic timer if present; otherwise create it from
default.- Parameters:
- sub_topicstr
Sub topic name.
- defaultfloat | TrackTime | None, default
None If a
cdxcore.util.TrackTimeinstance, it will be inserted and returned.If a
float, a stopped sub timer with that initial elapsed seconds is created.If
None, this returnsNone.
- clone_default: bool, default ``True``
If
Trueanddefaultis acdxcore.util.TrackTime, a clone of it is created for the sub topic. Otherwise, the same instance is used.
- Returns:
- sub_timer
cdxcore.util.TrackTime| None The sub topic timer if it exists, or if it was created from
default; otherwiseNone.
- sub_timer
- items()[source]#
dict.items()for the dictionary of sub topics
- keys()[source]#
dict.keys()for the dictionary of sub topics
- property lap_seconds: float#
Total seconds recorded for all laps, excluding the currently running time.
For example:
from cdxcore.util import TrackTime import time tt = TrackTime() time.sleep(1) tt.lap_time() time.sleep(1) tt.lap_time() time.sleep(1) print( tt.lap_seconds ) # -> around 2 NOT 3
- lap_time(sub_topic=None)[source]#
Record the elapsed time since the last lap and reset the lap reference time to “now”.
This computes the seconds elapsed since
cdxcore.util.TrackTime.reference_time, adds them to this timer, resets the reference time to “now”, and returns the elapsed seconds.If
sub_topicis provided, the same elapsed seconds are also recorded into the sub topic timerself[sub_topic].Example:
from cdxcore.util import TrackTime import time tt = TrackTime("f") for i in range(3): time.sleep(1) tt.lap_time() print(tt.average_lap_seconds) # -> around 1
This function can also be used to track lap time for sub topics:
from cdxcore.util import TrackTime import time tt = TrackTime("f") for i in range(5): time.sleep(1) tt.lap_time("t1") time.sleep(0.2) tt.lap_time("t2") print(tt["t1"].average_lap_seconds) # -> around 1 print(tt["t2"].average_lap_seconds) # -> around 0.2 print(tt.seconds) # -> around 6 print(tt.count) # == 10
- Parameters:
- sub_topicstr | None, default
None Sub topic to store the elapsed seconds under, or
Noneto only update this timer.
- sub_topicstr | None, default
- Returns:
- secondsfloat | None
The elapsed seconds since the last lap, or
Noneif the timer is stopped and no recording is requested.
- record(sub_topic, seconds)[source]#
Manually record
secondsa sub topic, and increase the lap count for the sub topic. Does not incrementself. Returns the associated sub timer. If a new timer is created, it will be in stopped mode.Use
cdxcore.util.TrackTime.lap_time()instead to record the seconds elapsed sincecdxcore.util.TrackTime.reference_timefor a sub topic.
- property reference_time: float | None#
Returns the reference time since when progress is measured, or
Noneif the current timer is stopped
- property seconds: float#
Total elapsed seconds since construction, including current time since last lap reference time if the timer is running.
- stop(record_lap=True)[source]#
Stops the timer, records a lap if
record_lapisTrue, and returns seconds elapsed so far
- time()[source]#
Return current time in seconds.
This calls
cdxcore.util.TrackTime.TIMER, which defaults totime.time().
- values()[source]#
dict.values()for the dictionary of sub topics
- cdxcore.util.expected_str_fmt_args(fmt)[source]#
Inspect a
{}Python format string and report what arguments it expects.- Returns:
- InformationMapping
A dictionary containing:
auto_positional: count of’{}’ fieldspositional_indices: explicit numeric field indices used (e.g.,{0},{2})keywords: named fields used (e.g.,{user},{price:.2f})
- cdxcore.util.fmt_big_byte_number(byte_cnt, str_B=True)[source]#
Return a formatted big byte string, e.g. 12.35MB. Uses 1024 as base for KB.
Use
cdxcore.util.fmt_big_number()for converting general numbers using 1000 blocks instead.- Parameters:
- byte_cntint
Number of bytes.
- str_Bbool
If
True, return"GB","MB"and"KB"units. Moreover, ifbyte_cnt` is less than 10KB, then this will add ``"bytes"e.g."1024 bytes".If
False, return"G","M"and"K"only, and do not add"bytes"to smallerbyte_cnt.
- Returns:
- Textstr
String.
- cdxcore.util.fmt_big_number(number)[source]#
Return a formatted big number string, e.g. 12.35M instead of all digits.
Uses decimal system and “B” for billions. Use
cdxcore.util.fmt_big_byte_number()for byte sizes i.e. 1024 units.- Parameters:
- numberint
Number to format.
- Returns:
- Textstr
String.
- cdxcore.util.fmt_date(dt)[source]#
Returns string representation for a date of the form “YYYY-MM-DD”.
If passed a
datetime.datetime, it will format itsdatetime.datetime.date().
- cdxcore.util.fmt_datetime(dt, *, sep=':', ignore_ms=False, ignore_tz=True)[source]#
Convert
datetime.datetimeto a string of the form “YYYY-MM-DD HH:MM:SS”.If present, microseconds are added as digits:
YYYY-MM-DD HH:MM:SS,MICROSECONDS
Optionally a time zone is added via:
YYYY-MM-DD HH:MM:SS+HH YYYY-MM-DD HH:MM:SS+HH:MM
Output is reduced accordingly if
dtis adatetime.timeordatetime.date.- Parameters:
- dt
datetime.datetime,datetime.date, ordatetime.time Input.
- sepstr, optional
Separator for hours, minutes, seconds. The default
':'is most appropriate for visualization but is not suitable for filenames.- ignore_msbool, optional
Whether to ignore microseconds. Default
False.- ignore_tzbool, optional
Whether to ignore the time zone. Default
True.
- dt
- Returns:
- Textstr
String.
- cdxcore.util.fmt_dict(dct, *, sort=False, none='-', link='and')[source]#
Return a readable representation of a dictionary.
This assumes that the elements of the dictionary itself can be formatted well with
str().For a dictionary
dict(a=1,b=2,c=3)this function will return"a: 1, b: 2, and c: 3".- Parameters:
- dctdict
The dictionary to format.
- sortbool, optional
Whether to sort the keys. Default is
False.- nonestr, optional
String to be used if dictionary is empty. Default is
"-".- linkstr, optional
String to be used to link the last element to the previous string. Default is
"and".
- Returns:
- Textstr
String.
- cdxcore.util.fmt_digits(integer, sep=',')[source]#
String representation of an integer with 1000 separators: 10000 becomes “10,000”.
- Parameters:
- integerint
The number. The function will
int()the input which allows for processing of a number of inputs (such as strings) but might cut off floating point numbers.- sepstr
Separator;
","by default.
- Returns:
- Textstr
String.
- cdxcore.util.fmt_filename(filename, by='default')[source]#
Replaces invalid filename characters such as
\\,:, or/by a different character. The returned string is technically a valid file name under both Windows and Linux.However, that does not prevent the filename to be a reserved name, for example “.” or “..”.
- Parameters:
- filenamestr
Input string.
- bystr | Mapping, optional.
A dictionary of characters and their replacement. The default value
"default"leads to usingcdxcore.util.DEF_FILE_NAME_MAP.
- Returns:
- Textstr
Filename
- cdxcore.util.fmt_list(lst, *, none='-', link='and', sort=False)[source]#
Returns a formatted string of a list, its elements separated by commas and (by default) a final ‘and’.
If the list is
[1,2,3]then the function will return"1, 2 and 3".- Parameters:
- lstlist.
The
list()operator is applied tolst, so it will resolve dictionaries and generators.- nonestr, optional
String to be used when
listis empty. Default is"-".- linkstr, optional
String to be used to connect the last item. Default is
"and".- sortbool, optional
Whether to sort the list. Default is
False.
- Returns:
- Textstr
String.
- cdxcore.util.fmt_now()[source]#
Returns the
cdxcore.util.fmt_datetime()applied todatetime.datetime.now()
- cdxcore.util.fmt_seconds(seconds, *, eps=1e-08)[source]#
Generate format string for seconds, e.g. “23s”” for
seconds=23, or “1:10” forseconds=70.- Parameters:
- secondsfloat
Seconds as a float.
- epsfloat
anything below
epsis considered zero. Default1E-8.
- Returns:
- Secondsstring
- cdxcore.util.fmt_time(dt, *, sep=':', ignore_ms=False)[source]#
Convers a time to a string with format “HH:MM:SS”.
Microseconds are added as digits:
HH:MM:SS,MICROSECONDS
If passed a
datetime.datetime, then this function will format only itsdatetime.datetime.time()part.Time Zones
Note that while
datetime.timeobjects may carry atzinfotime zone object, the correspondingdatetime.time.utcoffset()function returnsNoneif we do not provide adtparameter, see tzinfo documentation. That meansdatetime.time.utcoffset()is only useful if we havedatetime.datetimeobject at hand. That makes sense as a time zone can change date as well.We therefore here do not allow
dtto contain a time zone.Use
cdxcore.util.fmt_datetime()for time zone support- Parameters:
- dt
datetime.time Input.
- sepstr, optional
Separator for hours, minutes, seconds. The default
':'is most appropriate for visualization but is not suitable for filenames.- ignore_msbool
Whether to ignore microseconds. Default is
False.
- dt
- Returns:
- Textstr
String.
- cdxcore.util.fmt_timedelta(dt, *, sep='')[source]#
Returns string representation for a time delta in the form “DD:HH:MM:SS,MS”.
- Parameters:
- dt
datetime.timedelta Timedelta.
- sep
Identify the three separators: between days, and HMS and between microseconds:
DD*HH*MM*SS*MS 0 1 1 2
sepcan be a string, in which case:If it is an empty string, all separators are
''.A single character will be reused for all separators.
If the string has length 2, then the last character is used for
'2'.If the string has length 3, then the chracters are used accordingly.
sepcan also be a collection ie atupleorlist. In this case each element is used accordingly.
- dt
- Returns:
- Textstr
String with leading sign. Returns “” if
timedeltais 0.
- cdxcore.util.get_calling_function_name(default)[source]#
When called from a function, returns the name of the calling function. Otherwise returns
default.Example:
from cdxcore.util import get_calling_function_name def g(): print(get_calling_function_name("no caller")) # -> prints "f def f(): g()
- cdxcore.util.getsizeof(obj)[source]#
Approximates the size of an object.
In addition to calling
sys.getsizeof()this function also iterates embedded containers, numpy arrays, and panda dataframes. :meta private:
- cdxcore.util.is_atomic(o)[source]#
Whether an element is atomic.
Returns
Trueifois astring,int,float,datedatime.date,bool, or anumpy.generic
- cdxcore.util.is_filename(filename, by='default')[source]#
Tests whether a filename is indeed a valid filename.
- Parameters:
- filenamestr
Supposed filename.
- bystr | Collection, optional
A collection of invalid characters. The default value
"default"leads to using they keys ofcdxcore.util.DEF_FILE_NAME_MAP.
- Returns:
- Validitybool
Trueiffilenamedoes not contain any invalid characters contained inby.
- cdxcore.util.is_float(o)[source]#
Checks whether a type is a
floatwhich includes numpy floating types
- cdxcore.util.is_function(f)[source]#
Checks whether
fis a function in an extended sense.Check
cdxcore.util.types_functions()for what is tested against. In particularis_functiondoes not test positive for properties.
- cdxcore.util.plain(inn, *, sorted_dicts=False, native_np=False, dt_to_str=False)[source]#
Converts a python structure into a simple atomic/list/dictionary collection such that it can be read without the specific imports used inside this program.
For example, objects are converted into dictionaries of their data fields.
- Parameters:
- inn
some object.
- sorted_dictsbool, optional
use SortedDicts instead of dicts. Since Python 3.7 all dictionaries are sorted anyway.
- native_npbool, optional
convert numpy to Python natives.
- dt_to_strbool, optional
convert dates, times, and datetimes to strings.
- Returns:
- Textstr
Filename
- cdxcore.util.qualified_name(x, module=False)[source]#
Return qualified name including module name of some Python element.
For the most part, this function will try to
getattr()the__qualname__and__name__ofxor its type. If all of these fail, an attempt is made to converttype(x)into a string.Class Properties
When reporting qualified names for a
property(), there is a nuance: at class level, a property will be identified by its underlying function name. Once an object is created, though, the property will be identified by the return type of the property:class A(object): def __init__(self): self.x = 1 @property def p(self): return x qualified_name(A.p) # -> "A.p" qualified_name(A().p) # -> "int"
- Parameters:
- xAny
Some Python element.
- modulebool | str, default
False Whether to also return the containing module if available. Use a string as separator to append the module name to the returned name:
# define in module test.py def f(): pass # in another module from test import f qualified_name(f,"@") -> f@test
- Returns:
- qualified namestr
The name, if
moduleisFalse.- (qualified name, module_name)tuple
The name, if
moduleisTrue. Note that the module name returned might be""if no module name could be determined.- ``{qualified name}{module}{module_name}``str
If
moduleis a string.
- Raises:
RuntimeErrorif not qualfied name forxor its type could be found.