Csv2JsonConverter class
A class to convert from previous log structure (CSV) to a current log structure (JSON) with/without Brotli compression.
Files for conversion should be named as follows 'prefix.rank/node.extension' e.g. 'data.0.csv', 'data.1.csv' Changes input CSV files e.g. <time_step/phase>, <object-id>, <time> to JSON: {"phases":[ {"tasks":[ {"time":0.036448,"resource":"cpu","object":51539607559,"node":7}, {"time":0.0298901,"resource":"cpu","object":47244640263,"node":7} ], "id":0}]}
Static methods
- def _get_data_dir(dir_path: str) -> str
- Return a path to data directory.
- def _get_data_phase_sorted(data: list) -> dict
- Sort data with respect to the phase. Return dict with phases as keys.
- def _read_csv(file_to_read: str) -> list
- Read CSV and returns a list of dicts (phase, object_id, time) ready to save into JSON.
Methods
- def _Csv2JsonConverter__parse_args(self)
- Parse arguments.
- def _convert_file(self, file_path: tuple) -> None
- Convert a file and saves converted file to given path.
- def _get_files_for_conversion(self) -> list
- Return list of tuples as follows (file_to_convert_path, converted_file_path).
- def _write_json(self, output_path: str, data_to_convert: dict, node: int) -> None
- Convert data to JSON and saves to output path.
- def run(self)
- Get lists of files to convert. Iterate over it and converts each file.
Special methods
- def __init__(self)