Parsing Text

This page demonstrates the parsing process for text events.

Preparing Sample Event Logs

First, let’s import some libraries and prepare the environment for our sample event logs:

>>> import os
>>> import tempfile
>>> # Prepare temp dirs for storing event files
>>> tmpdirs = {}

Before parsing a event file, we need to generate it first. The sample event files are generated by three commonly used event log writers.

PyTorchTensorFlow2 / KerasTensorboardX

We can generate the events by PyTorch:

>>> tmpdirs['torch'] = tempfile.TemporaryDirectory()
>>> from torch.utils.tensorboard import SummaryWriter
>>> log_dir = tmpdirs['torch'].name
>>> writer = SummaryWriter(log_dir)
>>> writer.add_text('textA', 'lorem ipsum', 0)
>>> writer.add_text('textA', 'dolor sit amet', 1)
>>> writer.add_text('textB', 'consectetur adipiscing', 0)
>>> writer.add_text('textB', 'elit', 1)
>>> writer.close()

and quickly check the results:

>>> from tbparse import SummaryReader
>>> SummaryReader(log_dir, pivot=True).text
   step           textA                   textB
0     0     lorem ipsum  consectetur adipiscing
1     1  dolor sit amet                    elit

Parsing Event Logs

In different use cases, we will want to read the event logs in different styles. We further show different configurations of the tbparse.SummaryReader class.

In the following samples, we use the event files generated by PyTorch for simplicity. Event files generated by TensorFlow2/Keras or TensorboardX can be parsed in the same way. (escape the special characters in tags)

>>> log_dir = tmpdirs['torch'].name

Now we load the event logs as pandas.DataFrame.

Long FormatWide Format

>>> reader = SummaryReader(log_dir) # long format
>>> reader.text
   step    tag                   value
0     0  textA             lorem ipsum
1     1  textA          dolor sit amet
2     0  textB  consectetur adipiscing
3     1  textB                    elit

Warning

When accessing SummaryReader.text, the events stored in each event file are collected internally. The best practice is to store the returned results in a DataFrame as shown in the samples, instead of repeatedly accessing SummaryReader.text.

Extra Columns

See the Extra Columns page for more details.