API Reference

File System Accessors

Local file system

HDFS (Hadoop File System)

S3 (AWS S3)

Zip Archive

Error

Pathlib-like API

PFIO v2 API has utility tool that behaves like pathlib in Python’s standard library. Paths can be manipulated like this:

from pfio.v2 import from_url
from pfio.v2.pathlib import Path

with from_url('s3://your-bucket') as s3:
  p = Path('foo', fs=s3)
  p2 = p / 'bar'
  with p2.open() as fp:
    # yields s3://your-bucket/foo/bar
    fp.read()

It tries to be compatible with pathlib.Path as much as possible, but several methods are not yet implemented.

Cache API

PFIO provides experimental cache API to improve performance of repetitive access to the data collection.

Example

Here let us suppose we have a file that includes a list of paths to images.

/path/to/image1.jpg
/path/to/image2.jpg
...
/path/to/imageN.jpg

The PyTorch Dataset class with using NaiveCache as an example can be implemented as follows.

from pfio.cache import NaiveCache


class MyDataset(torch.utils.data.Dataset):
    def __init__(self, image_paths):
        self.paths = image_paths
        self.cache = NaiveCache(len(image_paths), do_pickle=True)

    def __len__(self):
        return len(self.paths)

    def _read_image(self, i):
        return cv2.imread(self.paths[i]).transpose(2, 0, 1)

    def __getitem__(self, i):
        x = self.cache.get_and_cache(i, self._read_image)

        # This is equivalent
        # x = self.cache.get(i)
        # if not x:
        #     x = cv2.imread(self.paths[i]).transpose(2, 0, 1)
        #     self.cache.put(i, x)

        return torch.Tensor(x)

By calling get_and_cache of the cache in __getitem__ method, it will check if the data for the specified index is already cached. If there already is, it reads the data from the cache and return, otherwise it calls the actual data loading function, add it to the cache, and return it. Therefore load the data from the storage only when necessary, which is at the first access to each data.

PFIO cache API provides NaiveCache, FileCache`and :class:`~MultiprocessFileCache. They all share the same core idea and interface. The difference is how to manage the cached data.

The NaiveCache keeps everything in memory, making it virtually zero overhead. The cache capacity is limited by the memory size, thus it would not be suitable for large-scale datasets.

The FileCache and the MultiprocessFileCache both store the cached data in a filesystem. The FileCache is designed for single-process data load. In case of parallelized data loading, which is relatively common in deep learning workloads, consider using MultiprocessFileCache.

Also, these file-based caches support cache data persistency. Once the cache is completely built, we can keep them as files by calling :func:FileCache.preserve, and we can recover the cache from the preserved files by calling :func:FileCache.preload. This is useful when we want to reuse the cache already built in a previous workload.

Currently deletion of a data from cache is not supported.

Toplevel Functions in v1(deprecated)

Note

Toplevel functions will be deprecated in 2.0 and removed in 2.1. Please use V2 API instead.

Note

With environment variable KRB5_KTNAME=path/to/your.keytab set, hdfs handler automatically starts automatic and periodical updating Kerberos ticket using krbticket . The update frequency is every 10 minutes by default.

Note

Only the username in the first entry in The keytab will be used to update the Kerberos ticket.