On torch.compile

When --use_torch_compile is set to True, the code will use torch.compile to AOT/JIT compile the model. During this process, Pytorch will store compilation caches in the /tmp folder of your machine (Linux). These caches can pile up to several GBs of data, which may block the /tmp folder for more important functions of the OS. To avoid any problems, it is recommended to point the target cache directory to a folder that is less OS-critical, e.g. by placing the folder somewhere into the /home directory. This can be easily set using the TORCHINDUCTOR_CACHE_DIR environment variable. To set this user-wide one can add it to the initialization of the default shell:

# inside ~/.bashrc
export TORCHINDUCTOR_CACHE_DIR=/home/path/to/cache_dir