Please tell me about the unitv2 v-training failure.



  • I get the following error:

    [TASK] 2022-11-10 02:02:03,694 - [line:545] - ERROR: main(args)
    File "/home/m5stack/VTrainingService/NanoDet/nanodet/tools/train.py", line 90, in main
    callbacks=[ProgressBar(refresh_rate=0)] # disable tqdm bar
    File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/utilities/argparse.py", line 339, in insert_env_defaults
    return fn(self, **kwargs)
    File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 481, in init
    gpu_ids, tpu_cores = self._parse_devices(gpus, auto_select_gpus, tpu_cores)
    File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1779, in _parse_devices
    return device_parser._parse_devices(gpus, auto_select_gpus, tpu_cores)
    File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/utilities/device_parser.py", line 61, in _parse_devices
    gpu_ids = parse_gpu_ids(gpus)
    File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/utilities/device_parser.py", line 105, in parse_gpu_ids
    return _sanitize_gpu_ids(gpus)
    File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/utilities/device_parser.py", line 188, in _sanitize_gpu_ids
    f"You requested gpu: {gpus}\n But your machine only has: {all_available_gpus}"
    pytorch_lightning.utilities.exceptions.MisconfigurationException: You requested gpu: [0]
    But your machine only has: []

    [TASK] 2022-11-10 02:02:03,694 - [line:677] - ERROR: Error occurred during training: But your machine only has: []
    Traceback (most recent call last):
    File "framework.py", line 670, in main
    trainNanodet_m()
    File "framework.py", line 548, in trainNanodet_m
    raise Exception("Error occurred during training: " + last_line)
    Exception: Error occurred during training: But your machine only has: []

    What could be the cause?


  • Global Moderator

    There is some problem with our traning GPUs. We are fixing.


  • Global Moderator

    There are some problems with our GPUs, we are fixing now.



  • I'm also getting this exact error.


  • Global Moderator

    I am sorry about that, because the GPU problem we are fixing the environment as fast as possible.



  • Is this also what is causing the hold on the V1 training?

    http://v-training.m5stack.com/



  • Thank you for your comment.
    I am looking forward to it.