TIMING TORCH HOW TO
TIMING TORCH CODE

Please read local_rank from os.environ('LOCAL_RANK') instead. WARNING::–use_env is deprecated and will be removed in future releases. Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. The module is deprecated and going to be removed in future.Migrate to home/ma/uname/maanaconda3/envs/mdetr/lib/python3.8/site-packages/torch/distributed/launch.py:163: DeprecationWarning: The ‘warn’ method is deprecated, use ‘warning’ instead Python -m -nproc_per_node=2 main_dino.py -arch resnet50 -optimizer sgd -weight_decay 1e-4 -weight_decay_end 1e-4 -global_crops_scale 0.14 1 -local_crops_scale 0.05 0.14 -data_path /home/ma/uname/dataset/imagenet/train -output_dir /temp

TIMING TORCH HOW TO

My python version is 3.8.11, pytorch version is 1.9.0, : 11.1.ĭoes anyone have any idea how to debug this error or to solve this problem? Thanks in Advance!

TIMING TORCH CODE

The code works fine when I train on a single gpu but crashes when I use 2 gpus. WARNING.I am trying to train dino with 2 A6000 gpus. WARNING.api:Sending process 4006362 closing signal SIGINT ^CWARNING.server.api:Received 2 death signal, shutting down workers 09:34:37,477 - mmdet - INFO - load checkpoint from http path: Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.3, CUDNN_VERSION=8.2.0, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, NVCC architecture flags: -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_80,code=sm_80 -gencode arch=compute_86,code=sm_86

LAPACK is enabled (usually provided by MKL) Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications PyTorch compiling details: PyTorch built with: 17:31:13,772 - mmdet - INFO - load checkpoint from http path: 17:33:20,582 - mmdet - INFO - initialize Bottleneck with init_cfg ]

Timing torch