slurm_parallel.out

[1/2]: Loading data...
######################################################
# Distributed Random Forest in Scikit-Learn with MPI #
######################################################

[0/2]: Loading data...
Using truly parallel dataloader...
File size is 2390277560 bytes.
[1/2]: Construct array with line starts and lengths in bytes.
After Allgatherv: All line starts: [         0        479        957 ... 2390276125 2390276602 2390277082]
[0/2]: Construct array with line starts and lengths in bytes.
[1/2]: Make global train-test split.
[0/2]: Make global train-test split.
[1/2]: Decode 1250000 test samples from file.
[0/2]: Decode 1250000 test samples from file.
[1/2]: Draw local 1875000 train indices.
[1/2]: Decode train lines from file.
[0/2]: Draw local 1875000 train indices.
[0/2]: Decode train lines from file.
Elapsed time data loading: global average 2.5e+02s, local 2.5e+02s
[0/2]: DONE.
Local train samples and targets have shapes (1875000, 18) and (1875000,).
Global test samples and targets have shapes (1250000, 18) and (1250000,).
Labels are [0. 0. 0. ... 0. 0. 1.]
Elapsed time forest creation: global average 2.8e-05s, local 3.8e-05s
[0/2]: Set up and train local random forest with 50 trees and random state 2.
[1/2]: DONE.
Local train samples and targets have shapes (1875000, 18) and (1875000,).
Global test samples and targets have shapes (1250000, 18) and (1250000,).
Labels are [1. 0. 0. ... 0. 0. 1.]
[1/2]: Set up and train local random forest with 50 trees and random state 3.
Elapsed time training: global average 8.8e+02s, local 8.8e+02s
[0/2]: Evaluate random forest.
[0/2]: Get predictions of individual sub estimators.
[1/2]: Evaluate random forest.
[1/2]: Get predictions of individual sub estimators.
[1/2]: Calculate majority vote via histograms.
[0/2]: Calculate majority vote via histograms.
[1/2]: Local accuracy is 0.7971136, global accuracy is 0.7998568.
[0/2]: Local accuracy is 0.79728, global accuracy is 0.7998568.
Elapsed time test: global average 56s, local 56s
[1/2]: Loading data...
######################################################
# Distributed Random Forest in Scikit-Learn with MPI #
######################################################

[0/2]: Loading data...
Using root-based dataloader with Scatterv...
There are 3750000 train and 1250000 test samples.
Local train samples: [1875000 1875000]
train_indices have shape (3750000,).
Elapsed time data loading: global average 35s, local 35s
[0/2]: DONE.
Local train samples and targets have shapes (1875000, 18) and (1875000,).
Global test samples and targets have shapes (1250000, 18) and (1250000,).
Labels are [0. 0. 0. ... 1. 0. 0.]
Elapsed time forest creation: global average 2e-05s, local 1.9e-05s
[0/2]: Set up and train local random forest with 50 trees and random state 2.
[1/2]: DONE.
Local train samples and targets have shapes (1875000, 18) and (1875000,).
Global test samples and targets have shapes (1250000, 18) and (1250000,).
Labels are [0. 1. 1. ... 0. 1. 0.]
[1/2]: Set up and train local random forest with 50 trees and random state 3.
Elapsed time training: global average 8.8e+02s, local 8.8e+02s
[0/2]: Evaluate random forest.
[0/2]: Get predictions of individual sub estimators.
[1/2]: Evaluate random forest.
[1/2]: Get predictions of individual sub estimators.
[0/2]: Calculate majority vote via histograms.
[1/2]: Calculate majority vote via histograms.
[0/2]: Local accuracy is 0.7977704, global accuracy is 0.800064.
[1/2]: Local accuracy is 0.7974024, global accuracy is 0.800064.
Elapsed time test: global average 57s, local 57s

============================= JOB FEEDBACK =============================

NodeName=uc2n[222,237]
Job ID: 22915390
Cluster: uc2
User/Group: ku4408/scc
State: COMPLETED (exit code 0)
Nodes: 2
Cores per node: 80
CPU Utilized: 01:11:56
CPU Efficiency: 1.23% of 4-01:36:00 core-walltime
Job Wall-clock time: 00:36:36
Memory Utilized: 3.12 GB
Memory Efficiency: 1.77% of 175.78 GB