1. Introduction to MOA Machine Learning for Streams¶
https://moa.cms.waikato.ac.nz/
"MOA is the most popular open source framework for data stream mining, with a very active growing community (blog). It includes a collection of machine learning algorithms (classification, regression, clustering, outlier detection, concept drift detection and recommender systems) and tools for evaluation. Related to the WEKA project, MOA is also written in Java, while scaling to more demanding problems."
It is written in Java. Unfortunatelly I am more comfortable with Python & PySpark. I also noticed that Python (and R) libraries often have much better documentation than anything in Java (my opinion).
There are 3 ways to use MOA :
- GUI - download prebuild MOA from official webpage or build the most current version from GitHub GUI is easy to use - if you know what you are looking for :D
- command line - run experiments by commands (or scripts), commands are show also in GUI and can be grouped to create scripts, for examples see:
https://github.com/5uperpalo/Machine-Learning/blob/master/Machine-Learning-Tools/weka_moa_elki_pig.md - java code - run the code or use jupyter notebook example in this notebook
2. How to run Java and MOA in Jupyter notebook¶
- Step-by-step procedure to install java kernel to jupyter notebook is available at :
https://github.com/SpencerPark/IJava If you have JRE (like me), than you have to install JDK and you have to change/adjust PATH variable in your OS to point to new java. - Use 'Jupyter magic' to load MOA either online from maven or offline from the downloaded MOA directory. Jupyter magic starts with symbol % and MUST BE in the 1st line of the cell. Commented line can go only after Jupyter magic.
// online
%maven nz.ac.waikato.cms.moa:moa:2019.05.0
// offline
%jars C:/#CVUT/work_current/FIREMAN/moa-release-2019.05.0/lib/moa.jar
3. Example of experiment with predefined task (clusterer / classifier evaluation)¶
3.1. Load local MOA library¶
In [2]:
Copied!
%jars C:/#CVUT/GitHub/FIREMAN/moa-release-2019.05.0/lib/moa.jar
%jars C:/#CVUT/GitHub/FIREMAN/moa-release-2019.05.0/lib/moa.jar
3.2. Load and prepare dataset¶
There are different load methods for classification and clustering. MOA works with arff files. CSV file stream is available only for clustering. ARFF file for clustering kept raising java lang error. => I am using arff for classificaiton and csv for clustering
In [3]:
Copied!
import moa.streams.ArffFileStream;
import moa.streams.clustering.FileStream;
import moa.streams.clustering.SimpleCSVStream;
String file_arff = "Tennessee_Event-Driven/datasets/dataset_standard_scaled.arff";
String file_csv = "Tennessee_Event-Driven/datasets/dataset_standard_scaled_moa.csv";
// classification
// specify last column as class column(-1)
ArffFileStream class_stream = new ArffFileStream(file_arff,-1);
class_stream.prepareForUse();
// clustering
SimpleCSVStream clust_stream = new SimpleCSVStream();
clust_stream.csvFileOption.setValueViaCLIString(file_csv);
clust_stream.classIndexOption.setValueViaCLIString("-1");
clust_stream.prepareForUse();
import moa.streams.ArffFileStream;
import moa.streams.clustering.FileStream;
import moa.streams.clustering.SimpleCSVStream;
String file_arff = "Tennessee_Event-Driven/datasets/dataset_standard_scaled.arff";
String file_csv = "Tennessee_Event-Driven/datasets/dataset_standard_scaled_moa.csv";
// classification
// specify last column as class column(-1)
ArffFileStream class_stream = new ArffFileStream(file_arff,-1);
class_stream.prepareForUse();
// clustering
SimpleCSVStream clust_stream = new SimpleCSVStream();
clust_stream.csvFileOption.setValueViaCLIString(file_csv);
clust_stream.classIndexOption.setValueViaCLIString("-1");
clust_stream.prepareForUse();
3.3. Prepare learners¶
In [7]:
Copied!
import moa.classifiers.meta.AdaptiveRandomForest;
//import moa.classifiers.lazy.kNN;
import moa.clusterers.clustree.ClusTree;
// kNN knn_classifier = new kNN();
AdaptiveRandomForest ARF = new AdaptiveRandomForest();
// set some example options
// https://www.cs.waikato.ac.nz/~abifet/MOA/API/classmoa_1_1options_1_1_int_option.html;
ClusTree clustree = new ClusTree();
clustree.horizonOption.setValue(100);
clustree.maxHeightOption.setValue(2);
import moa.classifiers.meta.AdaptiveRandomForest;
//import moa.classifiers.lazy.kNN;
import moa.clusterers.clustree.ClusTree;
// kNN knn_classifier = new kNN();
AdaptiveRandomForest ARF = new AdaptiveRandomForest();
// set some example options
// https://www.cs.waikato.ac.nz/~abifet/MOA/API/classmoa_1_1options_1_1_int_option.html;
ClusTree clustree = new ClusTree();
clustree.horizonOption.setValue(100);
clustree.maxHeightOption.setValue(2);
3.4. Evaluate learners¶
Thre are 3 ways to find out methods and options we can/must apply.
- GUI - check the the options, final command(visible below the tabs in GUI) and find methods in code/documentation
- documentation - easiest to understand, but often not up-to-date and some methods are not well documented https://www.cs.waikato.ac.nz/~abifet/MOA/API/classmoa_1_1tasks_1_1_evaluate_clustering.html#afb62647b811d912c9e985f173f17d9bc
- read comments in the code and figure out the options https://github.com/Waikato/moa/blob/master/moa/src/main/java/moa/tasks/EvaluateClustering.java
3.4.1. Evaluate classifier¶
- create task
- set stream/learner
- add adwin concept drift detection
Note : File sizeofag-1.0.4.jar is being used to calculate RAM memory time(s) and in most of the cases is not important but generates a WARNING if not included. At the moment I have no idea how to include it.
In [11]:
Copied!
import moa.tasks.EvaluatePrequentialCV;
// import moa.evaluation.AdwinClassificationPerformanceEvaluator;
import moa.evaluation.WindowClassificationPerformanceEvaluator;
EvaluatePrequentialCV eval = new EvaluatePrequentialCV();
eval.streamOption.setCurrentObject(class_stream);
//eval.learnerOption.setCurrentObject(knn_classifier);
eval.learnerOption.setCurrentObject(ARF);
//AdwinClassificationPerformanceEvaluator evalopt = new AdwinClassificationPerformanceEvaluator();
WindowClassificationPerformanceEvaluator evalopt = new WindowClassificationPerformanceEvaluator();
evalopt.widthOption.setValue(200);
eval.evaluatorOption.setCurrentObject(evalopt);
eval.sampleFrequencyOption.setValue(200);
eval.prepareForUse();
eval.dumpFileOption.setValueViaCLIString("C:/#CVUT/GitHub/FIREMAN/Tennessee_Event-Driven/results/moa_ARF_results.csv");
eval.doTask();
import moa.tasks.EvaluatePrequentialCV;
// import moa.evaluation.AdwinClassificationPerformanceEvaluator;
import moa.evaluation.WindowClassificationPerformanceEvaluator;
EvaluatePrequentialCV eval = new EvaluatePrequentialCV();
eval.streamOption.setCurrentObject(class_stream);
//eval.learnerOption.setCurrentObject(knn_classifier);
eval.learnerOption.setCurrentObject(ARF);
//AdwinClassificationPerformanceEvaluator evalopt = new AdwinClassificationPerformanceEvaluator();
WindowClassificationPerformanceEvaluator evalopt = new WindowClassificationPerformanceEvaluator();
evalopt.widthOption.setValue(200);
eval.evaluatorOption.setCurrentObject(evalopt);
eval.sampleFrequencyOption.setValue(200);
eval.prepareForUse();
eval.dumpFileOption.setValueViaCLIString("C:/#CVUT/GitHub/FIREMAN/Tennessee_Event-Driven/results/moa_ARF_results.csv");
eval.doTask();
Out[11]:
learning evaluation instances,evaluation time (cpu seconds),model cost (RAM-Hours),[avg] classified instances,[err] classified instances,[avg] classifications correct (percent),[err] classifications correct (percent),[avg] Kappa Statistic (percent),[err] Kappa Statistic (percent),[avg] Kappa Temporal Statistic (percent),[err] Kappa Temporal Statistic (percent),[avg] Kappa M Statistic (percent),[err] Kappa M Statistic (percent) 200.0,0.484375,0.0,200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,0.0,0.0 400.0,0.578125,0.0,400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,0.0,0.0 600.0,0.671875,0.0,600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,0.0,0.0 800.0,0.765625,0.0,800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,0.0,0.0 1000.0,0.84375,0.0,1000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,0.0,0.0 1200.0,0.9375,0.0,1200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,0.0,0.0 1400.0,1.078125,0.0,1400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,0.0,0.0 1600.0,1.46875,0.0,1600.0,0.0,95.85,0.6258327785172871,95.85,0.6258327785172871,-730.0,125.16655570345725,94.07142857142858,0.894046826453262 1800.0,1.609375,0.0,1800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 2000.0,1.6875,0.0,2000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 2200.0,1.78125,0.0,2200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 2400.0,1.875,0.0,2400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 2600.0,1.953125,0.0,2600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 2800.0,2.046875,0.0,2800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 3000.0,2.484375,0.0,3000.0,0.0,90.15,0.6258327785172848,90.15,0.6258327785172848,-1869.9999999999977,125.1665557034571,90.15,0.6258327785172848 3200.0,2.671875,0.0,3200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 3400.0,2.78125,0.0,3400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 3600.0,2.890625,0.0,3600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 3800.0,2.96875,0.0,3800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 4000.0,3.046875,0.0,4000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 4200.0,3.125,0.0,4200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 4400.0,3.40625,0.0,4400.0,0.0,95.75,0.4859126579037736,95.75,0.4859126579037736,-750.0,97.182531580755,95.75,0.4859126579037736 4600.0,3.703125,0.0,4600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 4800.0,3.859375,0.0,4800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 5000.0,4.03125,0.0,5000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 5200.0,4.1875,0.0,5200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 5400.0,4.265625,0.0,5400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 5600.0,4.390625,0.0,5600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 5800.0,4.59375,0.0,5800.0,0.0,95.60000000000001,0.7378647873726214,95.60000000000001,0.7378647873726214,-780.0,147.57295747452437,95.60000000000001,0.7378647873726214 6000.0,5.078125,0.0,6000.0,0.0,92.85,1.8566696588844833,92.85,1.8566696588844833,0.0,0.0,92.85,1.8566696588844833 6200.0,5.203125,0.0,6200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 6400.0,5.296875,0.0,6400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 6600.0,5.390625,0.0,6600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 6800.0,5.5,0.0,6800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 7000.0,5.59375,0.0,7000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 7200.0,5.6875,0.0,7200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 7400.0,6.0625,0.0,7400.0,0.0,94.94999999999999,0.5502524673073088,94.94999999999999,0.5502524673073088,-910.0,110.05049346146119,94.94999999999999,0.5502524673073088 7600.0,6.21875,0.0,7600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 7800.0,6.328125,0.0,7800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 8000.0,6.4375,0.0,8000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 8200.0,6.546875,0.0,8200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 8400.0,6.640625,0.0,8400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 8600.0,6.75,0.0,8600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 8800.0,7.140625,0.0,8800.0,0.0,82.75,1.8745369798670002,82.75,1.8745369798670002,-3349.9999999999977,374.9073959733992,82.75,1.8745369798670002 9000.0,7.25,0.0,9000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 9200.0,7.375,0.0,9200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 9400.0,7.484375,0.0,9400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 9600.0,7.578125,0.0,9600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 9800.0,7.671875,0.0,9800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 10000.0,7.78125,0.0,10000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 10200.0,8.078125,0.0,10200.0,0.0,89.10000000000001,0.5163977794943219,89.10000000000001,0.5163977794943219,-2079.9999999999977,103.2795558988643,89.10000000000001,0.5163977794943219 10400.0,8.3125,0.0,10400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 10600.0,8.484375,0.0,10600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 10800.0,8.671875,0.0,10800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 11000.0,8.875,0.0,11000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 11200.0,9.078125,0.0,11200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 11400.0,9.34375,0.0,11400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 11600.0,9.65625,0.0,11600.0,0.0,83.75000000000001,1.4953632037438551,83.75000000000001,1.4953632037438551,-3149.9999999999977,299.0726407487727,83.75000000000001,1.4953632037438551 11800.0,9.78125,0.0,11800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 12000.0,9.90625,0.0,12000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 12200.0,10.015625,0.0,12200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 12400.0,10.125,0.0,12400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 12600.0,10.25,0.0,12600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 12800.0,10.375,0.0,12800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 13000.0,10.515625,0.0,13000.0,0.0,93.60000000000001,0.31622776601683755,93.60000000000001,0.31622776601683755,-1179.9999999999986,63.24555320336679,93.60000000000001,0.31622776601683755 13200.0,10.921875,0.0,13200.0,0.0,94.39999999999998,0.21081851067789611,94.39999999999998,0.21081851067789611,0.0,0.0,94.39999999999998,0.21081851067789611 13400.0,11.25,0.0,13400.0,0.0,96.75,0.48591265790377464,96.75,0.48591265790377464,0.0,0.0,96.75,0.48591265790377464 13600.0,11.703125,0.0,13600.0,0.0,99.89999999999999,0.3162277660168389,99.89999999999999,0.3162277660168389,0.0,0.0,99.89999999999999,0.3162277660168389 13800.0,12.0,0.0,13800.0,0.0,99.95,0.15811388300841947,99.95,0.15811388300841947,0.0,0.0,99.95,0.15811388300841947 14000.0,12.28125,0.0,14000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 14200.0,12.46875,0.0,14200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 14400.0,12.65625,0.0,14400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 14600.0,12.96875,0.0,14600.0,0.0,81.95,1.5890248582070703,81.95,1.5890248582070703,-3509.9999999999973,317.8049716414133,81.95,1.5890248582070703 14800.0,13.0625,0.0,14800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 15000.0,13.15625,0.0,15000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 15200.0,13.25,0.0,15200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 15400.0,13.34375,0.0,15400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 15600.0,13.4375,0.0,15600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 15800.0,13.625,0.0,15800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 16000.0,13.953125,0.0,16000.0,0.0,83.65,0.6258327785172885,83.65,0.6258327785172885,-3169.9999999999977,125.16655570345725,83.65,0.6258327785172885 16200.0,14.0625,0.0,16200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 16400.0,14.171875,0.0,16400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 16600.0,14.28125,0.0,16600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 16800.0,14.390625,0.0,16800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 17000.0,14.5,0.0,17000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 17200.0,14.59375,0.0,17200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 17400.0,14.875,0.0,17400.0,0.0,85.05,0.15811388300841947,85.05,0.15811388300841947,-2889.9999999999977,31.622776601683793,85.05,0.15811388300841947 17600.0,14.984375,0.0,17600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 17800.0,15.09375,0.0,17800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 18000.0,15.203125,0.0,18000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 18200.0,15.3125,0.0,18200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 18400.0,15.40625,0.0,18400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 18600.0,15.515625,0.0,18600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 18800.0,15.96875,0.0,18800.0,0.0,82.8,0.42163702135578374,82.8,0.42163702135578374,-3339.9999999999977,84.3274042711568,82.8,0.42163702135578374 19000.0,16.15625,0.0,19000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 19200.0,16.28125,0.0,19200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 19400.0,16.421875,0.0,19400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 19600.0,16.546875,0.0,19600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 19800.0,16.671875,0.0,19800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 20000.0,16.78125,0.0,20000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 20200.0,16.9375,0.0,20200.0,0.0,94.75,0.26352313834736807,94.75,0.26352313834736807,-950.0,52.70462766947298,94.75,0.26352313834736807 20400.0,17.53125,0.0,20400.0,0.0,90.05,1.7709068612185974,90.05,1.7709068612185974,0.0,0.0,90.05,1.7709068612185974 20600.0,17.765625,0.0,20600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 20800.0,18.09375,0.0,20800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 21000.0,18.296875,0.0,21000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 21200.0,18.5,0.0,21200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 21400.0,18.6875,0.0,21400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 21600.0,18.890625,0.0,21600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 21800.0,19.203125,0.0,21800.0,0.0,80.39999999999999,0.699205898780102,80.39999999999999,0.699205898780102,-3819.9999999999955,139.84117975601998,80.39999999999999,0.699205898780102 22000.0,19.328125,0.0,22000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 22200.0,19.5,0.0,22200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 22400.0,19.625,0.0,22400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 22600.0,19.75,0.0,22600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 22800.0,19.890625,0.0,22800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 23000.0,20.0625,0.0,23000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 23200.0,20.453125,0.0,23200.0,0.0,81.7,0.2581988897471606,81.7,0.2581988897471606,-3559.9999999999977,51.63977794943222,81.7,0.2581988897471606 23400.0,20.59375,0.0,23400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 23600.0,20.734375,0.0,23600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 23800.0,20.859375,0.0,23800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 24000.0,20.984375,0.0,24000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 24200.0,21.109375,0.0,24200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 24400.0,21.25,0.0,24400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 24600.0,21.578125,0.0,24600.0,0.0,80.35000000000001,1.1559027256266454,80.35000000000001,1.1559027256266454,-3829.9999999999964,231.1805451253287,80.35000000000001,1.1559027256266454 24800.0,21.75,0.0,24800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 25000.0,22.0625,0.0,25000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 25200.0,22.34375,0.0,25200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 25400.0,22.515625,0.0,25400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 25600.0,22.671875,0.0,25600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 25800.0,22.8125,0.0,25800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 26000.0,23.125,0.0,26000.0,0.0,82.25,0.2635231383473657,82.25,0.2635231383473657,-3449.9999999999977,52.70462766947305,82.25,0.2635231383473657 26200.0,23.296875,0.0,26200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 26400.0,23.4375,0.0,26400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 26600.0,23.578125,0.0,26600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 26800.0,23.734375,0.0,26800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 27000.0,23.875,0.0,27000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 27200.0,24.015625,0.0,27200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 27400.0,24.1875,0.0,27400.0,0.0,95.1,0.3162277660168372,95.1,0.3162277660168372,-880.0,63.245553203367585,95.1,0.3162277660168372 27600.0,24.90625,0.0,27600.0,0.0,96.3,0.5374838498865714,96.3,0.5374838498865714,0.0,0.0,96.3,0.5374838498865714 27800.0,25.5,0.0,27800.0,0.0,99.35,0.24152294576982447,99.35,0.24152294576982447,0.0,0.0,99.35,0.24152294576982447 28000.0,26.015625,0.0,28000.0,0.0,99.89999999999998,0.21081851067789611,99.89999999999998,0.21081851067789611,0.0,0.0,99.89999999999998,0.21081851067789611 28200.0,26.640625,0.0,28200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 28400.0,27.390625,0.0,28400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 28600.0,27.921875,0.0,28600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 28800.0,28.4375,0.0,28800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 29000.0,29.15625,0.0,29000.0,0.0,78.75,0.35355339059327456,78.75,0.35355339059327456,-4149.999999999995,70.71067811865473,78.75,0.35355339059327456 29200.0,29.40625,0.0,29200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 29400.0,29.578125,0.0,29400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 29600.0,29.75,0.0,29600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 29800.0,29.9375,0.0,29800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 30000.0,30.109375,0.0,30000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 30200.0,30.265625,0.0,30200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 30400.0,30.609375,0.0,30400.0,0.0,83.44999999999999,0.28382310609877776,83.44999999999999,0.28382310609877776,-3209.9999999999977,56.764621219754794,83.44999999999999,0.28382310609877776 30600.0,30.8125,0.0,30600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 30800.0,31.03125,0.0,30800.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 31000.0,31.25,0.0,31000.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 31200.0,31.40625,0.0,31200.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 31400.0,31.5625,0.0,31400.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 31600.0,31.71875,0.0,31600.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0 31700.0,31.796875,0.0,31700.0,0.0,100.0,0.0,100.0,0.0,0.0,0.0,100.0,0.0
3.4.2. Evaluate clusterer¶
- create task
- set stream/learner
- set evaluation metric to F1
In [5]:
Copied!
import moa.tasks.EvaluateClustering;
EvaluateClustering eval = new EvaluateClustering();
eval.streamOption.setCurrentObject(clust_stream);
eval.learnerOption.setCurrentObject(clustree);
//eval.instanceLimitOption.setValueViaCLIString("-1");
eval.f1Option.setValueViaCLIString("f");
eval.prepareForUse();
eval.doTask();
import moa.tasks.EvaluateClustering;
EvaluateClustering eval = new EvaluateClustering();
eval.streamOption.setCurrentObject(clust_stream);
eval.learnerOption.setCurrentObject(clustree);
//eval.instanceLimitOption.setValueViaCLIString("-1");
eval.f1Option.setValueViaCLIString("f");
eval.prepareForUse();
eval.doTask();
Out[5]:
EvaluateClustering does not support custom output file (> [filename]). Check out the dump file to see the results (if you haven't specified, dumpClustering.csv by default).
Read/print the dumpClustering.csv file¶
In [10]:
Copied!
try (BufferedReader br = new BufferedReader(new FileReader("dumpClustering.csv"))) {
String line;
while ((line = br.readLine()) != null) {
System.out.println(line);
}
}
try (BufferedReader br = new BufferedReader(new FileReader("dumpClustering.csv"))) {
String line;
while ((line = br.readLine()) != null) {
System.out.println(line);
}
}
Nr;Event;F1-P;F1-R;Purity; 0;;0.0;0.0;0.0; 1;;0.0;0.0;0.0; 2;;0.0;0.0;0.0; 3;;0.0;0.0;0.0; 4;;0.0030257186081694403;0.0015128593040847202;1.0; 5;;0.0;0.0;0.0; 6;;0.0;0.0;0.0; 7;;0.0;0.0;0.0; 8;;0.0;0.0;0.0; 9;;0.0;0.0;0.0; 10;;0.0;0.0;0.0; 11;;0.0;0.0;0.0; 12;;0.0;0.0;0.0; 13;;0.0;0.0;0.0; 14;;0.0;0.0;0.0; 15;;0.014184397163120567;0.0070921985815602835;1.0; 16;;0.0;0.0;0.0; 17;;0.0;0.0;0.0; 18;;0.0;0.0;0.0; 19;;0.0;0.0;0.0; 20;;0.0;0.0;0.0; 21;;0.005249343832020998;0.002624671916010499;1.0; 22;;0.0;0.0;0.0; 23;;0.0021253985122210413;0.0010626992561105207;1.0; 24;;0.0;0.0;0.0; 25;;0.0;0.0;0.0; 26;;0.0;0.0;0.0; 27;;0.0;0.0;0.0; 28;;0.011049723756906079;0.005524861878453039;1.0; 29;;0.0;0.0;0.0; 30;;0.0;0.0;0.0;
4. Example of implementation from Prof. Alber Bifet¶
https://github.com/abifet/moa-notebooks/blob/master/MOA-Prequential-Evaluation.ipynb
Prequential Evaluation Example
Let’s run a very simple experiment: using a decision tree (Hoeffding Tree) with data generated from an artificial stream generator (RandomRBFGenerator).
We should start importing the classes that we need, and defining the stream and the learner.
In [1]:
Copied!
%maven nz.ac.waikato.cms.moa:moa:2018.6.0
import moa.classifiers.trees.HoeffdingTree;
import moa.streams.generators.RandomRBFGenerator;
HoeffdingTree learner = new HoeffdingTree();
RandomRBFGenerator stream = new RandomRBFGenerator();
%maven nz.ac.waikato.cms.moa:moa:2018.6.0
import moa.classifiers.trees.HoeffdingTree;
import moa.streams.generators.RandomRBFGenerator;
HoeffdingTree learner = new HoeffdingTree();
RandomRBFGenerator stream = new RandomRBFGenerator();
Now, we need to initialize the stream and the classifier:
In [2]:
Copied!
stream.prepareForUse();
learner.setModelContext(stream.getHeader());
learner.prepareForUse();
stream.prepareForUse();
learner.setModelContext(stream.getHeader());
learner.prepareForUse();
And finally, let’s run a prequential evaluation, as in Tutorial 2 (Introduction to the API of MOA).
Note[Pavol]: It seems the original example is not usable anymore.
In [4]:
Copied!
%maven org.knowm.xchart:xchart:3.5.2
import org.knowm.xchart.*;
import moa.core.TimingUtils;
import com.yahoo.labs.samoa.instances.Instance;
int numInstances = 1000000;
int sampleSize = 1000;
boolean isTesting = true;
double[] xData = new double[numInstances/sampleSize];
double[] yData = new double[numInstances/sampleSize];
int numberSamplesCorrect = 0;
int numberSamples = 0;
boolean preciseCPUTiming = TimingUtils.enablePreciseTiming();
long evaluateStartTime = TimingUtils.getNanoCPUTimeOfCurrentThread();
while (stream.hasMoreInstances() && numberSamples < numInstances) {
Instance trainInst = stream.nextInstance().getData();
if (isTesting) {
if (learner.correctlyClassifies(trainInst)){
numberSamplesCorrect++;
}
}
if (numberSamples % sampleSize == 0){
xData[numberSamples / sampleSize] = numberSamples / sampleSize;
yData[numberSamples / sampleSize] = 100.0 * (double) numberSamplesCorrect/ (double) numberSamples;
}
numberSamples++;
learner.trainOnInstance(trainInst);
}
double accuracy = 100.0 * (double) numberSamplesCorrect/ (double) numberSamples;
double time = TimingUtils.nanoTimeToSeconds(TimingUtils.getNanoCPUTimeOfCurrentThread()- evaluateStartTime);
System.out.println(numberSamples + " instances processed with " + accuracy + "% accuracy in "+time+" seconds.");
XYChart chart = QuickChart.getChart("Prequential Evaluation", "#Instances", "Accuracy", "y(x)", xData, yData);
BitmapEncoder.getBufferedImage(chart);
%maven org.knowm.xchart:xchart:3.5.2
import org.knowm.xchart.*;
import moa.core.TimingUtils;
import com.yahoo.labs.samoa.instances.Instance;
int numInstances = 1000000;
int sampleSize = 1000;
boolean isTesting = true;
double[] xData = new double[numInstances/sampleSize];
double[] yData = new double[numInstances/sampleSize];
int numberSamplesCorrect = 0;
int numberSamples = 0;
boolean preciseCPUTiming = TimingUtils.enablePreciseTiming();
long evaluateStartTime = TimingUtils.getNanoCPUTimeOfCurrentThread();
while (stream.hasMoreInstances() && numberSamples < numInstances) {
Instance trainInst = stream.nextInstance().getData();
if (isTesting) {
if (learner.correctlyClassifies(trainInst)){
numberSamplesCorrect++;
}
}
if (numberSamples % sampleSize == 0){
xData[numberSamples / sampleSize] = numberSamples / sampleSize;
yData[numberSamples / sampleSize] = 100.0 * (double) numberSamplesCorrect/ (double) numberSamples;
}
numberSamples++;
learner.trainOnInstance(trainInst);
}
double accuracy = 100.0 * (double) numberSamplesCorrect/ (double) numberSamples;
double time = TimingUtils.nanoTimeToSeconds(TimingUtils.getNanoCPUTimeOfCurrentThread()- evaluateStartTime);
System.out.println(numberSamples + " instances processed with " + accuracy + "% accuracy in "+time+" seconds.");
XYChart chart = QuickChart.getChart("Prequential Evaluation", "#Instances", "Accuracy", "y(x)", xData, yData);
BitmapEncoder.getBufferedImage(chart);
--------------------------------------------------------------------------- java.lang.NullPointerException: null at .(#41:1)
In [5]:
Copied!
import moa.DoTask;
DoTask.main("EvaluatePrequential -l trees.HoeffdingTree -i 1000000".split(" "));
import moa.DoTask;
DoTask.main("EvaluatePrequential -l trees.HoeffdingTree -i 1000000".split(" "));
{M}assive {O}nline {A}nalysis
Version: 18.06 June 2018
Copyright: (C) 2007-2018 University of Waikato, Hamilton, New Zealand
Web: http://moa.cms.waikato.ac.nz/
Can not access instrumentation environment...
Please check if jar file containing SizeOfAgent class is
specified in the java's "-javaagent" command line argument.
Task completed in 5.20s (CPU time)
learning evaluation instances,evaluation time (cpu seconds),model cost (RAM-Hours),classified instances,classifications correct (percent),Kappa Statistic (percent),Kappa Temporal Statistic (percent),Kappa M Statistic (percent),model training instances,model serialized size (bytes),tree size (nodes),tree size (leaves),active learning leaves,tree depth,active leaf byte size estimate,inactive leaf byte size estimate,byte size estimate overhead 100000.0,1.125,0.0,100000.0,92.10000000000001,84.09118369648397,82.93736501079914,82.63736263736264,100000.0,0.0,187.0,118.0,118.0,5.0,0.0,0.0,1.0 200000.0,1.71875,0.0,200000.0,93.2,86.13619960610498,85.15283842794761,84.29561200923789,200000.0,0.0,290.0,180.0,180.0,6.0,0.0,0.0,1.0 300000.0,2.0625,0.0,300000.0,93.7,87.0415165128104,86.76470588235296,85.14150943396228,300000.0,0.0,368.0,228.0,228.0,6.0,0.0,0.0,1.0 400000.0,2.40625,0.0,400000.0,95.1,90.00701548300785,90.18036072144288,88.57808857808857,400000.0,0.0,489.0,311.0,311.0,7.0,0.0,0.0,1.0 500000.0,2.890625,0.0,500000.0,94.8,89.19332313626387,88.9596602972399,87.12871287128712,500000.0,0.0,598.0,370.0,370.0,7.0,0.0,0.0,1.0 600000.0,3.28125,0.0,600000.0,95.6,91.0510537384223,91.6030534351145,89.69555035128805,600000.0,0.0,687.0,428.0,428.0,7.0,0.0,0.0,1.0 700000.0,3.640625,0.0,700000.0,96.1,91.99382498090834,91.82389937106917,90.8450704225352,700000.0,0.0,792.0,497.0,497.0,8.0,0.0,0.0,1.0 800000.0,4.078125,0.0,800000.0,95.39999999999999,90.62666835114945,90.49586776859503,89.37644341801385,800000.0,0.0,924.0,584.0,584.0,8.0,0.0,0.0,1.0 900000.0,4.546875,0.0,900000.0,96.8,93.50965438909621,93.73776908023483,92.72727272727272,900000.0,0.0,1020.0,647.0,647.0,8.0,0.0,0.0,1.0 1000000.0,5.078125,0.0,1000000.0,96.7,93.26222599719055,93.25153374233128,92.28971962616822,1000000.0,0.0,1124.0,720.0,720.0,9.0,0.0,0.0,?