1. NJUST @ CLSciSumm-2018 What to submit Task Framework Task 1A Task 1B
Shutian Ma; Heng Zhang; Jin Xu; Chengzhi Zhang
Department of Information Management, Nanjing University of Science and Technology, Nanjing, China,
What to submit
Task Framework
Voting Weights of Precision, Recall and F1-Oriented 3-Classifiers System
Voting System
Classifiers
Voting Weight
Precision- oriented
SVM (RBF)
0.3116
Recall- oriented
0.2192
F1- oriented
0.2565 DT
0.2617
0.5236
0.4233 LG
0.4268
0.2572
0.3202
For Task 1A, Multi-classifiers using Voting System, another 10 running results based on single classifiers.
New Classifier: XGBOOST, an efficient and scalable implementation of gradient boosting framework
Task 1A
Feature Selection
Decision tree, Logistic regression, SVM (Linear, RBF), XGBOOST
Multi-Classifiers + Voting systems
Average F1 when using Precision-Oriented 3-Classifiers Voting System
Average F1 when using Precision-Oriented 4-Classifiers Voting System
Task 1B
Examples when Utilizing Rules to Expand Labeled Citation Text
Manual Dictionary
LLDA
XGBOOST
Manual dictionary + LLDA
POS Dictionary + LLDA
Average F1 when using Recall-Oriented 3-Classifiers Voting System
Average F1 when using Recall-Oriented 4-Classifiers Voting System
LLDA
Assume each identified facet is a topic label and that each citation sentence is a mixture of the expert-assigned topics that can be learned.
+LLDA strategy
Use dictionary-labeled testing data to be testing data for prediction.
Dictionary strategy
Find the best order of judging facets
POS dictionary
POS results are VB and JJ, meet frequency threshold.
Fixed Features + Selected Features
Iteratively evaluate a candidate subset of selected features set
Task 2
Average F1 when using F1-Oriented 3-Classifiers Voting System
Average F1 when using F1-Oriented 4-Classifiers Voting System
Group sentences into clusters based on its similarity with different parts of abstract.
Extract sentence based on score from each cluster and combine into summary.
Average F1 when #Negative/#Positive is 1
Average F1 when #Negative/#Positive is 2
Average F1 when #Negative/#Positive is 3
For Task 1B, we select the specific order according to F1.
Manual Dictionary, we choose top 3.
LLDA strategy, we pick the top 4.
XGBOOST strategy, we also select top 4.
People write summaries starting with some fixed phrases, such as “this paper”, “in this paper” or “we”.
Meanwhile, the last sentence are usually about results or conclusions.
Average F1 when #Negative/#Positive is 4
Average F1 when #Negative/#Positive is 5
Average F1 when #Negative/#Positive is 6
Acknowledgement
𝑆𝑐𝑜𝑟𝑒 𝑖 =2.5 𝑆 𝐽𝑎𝑐𝑐𝑎𝑟𝑑 +2.5 𝑆 𝐼𝐷𝐹 +2.5 𝑆 𝑇𝐹𝐼𝐷𝐹 𝑆 𝐿𝑒𝑛𝑔𝑡ℎ 𝑆 𝑃𝑜𝑠𝑖𝑡𝑖𝑜𝑛
Deepest gratitude goes to my greatest teammate.
Voting Weights of Precision, Recall and F1-Oriented 4-Classifiers System
Voting System
Classifiers
Voting Weight
Precision-oriented
SVM (Linear)
0.2160
Recall- oriented
0.0699
F1- oriented
0.0954
SVM (RBF)
0.2443
0.2039
0.2320 DT
0.2051
0.4870
0.3829 LG
0.3346
0.2392
0.2897
Tools
Contact Information
Shutian Ma:
Heng Zhang:
Jin Xu:
Chengzhi Zhang:
Porter Stemmer
Gensim, Word2Vec and Doc2Vec model
Scikit-learn, Classifiers
LDA and XGBOOST are applied with Python package