Please wait a minute...
Tsinghua Science and Technology  2018, Vol. 23 Issue (06): 660-670    doi: 10.26599/TST.2018.9010011
A Latent Entity-Document Class Mixture of Experts Model for Cumulative Citation Recommendation
Lerong Ma, Lejian Liao, Dandan Song*, Jingang Wang
∙ Lerong Ma, Lejian Liao, Dandan Song, and Jingang Wang are with Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications, Beijing Institute of Technology, Beijing 100081, China. E-mail:;;
∙ Lerong Ma is also with College of Mathematics and Computer Science, Yan’an University, Yan’an 716000, China. E-mail:
Download: PDF (1794 KB)      HTML
Export: BibTeX | EndNote (RIS)       Supporting Info


Knowledge Bases (KBs) are valuable resources of human knowledge which contribute to manyapplications. However, since they are manually maintained, there is a big lag between their contents and the up-to-date information of entities. Considering a target entity in KBs, this paper investigates how Cumulative Citation Recommendation (CCR) can be used to effectively detect its worthy-citation documents in large volumes of stream data. Most global relevant models only consider semantic and temporal features of entity-document instances, which does not sufficiently exploit prior knowledge underlying entity-document instances. To tackle this problem, we present a Mixture of Experts (ME) model by introducing a latent layer to capture relationships between the entity-document instances and their latent class information. An extensive set of experiments was conducted on TREC-KBA-2013 dataset. The results show that the model can significantly achieve a better performance gain compared to state-of-the-art models in CCR.

Key wordsknowledge base acceleration      cumulative citation recommendation      Mixture of Experts (ME)      Latent Entity-Document Classes (LEDCs)     
Received: 17 March 2017      Published: 28 December 2018
Corresponding Authors: Dandan Song   
About author:

Jingang Wang received the BS and PhD degrees in computer science from Beijing Institute of Technology (BIT), China, in 2010 and 2016, respectively. Currently, he is a senior algorithm engineer at Alibaba Group. His research interests include information retrieval, knowledge mining, and natural language processing.

Cite this article:

Lerong Ma, Lejian Liao, Dandan Song, Jingang Wang. A Latent Entity-Document Class Mixture of Experts Model for Cumulative Citation Recommendation. Tsinghua Science and Technology, 2018, 23(06): 660-670.

URL:     OR

N?(erel)# Entity e’s related entities found in its profile page
N?(d,e)# Occurrences of e in document d
N?(d,erel)# Occurrences of the related entities in document d
FPOS?(d,e)First occurrence position of e in d
FPOSn?(d,e)FPOS?(d,e) normalized by the document length
LPOS?(d,e)Last occurrence position of e in d
LPOSn?(d,e)LPOS?(d,e) normalized by the document length
Spreadn?(d,e)Spread?(d,e) normalized by document length
Source?(d)The source of d
weekday?(d)Weekday of d published
burst?(d)Burst weights of d
Table 1 The features of entity-document pairs.
Rating level
Training set1 6962 1211 0301 7026 549
Test set5 63011 5793 37910 54331 131
Table 2 Detailed annotation of the dataset.
MethodVital OnlyVital + Useful
Official Baseline0.1710.9420.2900.5400.9720.694
Table 3 Overall results of evaluated models on the TREC-KBA-2013 dataset.
[1]   Mihalcea R. and Csomai A., Wikify!: Linking documents to encyclopedic knowledge, in Proc. 16th ACM Conf. Conf. Information and Knowledge Management, Lisbon, Portugal, 2007, pp. 233–242.
[2]   Xu Y., Jones G. J. F., and Wang B., Query dependent pseudo-relevance feedback based on Wikipedia, in Proc. 32nd Int. ACM SIGIR Conf. Research and Development in Information Retrieval, Boston, MA, USA, 2009, pp. 59–66.
[3]   Dalton J., Dietz L., and Allan J., Entity query feature expansion using knowledge base links, in Proc. 37th Int. ACM SIGIR Conf. Research & Development in Information Retrieval, Queensland, Australia, 2014, pp. 365–374.
[4]   Zhang C. H., Zhou M., Han X., Hu Z., and Ji Y., Knowledge graph embedding for hyper-relational data, Tsinghua Sci. Technol., vol. 22, no. 2, pp. 185–197, 2017.
[5]   Dang H. T., Kelly D., and Lin J., Overview of the TREC 2007 question answering track, in Proc. 16th Text Retrieval Conf., Gaithersburg, MD, USA, 2007, pp. 1–18.
[6]   Balog K., Serdyukov P., and de Vries A. P., Overview of the TREC 2010 entity track, in Proc. 19th Text Retrieval Conf., Gaithersburg, MD, USA, 2010.
[7]   Frank J. R.,Weiner M. K.,Roberts D. A.,Niu F., Zhang C., Ré C., and Soboroff I., Building An Entity-centric Stream Filtering Test Collection for TREC 2012. Massachusetts Institute of Technology, 2012.
[8]   Wang J. G., Song D. D., Lin C. Y., and Liao L. J., Bit and MSRA at TREC KBA CCR track 2013, in Proc. 22nd Text Retrieval Conf., Gaithersburg, MD, USA, 2013.
[9]   Liu X. T., Darko J., and Fang H., A related entity based approach for knowledge base acceleration, in Proc. 22nd Text Retrieval Conf., Gaithersburg, MD, USA, 2013.
[10]   Frank J. R.,Bauer S. J.,Weiner M. K., Roberts D. A., Tripuraneni N., Zhang C., and Ré C., Evaluating stream filtering for entity profile updates for TREC 2013, in Proc. 22nd Text Retrieval Conf., Gaithersburg, MD, USA, 2013.
[11]   Ma L. R., Song D. D., Liao L.J., and Wang J. G., PSVM: A preference enhanced SVM model using preference data for classification, Science China Information Sciences, vol. 60, no. 12, pp. 1–14, 2017.
[12]   Jacobs R. A., Jordan M. I., Nowlan S. J., and Hinton G. E., Adaptive mixtures of local experts, Neural Computat., vol. 3, no. 1, pp. 79–87, 1991.
[13]   Bishop C. M., Pattern Recognition and Machine Learning. Springer, 2006.
[14]   Balog K., Ramampiaro H., Takhirov N., and N?rv?g K., Multi-step classification approaches to cumulative citation recommendation, in Proc. 10th Conference on Open Research Areas in Information Retrieval, Paris, France, 2013, pp. 121–128.
[15]   Balog K. and Ramampiaro H., Cumulative citation recommendation: Classification vs. ranking, in Proc. 36th Int. ACM SIGIR Conf. Research and Development in Information Retrieval, Dublin, Ireland, 2013, pp. 941–944.
[16]   Berendsen R., Meij E., Odijk D., de Rijke M., and Weerkamp W., The university of Amsterdam at TREC 2012, in Proc. 21st Text Retrieval Conf., Gaithersburg, MD, USA, 2012.
[17]   Bonnefoy L., Bouvier V., and Bellot P., A weakly-supervised detection of entity central documents in a stream, in Proc. 36th Int. ACM SIGIR Conf. Research and Development in Information Retrieval, Dublin, Ireland, 2013, pp. 769–772.
[18]   Gebremeskel G. G., He J. Y., de Vries A. P., and Lin J., Cumulative citation recommendation: A feature-aware comparison of approaches, in Proc. 25th Int. Workshop on Database and Expert Systems Applications, Munich, Germany, 2014, pp. 193–197.
[19]   Wang J. G.,Liao L. J., Song D. D., Ma L. R., Lin C. Y., and Rui Y., Resorting relevance evidences to cumulative citation recommendation for knowledge base acceleration, in Proc. 16th Int. Conf. on Web-Age Information Management, Qingdao, China, 2015, pp. 169–180.
[20]   Wang J. G.,Song D. D.,Wang Q. F., Zhang Z. W., Si L., Liao L. J., and Lin C. Y., An entity class-dependent discriminative mixture model for cumulative citation recommendation, in Proc. 38th Int. ACM SIGIR Conf. Research and Development in Information Retrieval, Santiago, Chile, 2015, pp. 635–644.
[21]   Chamroukhi F., Robust mixture of experts modeling using the t distribution, Neural Netw., vol. 79, pp. 20–36, 2016.
[22]   Waterhouse S. R. and Robinson A. J., Classification using hierarchical mixtures of experts, in Proc. 1994 IEEE Workshop on Neural Networks for Signal Processing, Ermioni, Greece, 1994, pp. 177–186.
[23]   Yuksel S. E., Wilson J. N., and Gader P. D., Twenty years of mixture of experts, IEEE Trans. Neural Netw. Learn. Syst., vol. 23, no. 8, pp. 1177–1193, 2012.
[24]   Yao B. P., Walther D. B., Beck D. M., and Li F. F., Hierarchical mixture of classification experts uncovers interactions between brain regions, in Proc 22nd Int. Conf. on Neural Information Processing Systems, Vancouver, Canada, 2009, pp. 2178–2186.
[25]   Yuksel S. E. and Gader P. D., Variational mixture of experts for classification with applications to landmine detection, in Proc. 20th Int. Conf. Pattern Recognition, Istanbul, Turkey, 2010, pp. 2981–2984.
[26]   Dempster A. P., Laird N. M., and Rubin D. B., Maximum likelihood from incomplete data via the EM algorithm, J. Roy. Stat. Soc. Ser. B (Methodological), vol. 39, no. 1, pp. 1–38, 1977.
[27]   Schmidt M., MinFunc, , 2005.
[28]   Rehurek R. and Sojka P., Gensim, , 2010.
[29]   Tu N. C., A Java Implementation of Latent Dirichlet Allocation (LDA), , 2018.
[30]   TREC, KBA Stream Corpus 2013, , 2013.
No related articles found!