Tsinghua Science and Technology  2021, Vol. 26 Issue (4): 426-439    doi: 10.26599/TST.2019.9010080
Modeling and Analyzing the Performance of High-Speed Packet I/O
Xuesong Li*(),Fengyuan Ren(),Bailong Yang()
Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China.
Xi’an Research Institute of Hi-Tech, Xi’an 710025, China.

Abstract

Recently, 10 Gbps or higher speed links are being widely deployed in data centers. Novel high-speed packet I/O frameworks have emerged to keep pace with such high-speed links. These frameworks mainly use techniques, such as memory preallocation, busy polling, zero copy, and batch processing, to replace costly operations (e.g., interrupts, packet copy, and system call) in native OS kernel stack. For high-speed packet I/O frameworks, costs per packet, saturation throughput, and latency are performance metrics that are of utmost concern, and various factors have an effect on these metrics. To acquire a comprehensive understanding of high-speed packet I/O, we propose an analytical model to formulate its packet forwarding (receiving-processing-sending) flow. Our model takes the four main techniques adopted by the frameworks into consideration, and the concerned performance metrics are derived from it. The validity and correctness of our model are verified by real system experiments. Moreover, we explore how each factor impacts the three metrics through a model analysis and then provide several useful insights and suggestions for performance tuning.

Received: 15 November 2019      Published: 12 January 2021
Fund:  National Key Research and Development Program of China(2018YFB1700103);National Natural Science Foundation of China(61872208)
Corresponding Authors: Xuesong Li     E-mail: lixs16@mails.tsinghua.edu.cn;renfy@tsinghua.edu.cn;xa_403@163.com
About author: Xuesong Li received the BEng degree from Tsinghua University in 2013, and the MS degree from Xi’an Research Institute of Hi-Tech in 2015. He currently is a PhD candidate jointly trained by Xi’an Research Institute of Hi-Tech and Tsinghua University, under the advising of Prof. Bailong Yang and Prof. Fengyuan Ren. His research interests include high-speed packet I/O and energy-aware data center network.|Fengyuan Ren received the BS and MS degrees from Northwestern Polytechnic University, Xi’an, China in 1993 and 1996, respectively. In Dec. 1999, he obtained the PhD degree from Northwestern Polytechnic University. He is now a professor at the Department of Computer Science and Technology, Tsinghua University, Beijing, China. From 2000 to 2001, he worked at the Electronic Engineering Department of Tsinghua University as a post doctoral researcher. In Jan. 2002, he moved to the Computer Science and Technology Department of Tsinghua University. His research interests include network traffic management and control, control in/over computer networks, and wireless networks and wireless sensor networks. He authored / co-authored more than 80 international journal and conference papers. He is a member of the IEEE, and has served as a technical program committee member and local arrangement chair for various IEEE and ACM international conferences.|Bailong Yang received the BS and MS degrees from Xi’an Research Institute of Hi-Tech, Xi’an, China in 1990 and 1993, respectively. He received the PhD degree from Xi’an Research Institute of Hi-Tech, Xi’an, China in 2001. He is currently a professor of Xi’an Research Institute of Hi-Tech. His research interests include complex network and computer simulation.
 Cite this article: Xuesong Li,Fengyuan Ren,Bailong Yang. Modeling and Analyzing the Performance of High-Speed Packet I/O. Tsinghua Science and Technology, 2021, 26(4): 426-439. URL:
 Fig. 1 Packet forwarding flow of high-speed packet I/O. Table 1 Values of some used parameters. Fig. 2 CDF of batch size distribution. Fig. 3 Validation of analytical model ( "Model" for results of analytical model, "Expt." for results of DPDK experiment). 𝝍."> Fig. 4 Impact of max arriving-batch size $𝝍$. Fig. 5 Impact of bursty degree v. Fig. 6 Impact of frequency F on CPP. Fig. 7 Impact of max batch processing size. 𝝍?=10, 20, and 30."> Fig. 8 Evolution of latency with utilization when $𝝍?=$10, 20, and 30. Fig. 9 Evolution of latency with utilization when v=0.2, 0.7, and 1.2. Fig. 10 Evolution of normalized latency with utilization when service capabilities varying. Fig. 11 Evolution of normalized latency with utilization when B=1, 8, 32, and 128. C𝐭𝐚𝐬𝐤 on latency."> Fig. 12 Interaction of B and $C𝐭𝐚𝐬𝐤$ on latency.
 [1] Rizzo L., Netmap: A novel framework for fast packet I/O, in Proceedings of USENIX Annual Technical Conference, Boston, MA, USA, 2012, pp. 101-112. [2] Jeong E., Woo S., Jamshed M. A., Jeong H., Ihm S., Han D., and Park K., mTCP: A highly scalable user-level TCP stack for multicore systems, in Proceedings of USENIX Symposium on Networked Systems Design and Implementation, Seattle, WA, USA, 2014, pp. 489-502. [3] Intel, Intel Data Plane Development Kit (DPDK), , 2012. [4] ntop, PF_RING ZC (Zero Copy), , 2014. [5] Han S., Jang K., Park K., and Moon S., Packetshader: A GPU-accelerated software router, in Proc. of ACM SIGCOMM Computer Communication Review, vol. 40, no. 4, pp. 195-206, 2010. [6] Gallenmüller S., Emmerich P., Sch?nberger R., Raumer D., and Carle G., Building fast but flexible software routers, in Proceedings of ACM/IEEE Symposium on Architectures for Networking and Communications Systems, Beijing, China, 2017, pp. 101-102. [7] Pfaff B., Pettit J., Koponen T., Jackson E., Zhou A., Rajahalme J., Gross J., Wang A., Stringer J., Shelar P., et al., The design and implementation of open vSwitch, in Proceedings of USENIX Symposium on Networked Systems Design and Implementation, Oakland, CA, USA, 2015, pp. 117-130. [8] Nicira, Open vSwitch with DPDK, , 2018. [9] Martins J., Ahmed M., Raiciu C., Olteanu V., Honda M., Bifulco R., and Huici F., Clickos and the art of network function virtualization, in Proceedings of USENIX Symposium on Networked Systems Design and Implementation, Seattle, WA, USA, 2014, pp. 459-473. [10] Hwang J., Ramakrishnan K., and Wood T., Netvm: High performance and flexible networking using virtualization on commodity platforms, IEEE Transactions on Network and Service Management, vol. 12, no. 1, pp. 34-47, 2015. [11] Steve P. and David R., Introduction to OpenOnload-building application transparency and protocol conformance into application acceleration middleware, , 2011. [12] Snabb Switch: Fast open source packet processing, , 2020. [13] Rizzo L. and Lettieri G., Vale, a switched ethernet for virtual machines, in Proceedings of the 8th International Conference on Emerging Networking Experiments and Technologies, Nice, France, 2012, pp. 61-72. [14] Fd.io, , 2018. [15] Cisco, Trex: Realistic traffic generator, , 2015. [16] Lim H., Han D., Andersen D. G., and Kaminsky M., MICA: A holistic approach to fast in-memory key-value storage, in Proceedings of USENIX Symposium on Networked Systems Design and Implementation, Seattle, WA, USA, 2014, pp. 429-444. [17] Gallenmüller S., Emmerich P., Wohlfart F., Raumer D., and Carle G., Comparison of frameworks for high-performance packet IO, in Proceedings of ACM/IEEE Symposium on Architectures for Networking and Communications Systems, Oakland, CA, USA, 2015, pp. 29-38. [18] Emmerich P., Raumer D., Wohlfart F., and Carle G., Assessing soft-and hardware bottlenecks in PC-based packet forwarding systems, in Proceedings of the 14th International Conference on Networks, Barcelona, Spain, 2015, pp. 78-83. [19] Emmerich P., Raumer D., Gallenmüller S., Wohlfart F., and Carle G., Throughput and latency of virtual switching with open vswitch: A quantitative analysis, Journal of Network and Systems Management, vol. 26, no. 2, pp. 314-338, 2018. [20] Jarschel M., Oechsner S., Schlosser D., Pries R., Goll S., and TranGia P., Modeling and performance evaluation of an openflow architecture, in Proceedings of the 23rd International Teletraffic Congress, San Francisco, CA, USA, 2011, pp. 1-7. [21] Bolla R., Bruschi R., Carrega A., and Davoli F., Green networking with packet processing engines: Modeling and optimization, IEEE/ACM Transactions on Networking, vol. 22, no. 1, pp. 110-123, 2014. [22] Su Z., Baynat B., and Begin T., A new model for DPDK-based virtual switches, in Proceedings of 2017 IEEE Conference on Network Softwarization, Bologna, Italy, 2017, pp. 1-5. [23] Zabala L., Ferro A., and Pineda A., Modelling packet capturing in a traffic monitoring system based on linux, in Proceedings of 2012 International Symposium on Performance Evaluation of Computer & Telecommunication Systems, Genoa, Italy, 2012, pp. 1-6. [24] Salah K., El-Badawi K., and Haidari F., Performance analysis and comparison of interrupt-handling schemes in gigabit networks, Computer Communications, vol. 30, no. 17, pp. 3425-3441, 2007. [25] Uyeda F., Foschini L., Baker F., Suri S., and Varghese G., Efficiently measuring bandwidth at all time scales, in Proceedings of USENIX Symposium on Networked Systems Design and Implementation, Boston, MA, USA, 2011, pp. 71-84. [26] Benson T., Anand A., Akella A., and Zhang M., Understanding data center traffic characteristics, ACM SIGCOMM Computer Communication Review, vol. 40, no. 1, pp. 92-99, 2010. [27] Salvador P., Pacheco A., and Valadas R., Modeling IP traffic: Joint characterization of packet arrivals and packet sizes using BMAPs, Computer Networks, vol. 44, no. 3, pp. 335-352, 2004. [28] Klemm A., Lindemann C., and Lohmann M., Modeling IP traffic using the batch Markovian arrival process, Performance Evaluation, vol. 54, no. 2, pp. 149-173, 2003. [29] Erramilli A., Narayan O., and Willinger W., Experimental queueing analysis with long-range dependent packet traffic, IEEE/ACM Transactions on Networking, vol. 4, no. 2, pp. 209-223, 1996. [30] Willinger W., Paxson V., and Taqqu M., Self-similarity and heavy tails: Structural modeling of network traffic, A Practical Guide to Heavy Tails: Statistical Techniques and Applications, vol. 23, pp. 27-53, 1998. [31] Chaudhry M. and Templeton J. G., A First Course in Bulk Queues. New York, NY, USA: Wiley, 1983. [32] Nelson R., Probability, Stochastic Processes, and Queueing Theory: The Mathematics of Computer Performance Modeling. Berlin, Germany: Springer Science & Business Media, 2013. [33] Emmerich P., Gallenmüller S., Raumer D., Wohlfart F., and Carle G., MoonGen: A scriptable high-speed packet generator, in Proceedings of the 2015 Internet Measurement Conference, Tokyo, Japan, 2015, pp. 275-287. [34] CAIDA, The CAIDA Anonymized Internet Traces 2016 Dataset-20160406, , 2016.
 [1] Kuai Xu, Feng Wang, Haiyan Wang, Bo Yang. Detecting Fake News Over Online Social Media via Domain Reputations and Content Understanding[J]. Tsinghua Science and Technology, 2020, 25(1): 20-27. [2] Zebang Shen, Binbin Yong, Gaofeng Zhang, Rui Zhou, Qingguo Zhou. A Deep Learning Method for Chinese Singer Identification[J]. Tsinghua Science and Technology, 2019, 24(04): 371-378. [3] Li Li, Shuofeng Wang, Meng Li, Jiyuan Tan. Comparison of Travel Mode Choice Between Taxi and Subway Regarding Traveling Convenience[J]. Tsinghua Science and Technology, 2018, 23(2): 135-144. [4] Darpan Triboan,Liming Chen,Feng Chen,Zumin Wang. Towards a Service-Oriented Architecture for a Mobile Assistive System with Real-time Environmental Sensing[J]. Tsinghua Science and Technology, 2016, 21(6): 581-597. [5] Yang Pu,David C. Samuels,Layne T. Watson,Yang Cao. Randomness in the Hybrid Modeling and Simulation of Insulin Secretion Pathways in Pancreatic Islets[J]. Tsinghua Science and Technology, 2015, 20(5): 441-452. [6] Yang Li,Xiaoming Tao,Jianhua Lu. Effectively Lossless Subspace Appearance Model Compression Using Prior Information[J]. Tsinghua Science and Technology, 2015, 20(4): 409-416. [7] Gong Jibing, Tang Jie, C. M. Fong A.. ACTPred: Activity Prediction in Mobile Social Networks[J]. Tsinghua Science and Technology, 2014, 19(3): 265-274. [8] . Development of a BGA Package Based on Si Interposer with Through Silicon Via[J]. Tsinghua Science and Technology, 2011, 16(4): 408-413. [9] . Modeling of a Schottky Diode in CMOS Process with a Flexible “Open-Through” On-Chip De-embedding Method[J]. Tsinghua Science and Technology, 2011, 16(2): 175-180. [10] . Modeling and Experimental Investigation of a Variable Speed Drive Water Source Heat Pump[J]. Tsinghua Science and Technology, 2010, 15(4): 434-440. [11] . Modeling and Experimental Study of PEM Fuel Cell Transient Response for Automotive Applications[J]. Tsinghua Science and Technology, 2009, 14(5): 639-645. [12] . Spatially Adaptive Subsampling for Motion Detection[J]. Tsinghua Science and Technology, 2009, 14(4): 423-433. [13] . Plausibility in Early Stages of Architectural Design: A New Tool for High-Rise Residential Buildings[J]. Tsinghua Science and Technology, 2009, 14(3): 327-332. [14] . Examining Antecedents and Consequences of Tourist Satisfaction: A Structural Modeling Approach[J]. Tsinghua Science and Technology, 2009, 14(3): 397-406. [15] . Empirical Study of Consumers' Purchase Intentions in C2C Electronic Commerce[J]. Tsinghua Science and Technology, 2008, 13(3): 287-292.