Eric Dumazet b193722731
net: reorganize sk_buff for faster __copy_skb_header() ...
With proliferation of bit fields in sk_buff, __copy_skb_header() became
quite expensive, showing as the most expensive function in a GSO
workload.
__copy_skb_header() performance is also critical for non GSO TCP
operations, as it is used from skb_clone()
This patch carefully moves all the fields that were not copied in a
separate zone : cloned, nohdr, fclone, peeked, head_frag, xmit_more
Then I moved all other fields and all other copied fields in a section
delimited by headers_start[0]/headers_end[0] section so that we
can use a single memcpy() call, inlined by compiler using long
word load/stores.
I also tried to make all copies in the natural orders of sk_buff,
to help hardware prefetching.
I made sure sk_buff size did not change.
Signed-off-by: Eric Dumazet <edumazet@google.com >
Signed-off-by: David S. Miller <davem@davemloft.net >
2014-09-29 12:27:20 -04:00
2014-09-09 15:33:05 -04:00
2014-08-06 18:01:23 -07:00
2014-07-23 10:22:35 +09:00
2014-09-02 10:09:43 +02:00
2014-08-25 21:47:22 +01:00
2014-09-23 12:09:27 -04:00
2014-08-19 11:55:38 -07:00
2014-08-24 19:33:10 +02:00
2014-08-25 16:15:32 -07:00
2014-08-28 09:59:42 +01:00
2014-08-30 19:28:04 +01:00
2014-08-07 14:40:09 -04:00
2014-02-11 23:30:50 +05:30
2012-10-13 10:46:48 +01:00
2013-07-30 11:53:12 -04:00
2012-02-09 09:04:23 -08:00
2005-04-16 15:20:36 -07:00
2013-11-09 00:16:28 -05:00
2007-02-09 17:08:57 +00:00
2012-10-08 13:50:20 +10:30
2013-09-24 10:35:17 +01:00
2013-12-02 11:24:18 +00:00
2013-03-12 13:59:14 -07:00
2013-01-03 15:57:16 -08:00
2013-10-27 08:40:39 -04:00
2012-10-13 10:46:48 +01:00
2013-10-17 13:31:19 -07:00
2012-09-11 08:56:37 +02:00
2012-03-16 10:38:24 -04:00
2014-04-10 17:51:29 -04:00
2009-04-01 08:59:23 -07:00
2013-12-24 09:35:17 -08:00
2014-04-03 16:20:49 -07:00
2014-02-06 10:33:47 +01:00
2012-09-10 11:13:16 -07:00
2011-03-11 14:25:50 +00:00
2014-04-07 16:36:05 -07:00
2012-03-04 17:54:34 -05:00
2014-08-06 18:01:27 -07:00
2014-04-18 11:40:30 +02:00
2008-06-06 11:29:10 -07:00
2014-08-15 12:38:41 -06:00
2014-07-01 10:34:38 -06:00
2013-11-08 09:04:30 -07:00
2014-06-04 16:54:04 -07:00
2014-01-13 17:32:27 +01:00
2014-09-26 15:05:15 -04:00
2014-09-19 16:27:07 -04:00
2009-11-20 20:13:39 +01:00
2009-11-20 20:13:39 +01:00
2013-02-20 09:37:28 -05:00
2014-06-04 16:54:02 -07:00
2014-01-23 16:36:56 -08:00
2009-07-31 12:28:46 +02:00
2014-09-24 14:23:34 +08:00
2011-01-13 08:03:17 -08:00
2014-07-15 11:05:10 -04:00
2010-03-24 16:31:22 -07:00
2014-02-24 17:13:55 -08:00
2013-12-04 17:19:44 +01:00
2014-07-23 15:01:52 -07:00
2014-01-21 20:18:26 -08:00
2013-02-13 06:00:53 -08:00
2013-07-30 11:11:44 +02:00
2014-06-04 16:54:06 -07:00
2014-04-02 12:51:41 -07:00
2014-04-09 13:44:35 -07:00
2013-02-21 17:22:15 -08:00
2014-02-13 12:34:05 +01:00
2013-12-05 21:28:41 +08:00
2014-04-15 09:15:39 -07:00
2013-11-15 09:32:21 +09:00
2014-07-03 11:32:43 +01:00
2012-01-03 22:54:57 -05:00
2014-05-27 13:56:21 -07:00
2014-05-28 13:37:21 -07:00
2013-05-20 12:27:59 -07:00
2008-06-04 14:56:12 +01:00
2013-12-29 15:25:48 +01:00
2013-12-02 20:43:15 +01:00
2014-06-25 13:26:47 -07:00
2013-12-04 09:33:34 -04:00
2011-09-23 12:05:29 +05:30
2013-09-13 15:09:52 +02:00
2014-06-06 16:08:18 -07:00
2014-07-21 13:43:19 +02:00
2011-08-06 22:12:37 -07:00
2012-05-07 15:02:26 +02:00
2013-04-29 18:28:19 -07:00
2014-09-14 17:28:32 -07:00
2013-08-18 17:39:55 -07:00
2014-05-24 22:33:41 +09:00
2014-08-05 15:59:35 -07:00
2012-10-22 16:50:13 -07:00
2012-02-20 19:46:36 +11:00
2012-10-13 10:46:48 +01:00
2012-11-19 14:32:13 +01:00
2013-11-23 22:33:51 -08:00
2010-07-28 09:59:01 -04:00
2011-10-14 16:48:08 +02:00
2014-07-10 18:35:01 +02:00
2014-07-10 18:35:23 +02:00
2007-07-16 09:05:42 -07:00
2014-09-29 00:04:55 -04:00
2013-03-03 23:59:59 -08:00
2012-10-13 10:46:48 +01:00
2014-06-22 16:34:11 -06:00
2008-01-28 23:21:18 +01:00
2012-10-13 10:46:48 +01:00
2014-09-05 17:47:02 -07:00
2014-09-05 12:12:20 -07:00
2013-03-27 13:48:25 +01:00
2012-03-31 16:03:16 -04:00
2014-04-29 09:45:56 +09:00
2014-08-22 13:56:47 -07:00
2013-09-26 15:06:58 -07:00
2012-10-13 10:46:48 +01:00
2014-04-01 23:19:14 -04:00
2011-01-09 15:42:55 -08:00
2014-07-08 13:41:08 -07:00
2014-09-26 15:05:15 -04:00
2009-06-18 19:50:21 +08:00
2014-05-26 14:33:10 +02:00
2013-02-23 17:50:12 -08:00
2014-07-08 15:28:16 -07:00
2012-05-14 09:47:23 -03:00
2012-10-13 10:46:48 +01:00
2014-01-21 23:17:20 -08:00
2013-08-22 22:13:54 -07:00
2009-12-17 10:58:17 -05:00
2013-09-12 10:12:47 -07:00
2014-08-11 11:44:11 -07:00
2014-08-06 18:01:12 -07:00
2013-06-29 12:57:13 +04:00
2014-08-22 13:18:48 -04:00
2009-06-18 13:04:05 -07:00
2014-01-23 16:36:52 -08:00
2012-11-08 16:58:30 +01:00
2014-05-20 10:30:40 +09:00
2007-07-10 00:35:17 -04:00
2010-10-25 14:11:37 -07:00
2014-02-19 17:22:44 +01:00
2014-09-13 11:24:03 -07:00
2012-10-13 10:46:48 +01:00
2014-08-08 17:39:48 -07:00
2013-08-26 21:40:24 +02:00
2014-08-06 18:01:22 -07:00
2006-04-26 12:56:16 +01:00
2010-02-09 11:13:56 +01:00
2010-10-12 16:53:37 +02:00
2014-02-08 10:36:58 -05:00
2014-07-15 04:49:40 -04:00
2013-10-13 16:16:28 -07:00
2012-03-16 10:38:24 -04:00
2014-05-03 19:24:26 -04:00
2012-03-26 21:47:19 +02:00
2012-03-26 21:47:19 +02:00
2012-10-13 10:46:48 +01:00
2007-05-01 23:26:34 +02:00
2012-11-28 11:36:32 +01:00
2009-03-28 21:34:45 +01:00
2006-11-30 12:24:45 +00:00
2014-01-29 20:40:08 +01:00
2014-08-19 10:19:39 -05:00
2013-04-29 13:54:36 -04:00
2014-09-11 12:21:18 +02:00
2014-07-08 16:12:53 -07:00
2012-10-13 10:46:48 +01:00
2013-06-05 12:00:27 -07:00
2012-10-13 10:46:48 +01:00
2014-06-02 14:56:01 -07:00
2012-10-13 10:46:48 +01:00
2014-01-04 20:10:24 -05:00
2014-09-04 22:26:14 -07:00
2013-02-11 14:16:26 -05:00
2012-10-13 10:46:48 +01:00
2013-12-11 14:47:40 -05:00
2014-05-27 14:26:31 -07:00
2005-04-16 15:20:36 -07:00
2012-10-13 10:46:48 +01:00
2014-05-14 16:40:04 -07:00
2012-11-10 00:40:24 -08:00
2013-03-27 13:48:25 +01:00
2014-07-04 12:35:59 +02:00
2012-10-09 16:22:40 +09:00
2012-12-09 00:20:28 -05:00
2013-03-15 12:23:09 -07:00
2014-01-27 21:02:39 -08:00
2014-07-17 13:30:00 +00:00
2013-06-06 10:23:58 +09:00
2013-11-19 19:06:41 +01:00
2007-09-16 21:13:58 -07:00
2014-08-28 22:22:29 -04:00
2013-06-12 23:02:35 -04:00
2010-12-09 20:17:07 -08:00
2014-09-12 13:59:03 -07:00
2012-05-31 17:49:32 -07:00
2012-04-12 18:35:58 -07:00
2012-10-13 10:46:48 +01:00
2012-10-13 10:46:48 +01:00
2014-04-01 11:22:57 -07:00
2014-08-10 21:31:58 -07:00
2012-10-13 10:46:48 +01:00
2014-07-10 11:38:23 -07:00
2014-07-22 21:46:08 +01:00
2012-10-13 10:46:48 +01:00
2014-02-19 14:58:17 +01:00
2013-04-30 17:04:06 -07:00
2012-07-21 10:34:00 -07:00
2010-03-15 15:29:39 +01:00
2013-05-16 07:55:07 -07:00
2014-07-28 14:16:30 -04:00
2013-07-03 16:07:43 -07:00
2014-09-02 10:02:13 -07:00
2014-04-07 16:36:14 -07:00
2011-07-22 14:39:48 +09:30
2012-10-06 02:48:09 +09:00
2009-06-13 12:28:57 +02:00
2010-01-12 21:02:00 -08:00
2013-11-15 09:32:22 +09:00
2014-03-31 14:13:25 -07:00
2013-12-20 12:25:45 -08:00
2012-10-13 10:46:48 +01:00
2013-03-22 22:17:36 -06:00
2013-02-02 00:01:15 +01:00
2013-12-11 15:52:34 +01:00
2012-05-07 22:58:09 -04:00
2013-12-26 13:29:35 -05:00
2014-08-06 18:01:15 -07:00
2014-08-08 15:57:18 -07:00
2014-06-04 16:53:51 -07:00
2012-06-25 11:53:47 +02:00
2009-01-09 16:54:41 -08:00
2012-01-23 08:38:47 -08:00
2014-06-04 16:54:06 -07:00
2014-05-27 17:43:11 -07:00
2009-05-25 00:55:30 -07:00
2012-05-16 15:23:46 -04:00
2014-01-21 16:19:44 -08:00
2011-10-31 17:30:54 -07:00
2014-04-09 13:44:35 -07:00
2014-07-27 20:52:44 +09:30
2014-07-17 22:07:37 +02:00
2014-08-11 11:44:11 -07:00
2012-10-08 13:50:21 +10:30
2013-01-21 13:55:14 -05:00
2012-10-13 10:46:48 +01:00
2013-06-26 12:10:56 +02:00
2011-05-09 11:40:38 +10:00
2014-05-14 00:37:34 -04:00
2014-09-26 15:17:17 -04:00
2014-08-25 10:45:28 +02:00
2012-10-13 10:46:48 +01:00
2014-06-06 19:22:49 -04:00
2012-01-31 18:20:28 -05:00
2013-02-13 06:15:29 -08:00
2012-02-16 14:55:27 -05:00
2014-08-22 18:04:43 -04:00
2014-05-29 11:11:48 -04:00
2014-07-22 15:05:06 -07:00
2012-12-11 17:22:27 -08:00
2006-07-14 21:53:54 -07:00
2014-04-07 10:59:19 -07:00
2014-05-21 11:14:46 +02:00
2014-03-07 16:02:46 +01:00
2014-03-10 22:42:28 -07:00
2013-08-28 20:50:19 +01:00
2014-07-07 13:33:46 +01:00
2013-08-15 17:22:27 -07:00
2014-07-21 23:32:51 -07:00
2012-10-13 10:46:48 +01:00
2013-03-29 17:54:33 -07:00
2013-10-30 12:02:58 +08:00
2013-10-09 14:47:45 +02:00
2013-02-23 17:50:15 -08:00
2014-08-06 18:01:20 -07:00
2014-01-23 16:36:55 -08:00
2014-09-10 13:45:01 -06:00
2011-08-19 21:03:22 +04:00
2014-07-09 14:58:37 +01:00
2014-01-23 20:17:18 +00:00
2014-08-04 10:09:27 -07:00
2014-09-24 08:29:36 -06:00
2014-08-27 22:59:39 -07:00
2014-08-23 11:39:09 -07:00
2014-04-12 14:49:50 -07:00
2014-07-08 15:31:26 -07:00
2011-08-25 15:34:19 +02:00
2014-09-03 01:25:32 +02:00
2014-07-23 01:00:36 +02:00
2014-05-16 23:15:44 +02:00
2012-10-13 10:46:48 +01:00
2013-08-19 21:22:34 -07:00
2011-11-02 16:07:02 -07:00
2014-01-13 17:32:27 +01:00
2014-01-23 14:48:35 +01:00
2011-03-31 11:26:23 -03:00
2014-08-13 15:13:44 +10:00
2014-06-04 16:54:21 -07:00
2012-02-01 16:53:46 +08:00
2014-03-21 14:21:13 -04:00
2014-05-07 10:15:31 +02:00
2014-05-21 11:19:36 +02:00
2010-02-08 18:19:41 -06:00
2014-08-24 18:36:01 -07:00
2012-05-17 15:18:37 +02:00
2014-07-09 09:14:17 -07:00
2014-05-14 09:46:09 -07:00
2014-07-25 19:01:53 +01:00
2013-04-30 17:04:09 -07:00
2014-01-16 12:00:56 -08:00
2012-10-13 10:46:48 +01:00
2014-04-11 09:29:16 +02:00
2011-04-19 15:38:02 -04:00
2012-10-13 10:46:48 +01:00
2014-09-03 20:56:32 -07:00
2014-06-10 09:46:00 -04:00
2014-06-19 21:17:37 -07:00
2012-07-04 11:51:59 +03:00
2007-05-02 11:56:33 +01:00
2010-05-11 10:09:47 +02:00
2014-06-21 22:05:30 +02:00
2014-09-13 12:30:26 -04:00
2014-08-14 18:13:46 -06:00
2006-10-04 07:55:14 -07:00
2013-06-17 16:38:57 -07:00
2012-10-13 10:46:48 +01:00
2012-10-13 10:46:48 +01:00
2010-10-22 10:20:06 -07:00
2013-07-09 10:33:28 -07:00
2011-09-13 11:11:57 +02:00
2013-11-15 09:32:20 +09:00
2014-08-28 11:59:38 +05:30
2014-07-17 18:23:35 -07:00
2014-07-17 18:21:21 -07:00
2014-03-17 16:20:49 -07:00
2012-10-13 10:46:48 +01:00
2013-09-10 18:56:32 -04:00
2014-08-06 13:03:44 +02:00
2012-06-28 17:14:34 +01:00
2014-09-29 12:27:20 -04:00
2014-06-04 16:54:08 -07:00
2011-03-22 17:16:34 +09:00
2012-12-07 12:48:00 -05:00
2012-10-13 10:46:48 +01:00
2012-10-13 10:46:48 +01:00
2014-01-13 13:47:36 +01:00
2013-12-16 11:36:13 +01:00
2014-02-15 11:55:28 -08:00
2014-03-19 08:58:30 +00:00
2011-06-14 22:48:51 -04:00
2006-12-07 02:14:08 +01:00
2012-10-13 10:46:48 +01:00
2012-02-24 10:05:59 +01:00
2012-09-22 16:35:13 +03:00
2013-04-30 17:04:03 -07:00
2014-05-23 11:23:27 +09:00
2013-04-30 15:50:12 +05:30
2013-02-13 12:15:50 -08:00
2014-09-26 15:05:14 -04:00
2011-03-15 00:43:46 +01:00
2014-08-08 15:57:24 -07:00
2013-06-12 16:29:44 -07:00
2010-08-11 08:59:22 -07:00
2012-09-13 16:47:34 +02:00
2010-03-30 22:02:32 +09:00
2014-08-22 21:28:02 -07:00
2013-10-17 14:38:44 +02:00
2010-05-27 09:12:51 -07:00
2014-09-04 22:35:59 +02:00
2014-07-30 09:26:25 +02:00
2014-07-23 15:01:55 -07:00
2012-08-21 16:28:31 +02:00
2012-10-09 16:22:32 +09:00
2013-10-10 12:31:43 -07:00
2014-06-04 16:53:59 -07:00
2010-11-29 08:55:25 +11:00
2014-07-29 23:10:56 +02:00
2014-08-06 13:03:43 +02:00
2014-08-07 20:39:36 -04:00
2014-07-10 16:05:19 -07:00
2008-07-25 10:53:26 -07:00
2014-05-20 16:54:21 -06:00
2014-03-14 22:41:36 -04:00
2013-04-15 21:23:03 +01:00
2014-09-01 21:36:28 -07:00
2013-11-26 20:55:33 -08:00
2014-06-30 22:47:18 -07:00
2012-10-13 10:46:48 +01:00
2011-06-15 20:04:00 -07:00
2012-11-20 04:19:49 -08:00
2012-10-13 10:46:48 +01:00
2014-05-15 17:02:21 +01:00
2014-09-18 19:21:20 -04:00
2014-09-16 15:02:55 -06:00
2010-11-10 12:04:41 +09:00
2010-05-11 16:07:59 -06:00
2012-10-13 10:46:48 +01:00
2013-10-29 11:28:11 +10:30
2014-06-02 12:41:33 -07:00
2014-04-07 16:35:53 -07:00
2014-08-06 18:01:19 -07:00
2013-12-03 11:15:58 -08:00
2014-06-04 16:53:57 -07:00
2013-12-02 20:43:15 +01:00
2014-01-23 16:37:04 -08:00
2013-01-31 19:56:35 -05:00
2012-10-13 10:46:48 +01:00
2014-09-13 05:13:08 +09:00
2014-07-16 15:10:39 +02:00
2013-07-12 12:07:46 +02:00
2013-07-25 19:30:03 +10:00
2014-08-08 15:57:26 -07:00
2014-08-06 18:01:23 -07:00
2014-01-30 16:56:55 -08:00
b193722731