<div dir="ltr"><div class="gmail_default" style="font-family:trebuchet ms,sans-serif"><span style="font-family:arial">On Tue, Nov 11, 2014 at 1:42 AM, Jeroen Massar </span><span dir="ltr" style="font-family:arial"><<a href="mailto:jeroen@massar.ch" target="_blank" class="cremed">jeroen@massar.ch</a>></span><span style="font-family:arial"> wrote:</span><br></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 2014-11-11 02:18, Lorenzo Colitti wrote:<br>
> On Sun, Nov 9, 2014 at 8:10 PM, Jeroen Massar <<a href="mailto:jeroen@massar.ch" class="cremed">jeroen@massar.ch</a><br>
> <mailto:<a href="mailto:jeroen@massar.ch" class="cremed">jeroen@massar.ch</a>>> wrote:<br>
><br>
> > Another fun question is why folks are relying on PMTUD instead of<br>
> > adjusting their MTU settings (e.g., via RAs).<br>
><br>
> Because why would anybody want to penalize their INTERNAL network?<br>
><br>
><br>
> Lowering the MTU from 1500 to 1280 is only a 1% penalty in throughput.<br>
> I'd argue that that 1% is way less important than the latency penalty.<br>
<br>
1%? You do mean at least: (1500 - 1280 = 220) / 1500 * 100% = 3.3%<br>
<br>
And that is only if looking at the packet size.<br>
<br>
But that includes a IPv6 and TCP header, thus another 20 + 20 is lost<br>
for every packet (ignoring TCP options etc).<br>
<br>
Thus we can either send:<br>
1500 - 40 = 1460<br>
or<br>
1280 - 40 = 1240<br>
<br>
1460 / 1260 * 100 = 115.87301587 ~= 115% => 15% bigger packets<br>
<br>
According to a random google search for "youtube video size" apparently<br>
a a 25 minute clip is about 200 MiB.<br>
<br>
Hence, at MTU = 1500 and optimal splitting you are sending:<br>
(200 * 1024 * 1024) / 1460 = 143640+X packets<br>
(200 * 1024 * 1024) / 1240 = 169125+X packets<br>
<br>
Ignoring of course the small packets for ACKs and other such stuff.<br>
<br>
(169125 / 143640) * 100 = 117.7422723 ~= 118% => 18% more packets<br>
<br>
Hence 18% more packets than with a MTU of 1500.<br>
<br>
But, lets look at your "small" page: <a href="https://www.google.com" target="_blank" class="cremed">https://www.google.com</a><br>
<br>
According to Chrome, that is 441 KiB in 6.85 seconds here (that is<br>
excluding ads/socialcrap, thank you AdBlock, Disconnect & Ghostery)<br>
<br>
(441 * 1024) / 1460 = 309+X packets<br>
(441 * 1024) / 1240 = 364+X packets<br>
<br>
(364 / 309) * 100 = 117.79935275 ~= 118% => 18% more packets<br>
<br>
Like magic, the same 18%, which is close to the 15% larger size.<br>
<br>
(ignoring the fact that those where 23 separate requests, hence all the<br>
TCP overhead of syn/ack which are mostly empty etc).<br>
<br>
Thus maybe your 1% is a worst-case situation where not all packets are<br>
being 'filled' and my 18% is the best-case situation where all packets<br>
are fully 'filled'.<br>
<br>
<br>
As the problem with data transfer is the speed of light in a lot of<br>
cases, having the need to send more packets thus hurts. Sending 18% more<br>
packets thus definitely impacts your speed of transfer.<br>
<br>
Thus limiting the whole world to the same low MTU, hurts the whole world.<br>
<br>
<br>
Can we thus please NOT limit the world at 1280 and actually find the<br>
culprits and resolve this properly.<br>
<br>
<br>
<br>
> Because you can't know if that is always the case.<br>
><br>
><br>
> I'm not saying that PMTUD shouldn't work. I'm saying that if you know<br>
> that your Internet connection has an MTU of 1280, setting an MTU of 1500<br>
> on your host is a bad idea, because you know for sure that you will<br>
> experience a 1-RTT delay every time you talk to a new destination.<br>
<br>
Please realize that while Google might think they are the center of the<br>
Internet, a lot of people use *LOCAL* resources, that are not accessed<br>
over their puny WAN link.<br>
<br>
<br>
<br>
> As you work at Google, ever heard of this QUIC protocol that does not<br>
><br>
> use TCP?<br>
><br>
> Maybe you want to ask your colleagues about that :)<br>
><br>
><br>
> Does QUIC work from behind your tunnel? If so, maybe my colleagues have<br>
> already solved that problem.<br></blockquote><div><br></div><div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif">If the MTU is 1280, QUIC will not (currently) work, and Chrome will fall back to using TCP.</div></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
From:<br>
<a href="https://docs.google.com/document/d/1RNHkx_VvKWyWg6Lr8SZ-saqsQx7rFV-ev2jRFUoVD34/mobilebasic" target="_blank" class="cremed">https://docs.google.com/document/d/1RNHkx_VvKWyWg6Lr8SZ-saqsQx7rFV-ev2jRFUoVD34/mobilebasic</a><br>
"UDP PACKET FRAGMENTATION" but IPv6 dos not fragment...<br>
no mention on handling ICMP PTBs either, let alone mention of IPv6.<br></blockquote><div><br></div><div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif">Yup, if the MTU is too small to accommodate our packets we detect QUIC as broken and fallback to TCP. We assume that fragmentation and reassembly is rarely successful.</div></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
so it is at least not anywhere mentioned in the "design document"<br>
<br>
But there are apparently things like 'resending important packets' or<br>
even sending them multiple times etc.<br>
<br>
But what if you could just avoid the loss altogether instead of doing<br>
guess work and retransmissions.<br>
<br>
<br>
Looking at Chromium source directly:<br>
<a href="https://src.chromium.org/svn/trunk/src/net/quic/quic_protocol.h" target="_blank" class="cremed">https://src.chromium.org/svn/trunk/src/net/quic/quic_protocol.h</a><br>
<br>
8<----------------------------<br>
// Default and initial maximum size in bytes of a QUIC packet.<br>
const QuicByteCount kDefaultMaxPacketSize = 1350;<br>
<br>
// The maximum packet size of any QUIC packet, based on ethernet's max size,<br>
// minus the IP and UDP headers. IPv6 has a 40 byte header, UPD adds an<br>
// additional 8 bytes. This is a total overhead of 48 bytes. Ethernet's<br>
// max packet size is 1500 bytes, 1500 - 48 = 1452.<br>
const QuicByteCount kMaxPacketSize = 1452;<br>
// Default maximum packet size used in Linux TCP implementations.<br>
const QuicByteCount kDefaultTCPMSS = 1460;<br>
----------------------------->8<br>
<br>
(Btw, note the "UPD" typo there, too minor to report that as a 'bug' ;)<br>
<br>
Seems it was 1200, but got upped to a magic 1350:<br>
<a href="https://codereview.chromium.org/427673005/" target="_blank" class="cremed">https://codereview.chromium.org/427673005/</a><br>
<a href="https://codereview.chromium.org/420313005/" target="_blank" class="cremed">https://codereview.chromium.org/420313005/</a><br>
<br>
Not much background there; but those sizes are likely<br></blockquote><div><br></div><div><div class="gmail_default" style="font-family:'trebuchet ms',sans-serif">Yes, currently our max packet size is 1350 bytes of UDP payload (8 bytes fewer for IPv6). We chose this number based on the results of experiments we ran which tried various packet sizes and saw what fraction of the net was able to send and received them. 1350 was a good compromise between reachability and maximum payload.</div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
And just in case, checking the current source:<br>
<br>
$ git clone <a href="https://chromium.googlesource.com/chromium/src.git" target="_blank" class="cremed">https://chromium.googlesource.com/chromium/src.git</a><br>
...<br>
remote: Sending approximately 2.54 GiB ...<br>
...<br>
Receiving objects: 100% (3272908/3272908), 2.53 GiB | 13.56 MiB/s, done.<br>
...<br>
<br>
(Wireless home networks behind tunnels are so slow...)<br>
<br>
Going through that, it does not seem to let the OS handle fragmentation<br>
as it seems they are just stuffing packets upto 1350.<br>
<br>
As in "solved", not really... ignoring is the right word.<br>
<br>
Now of course, that code might not match at all what is used in Chrome<br>
or the server side.<br>
<br>
Bcc'd some folks who might actually know how that is handled without me<br>
having to try and understand all the code (lots of stuff happening<br>
there...).<br>
<br>
> > (Some parts of) Google infrastructure do not do<br>
> > PMTUD for the latency reasons above and for reasons similar to those<br>
> > listed<br>
> > in <a href="https://tools.ietf.org/html/draft-v6ops-jaeggli-pmtud-ecmp-problem-00" target="_blank" class="cremed">https://tools.ietf.org/html/draft-v6ops-jaeggli-pmtud-ecmp-problem-00</a> .<br>
><br>
> As such, you are ON PURPOSE breaking PMTUD, instead trying to fix it<br>
> with some other bandaid.<br>
><br>
><br>
> The draft explains some of the reasons why infrastructure is often built<br>
> this way.<br>
<br>
Yes, and it is great that is documented. Thank you also for admitting<br>
that that is the problem. And it is not something that anybody who does<br>
not touch loadbalancers on a daily basis will think of either.<br>
<br>
Hence why I don't think the original IPv6 design accounted for this kind<br>
of setup, that is quite common nowadays, but was not back then.<br>
<br>
Thus can we please PROPERLY fix this MTU/ICMPv6-PTB problem before we<br>
stick the world to a 1280 MTU for as long as IPv6 will exist (which will<br>
outlast our lifetimes hopefully...)?<br>
<br>
See my proposal about stuffing the MTU in the flow-label that can avoid<br>
that nightmare. You where one of many in the BCC as you should be caring<br>
about it as it affects your companies network a lot.<br>
<br>
Greets,<br>
Jeroen<br>
<br>
</blockquote></div><br></div></div>