23 Oct
2013
23 Oct
'13
11:59 p.m.
Hey axel!
so, we've scratching our heads today, facing some MTU issues on our
shiny new libre-mesh network,
and after a few hours debugging, tcpdumping and discussing, we came to
these conclusions:
* the invented src-addr in the bmxOut tunnel makes it impossible for
hops along the path to return "Packet Too Big" to the originating node.
So, if a particular link has a smaller MTU than the first link (say, a
VPN is involved), the packet will be silently dropped.
A==1500==B---1400---C===1500===D
A wants to send a packet to a network announced by D; so it creates a
tunnel with D as destination, and a "D-derived" fake address as src,
that matches the bmxIn "catchall" tunnel peer-addr in D
Then sends a 1460 + 40 = 1500 bytes packet through that tunnel,
B cannot push that packet to C, then tries to send back a ICMPv6 PTB...
but the src-addr it finds in the encapsulation is not A, but instead the
"D-derived" fake addr. Then, the ICMPv6 PTB is lost and A can never find
out about the smaller MTU.
This fundamentally breaks PMTUD which is a bad idea in IPv6
to avoid this there are 3 options:
* set mtu=1280 on every bmxOut tunnel (yuck! :( )
* probe before establishing each tunnel, with the real src-addr so that
PMTUD can happen correctly, until it reaches the desired endpoint node.
Then, use discovered PMTU for new tunnel. (*downside*: this will only
work as long as path doesn't change to pass through some thinner link.
In that case, PMTU will not be rediscovered, and packets will be dropped
again.)
* use the current bmxIn "catchall" tunnels only for sending special bmx6
control packets, that ask for a symmetric tunnel.
i.e.
1) A sends to D (2001:db8::D) a packet (encapsulated with a fake
src-addr, to be catched by the catchall @ D) with content "I'm A and
this is my real address 2001:db8::A; please make a dedicated tunnel for me"
2) D gets that packet and creates a tunnel with "peer 2001:db8::A", then
sends back an ACK to A, again using "A-derived" fake-addr as src
3) A gets the ACK and creates the tunnel with "peer 2001:db8::D"
*** now both ends have a symmetric tunnel between them, with real src
and dest address ***
4) A finally sends the real payload through the symmetric tunnel, this
payload (may be bigger than 1280, say... 1450) will be encapsulated with
the real src-addr of A, so if any node in the path needs to send back a
Packet-too-big, will be able to, and PMTUD will happen correctly.
(at the cost of a full RTT latency before the first payload packet, but
with a reasonable tunnel expire time as it has currently, that shouldn't
be terrible)
back in April, i remember we discussed the idea of symmetric tunnels,
and you brought up this "control connection" idea which i'm simply
redescribing here.
But that discussion was in another context, more like a 'feature', and
finally didn't really solve the idea we had originally,
yet, this PMTUD issue was not taken into account AFAIR, so the
"symmetric tunnel" idea now becomes more like a bugfix (i.e. don't
create PMTUD blackholes)
what do you think?
Cheers!!
gui
12:24 p.m.
Hi Gui!
This is indeed a very very nice bug report. Despite the complexity of
the context.
unfortunately, What you describe makes totally sense :-(
Some more comments linline...
On 10/24/2013 01:59 AM, Gui Iribarren wrote:
Hey axel!
so, we've scratching our heads today, facing some MTU issues on our
shiny new libre-mesh network,
and after a few hours debugging, tcpdumping and discussing, we came to
these conclusions:
* the invented src-addr in the bmxOut tunnel makes it impossible for
hops along the path to return "Packet Too Big" to the originating
node. So, if a particular link has a smaller MTU than the first link
(say, a VPN is involved), the packet will be silently dropped.
A==1500==B---1400---C===1500===D
A wants to send a packet to a network announced by D; so it creates a
tunnel with D as destination, and a "D-derived" fake address as src,
that matches the bmxIn "catchall" tunnel peer-addr in D
Then sends a 1460 + 40 = 1500 bytes packet through that tunnel,
B cannot push that packet to C, then tries to send back a ICMPv6 PTB...
but the src-addr it finds in the encapsulation is not A, but instead
the "D-derived" fake addr. Then, the ICMPv6 PTB is lost and A can
never find out about the smaller MTU.
This fundamentally breaks PMTUD which is a bad idea in IPv6
to avoid this there are 3 options:
* set mtu=1280 on every bmxOut tunnel (yuck! :( )
must it be so small? mtu=1350 is too big for the potential intermediate
VPNs?
* probe before establishing each tunnel, with the real
src-addr so
that PMTUD can happen correctly, until it reaches the desired endpoint
node. Then, use discovered PMTU for new tunnel. (*downside*: this will
only work as long as path doesn't change to pass through some thinner
link. In that case, PMTU will not be rediscovered, and packets will be
dropped again.)
* use the current bmxIn "catchall" tunnels only for sending special
bmx6 control packets, that ask for a symmetric tunnel.
i.e.
1) A sends to D (2001:db8::D) a packet (encapsulated with a fake
src-addr, to be catched by the catchall @ D) with content "I'm A and
this is my real address 2001:db8::A; please make a dedicated tunnel
for me"
2) D gets that packet and creates a tunnel with "peer 2001:db8::A",
then sends back an ACK to A, again using "A-derived" fake-addr as src
using "A-derived" fake-addr as src does not work from A because this
implies that the fake-addr is a valid address on A and that causes any
incoming packet having this address as src address to be dropped.
Instead A's primary address could be used. But anyway I hope we can
solve it without a dedicated control connection.... see below.
3) A gets the ACK and creates the tunnel with
"peer 2001:db8::D"
*** now both ends have a symmetric tunnel between them, with real src
and dest address ***
4) A finally sends the real payload through the symmetric tunnel, this
payload (may be bigger than 1280, say... 1450) will be encapsulated
with the real src-addr of A, so if any node in the path needs to send
back a Packet-too-big, will be able to, and PMTUD will happen correctly.
(at the cost of a full RTT latency before the first payload packet,
but with a reasonable tunnel expire time as it has currently, that
shouldn't be terrible)
back in April, i remember we discussed the idea of symmetric tunnels,
and you brought up this "control connection" idea which i'm simply
redescribing here.
But that discussion was in another context, more like a 'feature', and
finally didn't really solve the idea we had originally,
yet, this PMTUD issue was not taken into account AFAIR, so the
"symmetric tunnel" idea now becomes more like a bugfix (i.e. don't
create PMTUD blackholes)
what do you think?
I think the reception of an ip6-encapsulated packet destinated to the
primary address of the node could be used as an implicit request to
setup the reverse tunnel (from the gw to the node). So the additional
setup delay (of one RTT) could be avoided but at the cost of not knowing
if the gateway accepted or will ever accept the implicit tunnel request.
I'll try to speed up the dev of symmetric tunnels.
/axel
Cheers!!
gui
_______________________________________________
Dev mailing list
Dev(a)lists.libre-mesh.org
https://lists.libre-mesh.org/mailman/listinfo/dev
19 Nov
19 Nov
11:22 a.m.
On 10/24/2013 02:24 PM, Axel Neumann wrote:
the question here would be: which src-address to use for that first packet?
if you use the primary address of the originating node, then AFAIU the
kernel will drop it, unless there's a point-to-point interface with that
specific address as "peer"
so my idea was to use your current "hacky" scheme of addresses (src-addr
is fake but predictable, based on dst-addr) to deliver the first packet
("I want a reverse tunnel")
but maybe it's not needed and i'm not seeing the big picture :)
something i would love to hear!
Hi Gui!
3) A gets the
ACK and creates the tunnel with "peer 2001:db8::D"
*** now both ends have a symmetric tunnel between them, with real src
and dest address ***
4) A finally sends the real payload through the symmetric tunnel, this
payload (may be bigger than 1280, say... 1450) will be encapsulated
with the real src-addr of A, so if any node in the path needs to send
back a Packet-too-big, will be able to, and PMTUD will happen correctly.
(at the cost of a full RTT latency before the first payload packet,
but with a reasonable tunnel expire time as it has currently, that
shouldn't be terrible)
back in April, i remember we discussed the idea of symmetric tunnels,
and you brought up this "control connection" idea which i'm simply
redescribing here.
But that discussion was in another context, more like a 'feature', and
finally didn't really solve the idea we had originally,
yet, this PMTUD issue was not taken into account AFAIR, so the
"symmetric tunnel" idea now becomes more like a bugfix (i.e. don't
create PMTUD blackholes)
what do you think?
I think the reception of an ip6-encapsulated packet destinated to the
primary address of the node could be used as an implicit request to
setup the reverse tunnel (from the gw to the node). So the additional
setup delay (of one RTT) could be avoided but at the cost of not knowing
if the gateway accepted or will ever accept the implicit tunnel request.
I'll try to speed up the dev of symmetric tunnels.
sounds great! in the meantime we're getting away with the mtu=1350, but
last week i bumped into some (suspected) mtu issues again, maybe because
it was a tunnel over a DSL line (thus adding 8bytes overhead), so anyway
a proper solution will be a great improvement overall, whenever it's
ready :D
cheers!
gui
/axel
Cheers!!
gui
_______________________________________________
Dev mailing list
Dev(a)lists.libre-mesh.org
https://lists.libre-mesh.org/mailman/listinfo/dev
25 Nov
25 Nov
12:18 p.m.
On 11/19/2013 12:22 PM, Gui Iribarren wrote:
On 10/24/2013 02:24 PM, Axel Neumann wrote:
the question here would be: which src-address to use for that first
packet?
if you use the primary address of the originating node, then AFAIU the
kernel will drop it, unless there's a point-to-point interface with
that specific address as "peer"
ack. but my initial idea was to catch it anyway via a pcap socket
(similar to how magic-icmp catches packets).
Its still an option but currently I am more in favor of whats described
below:
Hi Gui!
3) A gets
the ACK and creates the tunnel with "peer 2001:db8::D"
*** now both ends have a symmetric tunnel between them, with real src
and dest address ***
4) A finally sends the real payload through the symmetric tunnel, this
payload (may be bigger than 1280, say... 1450) will be encapsulated
with the real src-addr of A, so if any node in the path needs to send
back a Packet-too-big, will be able to, and PMTUD will happen
correctly.
(at the cost of a full RTT latency before the first payload packet,
but with a reasonable tunnel expire time as it has currently, that
shouldn't be terrible)
back in April, i remember we discussed the idea of symmetric tunnels,
and you brought up this "control connection" idea which i'm simply
redescribing here.
But that discussion was in another context, more like a 'feature', and
finally didn't really solve the idea we had originally,
yet, this PMTUD issue was not taken into account AFAIR, so the
"symmetric tunnel" idea now becomes more like a bugfix (i.e. don't
create PMTUD blackholes)
what do you think?
I think the reception of an ip6-encapsulated packet destinated to the
primary address of the node could be used as an implicit request to
setup the reverse tunnel (from the gw to the node). So the additional
setup delay (of one RTT) could be avoided but at the cost of not knowing
if the gateway accepted or will ever accept the implicit tunnel request.
so my idea was to use your current "hacky" scheme of addresses
(src-addr is fake but predictable, based on dst-addr) to deliver the
first packet ("I want a reverse tunnel")
Yes, that's an option. But would remain a hacky scheme :-(
If dedicated tunnel-request signals are needed anyway, then one could
also build based on standard IP/UDP packets and using the correct node
primary addresses for src and dest of the UDP/IP request.
The setup delay would depend on the handshake procedure.
With a full two-way handshake, for sure first packets would be delayed
by one RTT.
But maybe with an opportunistic handshake (which assumes that the GW
usually accepts the tunnel requests and immediately sets up the tunnel)
the first tunnel packets can be send almost immediately after the tunnel
request packet of the handshake.
The tunnel reply packet of the handshake would then only become an
acknowledgement to inform the GW-client node that the request was approved.
/axel
but maybe it's not needed and i'm not seeing the big picture :)
something i would love to hear!
I'll try to speed up the dev of symmetric tunnels.
sounds great! in the meantime we're getting away with the mtu=1350,
but last week i bumped into some (suspected) mtu issues again, maybe
because it was a tunnel over a DSL line (thus adding 8bytes overhead),
so anyway a proper solution will be a great improvement overall,
whenever it's ready :D
cheers!
gui
/axel
Cheers!!
gui
_______________________________________________
Dev mailing list
Dev(a)lists.libre-mesh.org
https://lists.libre-mesh.org/mailman/listinfo/dev
16 May
16 May
2:49 a.m.
On 25/11/13 09:18, Axel Neumann wrote:
On 11/19/2013 12:22 PM, Gui Iribarren wrote:
Hey Axel,
hope you're having great fun at the battlemesh!
down here we're pulling our hairs out with bmx6 and the broken PMTUD,
we've deployed bmx6 to connect different clouds/towns in Cordoba
province, at the same time connected to the National University of
Cordoba, where we are hosting a librenet6 tunnel endpoint, and from
there all the way to catalunya to get ipv6 connectivity
There's a natural salad of MTUs involved, and the mtu=1350 workaround
Just Doesn't Scale Anymore :(
furthermore, the impossibility of doing proper PMTUD makes
troubleshooting the setup *really* hard, with just 4 consecutive hops,
tinc tunnels mixed in with vlans and stuff
Isaac was also independently bitten by this bug at his homelab, and
fought with MTU issues for at least a day, until isolating the culprit
during a timely chat on #libre-mesh :)
So, as a gentle ping, i'd like to ask:
do you think there's any chance of you finding the time to implement the
symmetric tunnels in bmx6 in the near future?
[to sum up the discussed alternatives:]
a) pcap (voted down by axel)
b) tunnel-request as a normal ip/udp packet with src and dst set to real
ip of nodes, and an ack required to start sending payload packets
c) same as b) but opportunistic, payload is sent before getting ack
i'd vote for c) since it's the most elegant (and less latency), but i'm
not sure what will happen in case of packet loss (given it'll be udp)
pd. we had an evil plan of proposing an experiment for the battlemesh,
consisting of putting a random MTU (say, between 1300 and 1500) on every
node interface, and doing netperfs across clients announced through
tunIn, to see how each protocol handles the situation ^_^
but we opted for writing this email instead ;)
cheers!!
gui & nico
On 10/24/2013 02:24 PM, Axel Neumann wrote:
the question here would be: which src-address to use for that first
packet?
if you use the primary address of the originating node, then AFAIU the
kernel will drop it, unless there's a point-to-point interface with
that specific address as "peer"
ack. but my initial idea was to catch it anyway via a pcap socket
(similar to how magic-icmp catches packets).
Its still an option but currently I am more in favor of whats described
below:
Hi Gui!
3) A gets
the ACK and creates the tunnel with "peer 2001:db8::D"
*** now both ends have a symmetric tunnel between them, with real src
and dest address ***
4) A finally sends the real payload through the symmetric tunnel, this
payload (may be bigger than 1280, say... 1450) will be encapsulated
with the real src-addr of A, so if any node in the path needs to send
back a Packet-too-big, will be able to, and PMTUD will happen
correctly.
(at the cost of a full RTT latency before the first payload packet,
but with a reasonable tunnel expire time as it has currently, that
shouldn't be terrible)
back in April, i remember we discussed the idea of symmetric tunnels,
and you brought up this "control connection" idea which i'm simply
redescribing here.
But that discussion was in another context, more like a 'feature', and
finally didn't really solve the idea we had originally,
yet, this PMTUD issue was not taken into account AFAIR, so the
"symmetric tunnel" idea now becomes more like a bugfix (i.e. don't
create PMTUD blackholes)
what do you think?
I think the reception of an ip6-encapsulated packet destinated to the
primary address of the node could be used as an implicit request to
setup the reverse tunnel (from the gw to the node). So the additional
setup delay (of one RTT) could be avoided but at the cost of not knowing
if the gateway accepted or will ever accept the implicit tunnel request.
so my idea was to use your current "hacky" scheme of addresses
(src-addr is fake but predictable, based on dst-addr) to deliver the
first packet ("I want a reverse tunnel")
Yes, that's an option. But would remain a hacky scheme :-(
If dedicated tunnel-request signals are needed anyway, then one could
also build based on standard IP/UDP packets and using the correct node
primary addresses for src and dest of the UDP/IP request.
The setup delay would depend on the handshake procedure.
With a full two-way handshake, for sure first packets would be delayed
by one RTT.
But maybe with an opportunistic handshake (which assumes that the GW
usually accepts the tunnel requests and immediately sets up the tunnel)
the first tunnel packets can be send almost immediately after the tunnel
request packet of the handshake. The tunnel reply packet of the handshake would then
only become an
acknowledgement to inform the GW-client node that the request was approved.
/axel
>
> but maybe it's not needed and i'm not seeing the big picture :)
> something i would love to hear!
>
>>
>>
>> I'll try to speed up the dev of symmetric tunnels.
>
> sounds great! in the meantime we're getting away with the mtu=1350,
> but last week i bumped into some (suspected) mtu issues again, maybe
> because it was a tunnel over a DSL line (thus adding 8bytes overhead),
> so anyway a proper solution will be a great improvement overall,
> whenever it's ready :D
>
> cheers!
>
> gui
>
>>
>> /axel
>>
>>>
>>> Cheers!!
>>>
>>> gui
>>> _______________________________________________
>>> Dev mailing list
>>> Dev(a)lists.li
20 May
20 May
7:29 a.m.
On 05/16/2014 04:49 AM, Gui Iribarren wrote:
On 25/11/13 09:18, Axel Neumann wrote:
Good chances within next ~2 month. But based on the redesigned bmx6
version that implement the trust & security stuff I presented at is4cwn.
The delay with this symmetric-tunnel feature is essentially due to the
decision to base this and other feature-requests on this new bmx6
(currently bmx6/dev branch) because I hope that this is the feature
anyway and I prefer to not do&debug the things twice. Also supporting
encrypted tunnels would become MUCH easier.
On 11/19/2013 12:22 PM, Gui Iribarren wrote:
Hey Axel,
hope you're having great fun at the battlemesh!
down here we're pulling our hairs out with bmx6 and the broken PMTUD,
we've deployed bmx6 to connect different clouds/towns in Cordoba
province, at the same time connected to the National University of
Cordoba, where we are hosting a librenet6 tunnel endpoint, and from
there all the way to catalunya to get ipv6 connectivity
There's a natural salad of MTUs involved, and the mtu=1350 workaround
Just Doesn't Scale Anymore :(
furthermore, the impossibility of doing proper PMTUD makes
troubleshooting the setup *really* hard, with just 4 consecutive hops,
tinc tunnels mixed in with vlans and stuff
Isaac was also independently bitten by this bug at his homelab, and
fought with MTU issues for at least a day, until isolating the culprit
during a timely chat on #libre-mesh :)
So, as a gentle ping, i'd like to ask:
do you think there's any chance of you finding the time to implement the
symmetric tunnels in bmx6 in the near future? On 10/24/2013 02:24 PM, Axel Neumann wrote:
ack. but my initial idea was to catch it anyway via a pcap socket
(similar to how magic-icmp catches packets).
Its still an option but currently I am more in favor of whats described
below:
Hi Gui!
> 3) A gets the ACK and creates the tunnel with
"peer 2001:db8::D"
> *** now both ends have a symmetric tunnel between them, with real src
> and dest address ***
> 4) A finally sends the real payload through the symmetric tunnel, this
> payload (may be bigger than 1280, say... 1450) will be encapsulated
> with the real src-addr of A, so if any node in the path needs to send
> back a Packet-too-big, will be able to, and PMTUD will happen
> correctly.
>
> (at the cost of a full RTT latency before the first payload packet,
> but with a reasonable tunnel expire time as it has currently, that
> shouldn't be terrible)
>
> back in April, i remember we discussed the idea of symmetric tunnels,
> and you brought up this "control connection" idea which i'm simply
> redescribing here.
> But that discussion was in another context, more like a 'feature', and
> finally didn't really solve the idea we had originally,
> yet, this PMTUD issue was not taken into account AFAIR, so the
> "symmetric tunnel" idea now becomes more like a bugfix (i.e. don't
> create PMTUD blackholes)
>
> what do you think?
I think the reception of an ip6-encapsulated packet destinated to the
primary address of the node could be used as an implicit request to
setup the reverse tunnel (from the gw to the node). So the additional
setup delay (of one RTT) could be avoided but at the cost of not knowing
if the gateway accepted or will ever accept the implicit tunnel request.
the question here would be: which src-address to use for that first
packet?
if you use the primary address of the originating node, then AFAIU the
kernel will drop it, unless there's a point-to-point interface with
that specific address as "peer"
so my idea was to use your current "hacky" scheme of addresses
(src-addr is fake but predictable, based on dst-addr) to deliver the
first packet ("I want a reverse tunnel")
Yes, that's an option. But would remain a hacky scheme :-(
If dedicated tunnel-request signals are needed anyway, then one could
also build based on standard IP/UDP packets and using the correct node
primary addresses for src and dest of the UDP/IP request.
The setup delay would depend on the handshake procedure.
With a full two-way handshake, for sure first packets would be delayed
by one RTT.
But maybe with an opportunistic handshake (which assumes that the GW
usually accepts the tunnel requests and immediately sets up the tunnel)
the first tunnel packets can be send almost immediately after the tunnel
request packet of the handshake.
[to sum up the discussed alternatives:]
a) pcap (voted down by axel)
b) tunnel-request as a normal ip/udp packet with src and dst set to real
ip of nodes, and an ack required to start sending payload packets
c) same as b) but opportunistic, payload is sent before getting ack
i'd vote for c)
Ack!
since it's the most elegant (and less latency),
but i'm
not sure what will happen in case of packet loss (given it'll be udp)
if no ack nor nack is received its send again. So consequences should be
a delayed tunnel establishment.
pd. we had an evil plan of proposing an experiment for the battlemesh,
consisting of putting a random MTU (say, between 1300 and 1500) on every
node interface, and doing netperfs across clients announced through
tunIn, to see how each protocol handles the situation ^_^
but we opted for writing this email instead ;)
Nice, these are the kind of experiments to push developers.
Indeed there was a talk given by Saverio covering MTU problems. And in
his cases bmx6 was not involved at all.
So your findings are for sure one of the reasons that should be fixed :-(
But there maybe more involved...
ciao
/axel
cheers!!
gui & nico
The tunnel reply packet of the handshake would
then only become an
acknowledgement to inform the GW-client node that the request was approved.
/axel
>
> but maybe it's not needed and i'm not seeing the big picture :)
> something i would love to hear!
>
>>
>>
>> I'll try to speed up the dev of symmetric tunnels.
>
> sounds great! in the meantime we're getting away with the mtu=1350,
> but last week i bumped into some (suspected) mtu issues again, maybe
> because it was a tunnel over a DSL line (thus adding 8bytes overhead),
> so anyway a proper solution will be a great improvement overall,
> whenever it's ready :D
>
> cheers!
>
> gui
>
>>
>> /axel
>>
>>>
>>> Cheers!!
>>>
>>> gui
>>> _______________________________________________
>>> Dev mailing list
>>> Dev(a)lists.li
3623
days inactive
3832
days old
6 comments
3 participants
participants (3)
-
Axel Neumann -
Gioacchino Mazzurco -
Gui Iribarren