Fix the packet loss issues or lose a customer

  • 21 February 2023
  • 8 replies
  • 131 views

Badge

Every night around 7PM CST until 8 or 9 PM shadow has constant packet loss between 10 and 50% which makes any game with real time action completely unplayable. 

The only reason I use shadow over Geforce Now (which is cheaper and has better hardware) is because I can use mods. 

If these packet loss issues are not resolved soon (they occur over the last two hops to shadow in Texas) I definitely won’t stay subscribed after the promotional price ends.


This topic has been closed for comments

8 replies

Badge

Issue seems to be resolved, no issues with packet loss for the last week or so. 

Userlevel 5
Badge +4

@bluscandth One more thought on this. Can you please paste your entire traceroute, or provide the hop just prior to the ae2-666.spine8.tx1.shadow.guru. The transition between those two is likely a peering/transit connection, and that is likely the source of the issue (although still not possible for us to know which “side” of it).

Userlevel 5
Badge +4

@bluscandth 

Good questions! Let's go through them.

Q: "...but being that udp and tcp are both transport layers protocols that have to be wrapped in IP packets in order to be transmitted across a network, if one gets packet loss when testing IP packets, doesn’t this mean that lower layer protocols that rely on the network layer would not be able to get through either?"
A: No, it doesn't. TCP, UDP, and ICMP are all IP protocols as you say...but routing devices can be configured to treat each type differently...especially when there is contention (circuit congestion when there is not enough bandwidth to carry all of the data through). Decisions have to be made under those conditions. ICMP (if allowed at all) will typically be prioritized the least. Regarding TCP and UDP, TCP uses acknowledgements (verification of receipt by the destination) whereas UDP does not. Packet loss with TCP protocols means that the missing data will be retried, whereas with UDP (and especially with game streams) you get one shot - either the data is received or it's not. Even if some other mechanism is being leveraged to keep track of which data was missed, it's too late to resend it when doing realtime streaming.

Q: "How do you get UDP datagrams across the network without wrapping them in packets? Please explain."
A: Of course they are, in IP packets. See the distinction in the previous answer between TCP and UDP.
 
Q: "You obviously have no grasp of the OSI model. I’m not here to educate you but you have no idea how this works."
A: Okay, copy that. Please educate me, as I like to learn new things.

Q: "In this context, to get a UDP packet to our PC from shadow it is necessary to first encapsulate it in an IP layer packet. This packet is what transports your precious magic UDP packet across the different hops of a route until it lands at shadow’s endpoint. It is then decoded back into a udp datagram which is then decoded into the raw application data of the shadow streamer application."
A: Yep, no argument there.

Q: "If there is ICMP packet loss, there is UDP loss as well."
A: See the answer to the first question.

Q: "You’re an idiot."
A: Sometimes, absolutely...we're all human after all. In terms of networking knowledge, no...I don't believe I am. Perhaps we should get more opinions.

Q: "It’s 100% a shadow problem. The packet loss occurs at the addresses ae2-666.spine8.tx1.shadow.guru and 160-net-216-180-132.shadow.guru which are the last two hops to texas."
A: If you're still using traceroute, all this means is that replies to your ICMP messages aren't making it back. This is far different from a UDP stream originating from the remote side. It IS possible that there is congestion within Shadow's ISP's circuit(s), and I've already conceded this...but you're stating it as proven fact, which it isn't.

Q: "As I type this the first address is dropping 7% packets on average since I began measuring. Some texas users have gotten email about a migration to OVHCloud servers… this is probably realted to to that."
A: Dropping 7% of ICMP packets...see the first answer.

Closing thoughts: If you're trying to isolate packet loss on UDP streams, you're not going to get there with traceroute. However, being that with Shadow you have control over a remote computer...there ARE potential methods to check for UDP loss along portions of the route. I hereby extend an olive branch to you...if we can drop the personal attacks, I'll give you my thoughts on an approach.
 

Badge

It’s 100% a shadow problem. The packet loss occurs at the addresses ae2-666.spine8.tx1.shadow.guru and 160-net-216-180-132.shadow.guru which are the last two hops to texas. As I type this the first address is dropping 7% packets on average since I began measuring. 
Some texas users have gotten email about a migration to OVHCloud servers… this is probably realted to to that.  

Badge

No it's definitely a problem with Shadow and the traceroute clearly shows the packet loss coming from the last two hops to shadow both of which have ‘shadow’ in their domain names not to mention users on reddit are having the same issue with the Texas datacenter at around the same times every night when my issue occurs. 

I didn’t post for useless advice for internet know-it-alls that don’t even have all the information, I posted to let shadow know I am dissatisfied for this service I am paying for. 

Anyway what you said is completely false you can send UDP packets with traceroute and besides that if a hop is dropping packets it’s likely dropping all types of packets. T. Networking degree

 

@bluscandth That regional congestion could be happening in Texas, but not necessarily within Shadow’s ISP. As I said in my original reply, if that’s where the problem is occurring, Shadow networking staff (and the ISP) will be aware.

It is not possible to test for packet loss with high-speed UDP streams used by game-streaming services using traceroute. If you’d like to learn why, please give this a read (I wrote it for the GFN service, but it applies to Shadow as well): https://www.nvidia.com/en-us/geforce/forums/gfn-tech-support/46/484584/constant-packet-losses-network-problems-on-eu-cent/3211608/

If you have a networking degree, then apparently your training came from the back of a cereal box.

Sincerely, an actual network professional.

Correct me if I’m wrong Mr. Networking but being that udp and tcp are both transport layers protocols that have to be wrapped in IP packets in order to be transmitted across a network, if one gets packet loss when testing IP packets, doesn’t this mean that lower layer protocols that rely on the network layer would not be able to get through either? How do you get UDP datagrams across the network without wrapping them in packets? Please explain. 
You obviously have no grasp of the OSI model. I’m not here to educate you but you have no idea how this works. 
In this context, to get a UDP packet to our PC from shadow it is necessary to first encapsulate it in an IP layer packet. This packet is what transports your precious magic UDP packet across the different hops of a route until it lands at shadow’s endpoint. It is then decoded back into a udp datagram which is then decoded into the raw application data of the shadow streamer application. 

If there is ICMP packet loss, there is UDP loss as well. You’re an idiot. 

Userlevel 5
Badge +4

No it's definitely a problem with Shadow and the traceroute clearly shows the packet loss coming from the last two hops to shadow both of which have ‘shadow’ in their domain names not to mention users on reddit are having the same issue with the Texas datacenter at around the same times every night when my issue occurs. 

I didn’t post for useless advice for internet know-it-alls that don’t even have all the information, I posted to let shadow know I am dissatisfied for this service I am paying for. 

Anyway what you said is completely false you can send UDP packets with traceroute and besides that if a hop is dropping packets it’s likely dropping all types of packets. T. Networking degree

 

@bluscandth That regional congestion could be happening in Texas, but not necessarily within Shadow’s ISP. As I said in my original reply, if that’s where the problem is occurring, Shadow networking staff (and the ISP) will be aware.

It is not possible to test for packet loss with high-speed UDP streams used by game-streaming services using traceroute. If you’d like to learn why, please give this a read (I wrote it for the GFN service, but it applies to Shadow as well): https://www.nvidia.com/en-us/geforce/forums/gfn-tech-support/46/484584/constant-packet-losses-network-problems-on-eu-cent/3211608/

If you have a networking degree, then apparently your training came from the back of a cereal box.

Sincerely, an actual network professional.

Badge

@bluscandth This is most likely regional congestion; nothing to do with the Shadow systems. It’s also not possible to determine where that packet loss is occurring (due to the nature of UDP protocols), without having control of each hop along the route.

If there are capacity problems with Shadow’s ISP being used at the data center, then that will be known...and remediation taken at some point. But again, it’s likely not that.

Cloud gaming providers can’t be held responsible for the “entire Internet” between their systems and end-users.

No it's definitely a problem with Shadow and the traceroute clearly shows the packet loss coming from the last two hops to shadow both of which have ‘shadow’ in their domain names not to mention users on reddit are having the same issue with the Texas datacenter at around the same times every night when my issue occurs. 

I didn’t post for useless advice for internet know-it-alls that don’t even have all the information, I posted to let shadow know I am dissatisfied for this service I am paying for. 

Anyway what you said is completely false you can send UDP packets with traceroute and besides that if a hop is dropping packets it’s likely dropping all types of packets. T. Networking degree

Userlevel 5
Badge +4

@bluscandth This is most likely regional congestion; nothing to do with the Shadow systems. It’s also not possible to determine where that packet loss is occurring (due to the nature of UDP protocols), without having control of each hop along the route.

If there are capacity problems with Shadow’s ISP being used at the data center, then that will be known...and remediation taken at some point. But again, it’s likely not that.

Cloud gaming providers can’t be held responsible for the “entire Internet” between their systems and end-users.