We are really looking for help from the community we are getting issues where our clients website can no longer reach my api and timeout with.
Connection timed out after 10001 milliseconds.
We have contacted our hosts media temple and they cant give me any resolution and we really need some help.
Here is a video of my issue we are experiencing.
Here is a trace route to our api, it works fine for many of our users but some are now getting issues like below.
1 50.23.120.193-static.reverse.softlayer.com (50.23.120.193) 0.513 ms 0.491 ms 0.481 ms
2 ae0.dar02.sr01.sjc01.networklayer.com (50.23.118.132) 0.366 ms ae0.dar01.sr01.sjc01.networklayer.com (50.23.118.130) 0.406 ms ae0.dar02.sr01.sjc01.networklayer.com (50.23.118.132) 0.302 ms
3 50.97.19.164-static.reverse.softlayer.com (50.97.19.164) 0.775 ms 0.701 ms ae8.bbr01.eq01.sjc02.networklayer.com (173.192.18.248) 0.803 ms
4 equinix-ix.sjc1.us.voxel.net (206.223.116.4) 0.741 ms 0.794 ms 0.868 ms
5 173.231.161.193 (173.231.161.193) 48.109 ms 44.392 ms 44.397 ms
6 0.te1-2.tsr1.dal2.us.voxel.net (107.6.99.249) 79.040 ms 79.065 ms 80.685 ms
7 0.te2-2.tsr3.iad1.us.voxel.net (173.231.161.189) 78.472 ms 78.438 ms 78.414 ms
8 173.231.160.90 (173.231.160.90) 81.183 ms 81.158 ms 173.231.160.82 (173.231.160.82) 78.348 ms
9 internap.iad1.us.voxel.net (173.231.161.74) 78.815 ms 78.794 ms 78.751 ms
10 border2.te4-1-bbnet1.wdc012.pnap.net (216.52.127.39) 78.455 ms border2.pc2-bbnet2.wdc012.pnap.net (216.52.127.76) 78.282 ms 78.261 ms
11 net2ez-3.border2.wdc012.pnap.net (64.94.31.142) 79.218 ms 79.159 ms 79.094 ms
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
Network just drops out here is Media Temples response.
We have received feedback from our provider's network engineer and he asked if your clients or the customers of softlayer open a ticket with softlayer. From what testing they have done it appears that there are some routing peering issues between various ISPs and softlayer.
New update
From your traceroutes the commonality appears to be either Softlayer, Internap or NetDC ISPs. I've spoken with NetDC and they claim they are not filtering anything. We have also reached out to softlayer but have not heard back from them. What would help narrow this down if we can run another script like you did earlier:
Another update
Ok, I've reached out to Softlayer (your customer's ISP), Internap (Intermediary ISP), and NetDC (Intermediary ISP). One of them is filtering things, and I'm hoping I get a quick response from them. It is out of our control at this moment, but hopefully these ISPs will be responsive and allow us to track down whatever is blocking your customers from reaching you.
Latest update
Good news, I got a response from Softlayer just as I sent you that last correspondence. They are seeing the same issue from their endand they believe the issue is within Internap. They have looped in Internap as well since they are direct customers of Internap so hopefully something will be discovered.
I was able to determine that there are some ip subnets of ours that are accessible by the Softlayer router that was having issues connecting to your IP. I need to verify that we can actually migrate you to one of those IPs.
I'm having an internet routing change made so Internap will hopefully not have the preferred route to your server and thus be bypassed, I believe the issue is not limited to just your IP which is why I'm taking this more extreme step.
Can someone please help
FINAL UPDATE.
The final solution was to completely move away from Media Temple they have REALLY let us down with their hosting after 3 weeks of our customers having issues they could not provide a solution and when asking them this clear question.
how come i can post to any other server but media temple from the ip thats having the issue?
Even the people on support where just literally just transferring me to another support operative because they could not answer the question.
What worries me is how come a hosting provider cant fix something like this? i lost quite a few paying customers due to this issue.
Solution was to move everything over to Amazon and install it onto a EC2 instance which i managed to do in a day which i wish i had done from the start i also install a ssl cert for $3 whereas the one on Media Temple are $75, i also paid $100 for their Cloudtech service as well.
Great thing about Amazon is even if a error like this did happen again i can just create a AMI and then just fire up a new EC2 and job done.
Everyone be careful about issues like this with third party hosting if you service relies on a API and you are given ISP issue then your business can go under very quickly if your hosting cannot find you a solution after 3 weeks of try belief me your users don't want to hear (oh i am just awaiting a response from Media Temple).
So solution use Amazon Web Service flexible powerful and fully in your control, rant over
Media Temple let us down massively.