BASS uncovers Anomalous PLDT Routing » Manila Bulletin Technology

Manila Bulletin Philippines

Breaking News from the Nation's leading newspaper

Tempo

Online Newspaper

Showbiz and Celebrity News

Sports News

World News
News Asia

The Nation's Leading Newspaper

Friday, May 26, 2017 27° Rain

BASS uncovers Anomalous PLDT Routing

By: Wilson Chua

Updated

 

A-Ar Andrew Concepcion was the first to notice this. While testing the BASS (Bandwidth and Signal Strength) analyzer, Andrew noticed that he got unexpectedly low bandwidth speeds. BASS measures the bandwidth that a user actually receives on his mobile phone. This was strange.  It did not make sense to get low speeds. We were expecting higher speeds since he was using PLDT to fetch a file at DOST servers in PHOpenix.

Note: PLDT is directly linked to the Department of Science and Technology (DOST) via high speed fiber optic facilities to the DOST’s Philippine Open Internet Exchange (PHOpenIX)”.[1]  Conceptually, this is what was supposed to be happening:

1

  1. BASS requests a file from mirror.pregi.net
  2. PLDT sends the request over to PHOpenIX
  3. PHOpenix will notify the file server
  4. File Server (mirror.pregi.net) will answer the reques t and send the file.

The request should take around 20 to 30 milliseconds.

However, our investigation showed that PLDT was not using this direct link. PLDT was anomalously re-routing this traffic overseas to the US and then back to the Philippines. This action added 1200% more delay against the original (30 vs 399)

2

Why is this finding important?

This ‘smoking gun’ proves that the slow internet is NOT due to “user-abuse” as is often claimed by carriers. Neither is it due to lack of cell sites/towers.  At least in this instance, this is solid proof that PLDT is at fault for “inserting” delays to its own network.

If PLDT had used the direct fiber  link, their end users would have faster and more reliable internet. It is also a lot CHEAPER for PLDT to do so.  Not surprisingly, BASS results as of April 18, 2017 shows that Globe outperforms PLDT:3

But….why would PLDT use a longer more expensive route?

Why would PLDT want to use a longer route that adds more delay to its users? Why not use its direct link with PHOpenIX. This PHOpenIX peering link is cheaper, faster and helps decongest PLDT’s internet gateway. It does not make economic and PR sense. It also diminishes PLDT’s shareholder value.

PHOpenIX’s Bayani Benjamin R. Lara had this to say: “The BGP session between DOST and PLDT is up ever since the MOA signing, and they are advertising their routes to it. Perhaps not all of their routes? Or a policy route config on some router somewhere in their network prefers transit rather than peering routes”.

Jesus Romero, a well respected industry veteran shares: “The PhOpenIx connection is only available to connect to Government servers hosted by PLDT and accessed from other networks. Am not sure if it works vice versa”. He adds further: “If you want to access the PLDT (and subsidiary) networks, you need to be connected to PHIX which is a paid exchange

Other possible reasons might include:

  1. The route configuration was done by a newbie.
  2. The route configuration was done by a master who just had a bad day.
  3. There is someone out to embarrass / “Sabo” PLDT top brass.
  4. It’s a “business decision”.

PLDT’s statement on its website:
“The PLDT Group fully supports DOST and its landmark Philippine OPEN IX initiative, and its vision to operate an internet exchange that will interconnect local internet and data and service providers,” said PLDT EVP and ePLDT President and CEO Ernesto Alberto.”

Having known Mr Eric Alberto to be a straight shooter, I’m sure that this “routing anomaly” will be resolved soon. I am proud of the volunteers behind BASS. This discovery came within the first 15 days of BASS deployment.  Imagine the possibilities #ParaSaBayan

Please share, join, comment or forward.

UPDATE: Read PLDT’s Head of Public Affairs’ answer to this article.

Related Posts

  • deadswitch

    NSA of USA

  • Adrieyl Jezrahiah

    hope there’s a pcap for this findings.

    • Wilson Chua

      Yes we saved the screenshot of the traceroute from a PLDT client to the mirror.pregi.net server. a Pcap is overkill :)

  • Ronald Gonzales

    TSK TSK MANNY V. FOR MONEY KA LANG TALAGA PAHIRAP KA SA BAYAN!

  • Larry Silva

    Just like putting a cheap wine on a vintage bottle.

    I believe there are problems with PLDT’s much vaunted fiber interconnection

  • Quicksilver

    Down oligarchs

  • sassan

    Other possible reasons might include:

    The route configuration was done by a newbie. – nope. Basic networking… a noob should know how distance and routing affect speed of data transmission.

    The route configuration was done by a master who just had a bad day. – possible..

    There is someone out to embarrass / “Sabo” PLDT top brass.- possible.

    It’s a “business decision”. – let me edit this to it’s a “personal business decision”

    ….now we move on to how much commissions are given to people who’d be able to find clients for service providers….

    • siraniksu

      the newbie they meant is not the newbie that doesn’t know how distance and routing effects nor basic networking… since you can’t get the job position if you don’t know what you’re dealing with …

      this is more like a business decision for me … in my opinion so probably their network admin probably knows the problem but didn’t made any “fix” for it and zip your lip because top management said so …

      • sassan

        ah silly me.. of course… seniors and superiors would always win esp if you are a noob….

        what i meant though with personal business decision is somewhat gauging the same path as yours.. only that “probably” someone from above had an incentive for having and keeping the US server in the entire network.

  • George Caballero

    Hindi lang PH to PH yung may delay nila. Sometimes when I traceroute servers to Singapore it will go first to US then Singapore instead of directly Singapore. It also applies to some servers in nearby Asian countries like Japan and HK. Damn it.

    I hope they will notice this article and take action. :(

  • My best guess is that it’s part of hte US’s global surveillance. It routes traffic back to the US for the NSA to scoop up the data and mine it.

    • This

    • Highly probable!

    • Guchi S

      Nope, too obvious for NSA work. Think about this, they basically have the fund $$$ to hire and/or train the best technical minds in the world. If there wasn’t whistle blowers from the inside, the world would have never known their techniques and that they’re doing world wide surveillance in the first place.

      • dutdust

        Guchi well said .. this is too amateur for it to be an over complicated topic.. jukso naman simpleng rerouting lng saka ibang set ng IP iisipan na nila ng kung ano ano.

    • dutdust

      ughh wag nyo gawing komplikado … pag ka basic ng ginawa eh.. mali lng range ng IP nyan. Eto latest statement oh http://technology.mb.com.ph/2017/04/21/pldt-responds-to-anomalous-routing/. saka yung packet size baka di matchy sa mtu at bandwidth .

      • Timothy Manito

        I wonder what their route-map configuration is from PLDT’s side, if they are permitting all or they are filtering it. Routing age of a prefix will only start if PLDT permits it.

        As far as I know BGP does not use MTU and BW for best path selection.

        • dutdust

          Maling Ip nga daw hahahaha nasa bagong statement. LMAO

  • Leo Monera

    Malaki effect sa gaming. Kaya pala taas ng latency ng PLDT users.

  • flipballer02

    Di naman talaga si MVP me ari ng PLDT, SMART at iba pang companies, DUMMY lang sya ng Indonesian magnate

  • Imran

    US spying?

  • Wilson Chua

    The suspicion that PLDT is deliberately doing this isnt new. Check out:
    http://forum.lol.garena.ph/showthread.php?50649-ISPs-in-Philippines-the-real-facts-and-why-you-should-care

    • Hydrokat Bacani

      It even dates back way beyond that, sir. Back when garena didn’t have LoL, they had this VPN kind of thing to play LAN games with other people from other places. They had to set up different servers depending on your ISP since you can’t play with people with other ISP because of the lag (from 200-more than 800ms). Routing is that bad before. You even have a better latency playing in a Singapore server than in the Philippines server.

      And yes, globe is indeed better. However, for heavy users, capping is a turn-off.

  • Daves Zandueta

    Thanks for this article, sir. Makes me wonder if this is happening to other users in other areas.

    And now I hesitate yet again whether or not to sign up with PLDT, but only because one of their offerings meets my needs.

    • Wilson Chua

      Once we have more data points, our network bandwidth maps would be a big help for subscribers like you that wish to sign up for internet service.

  • PDI_ONLY

    first, PLDT being the biggest and long-running telecom player in the Philippines, had their statement i would consider as word-play or beating around the bush just so they would get the public pacified whether their leaders know this or not. very common PR strategy but i won’t buy it. no where in their statement that they are assuring fix of this. well, business as usual for the big boys :)

    as for the routing, i can only guess because of the limited bandwidth that they have between PLDT and DOST, they only allow certain special traffic to go through their peering and everything else is defaulted out to international peerings. so basically our traffic is a spill over, hence thrown over to a trash bin in the US or somewhere else :) whatever the case maybe, the leadership of PLDT has to be blamed. and why the US you ask? try to be creative :)

    as a paying customer, i would just need their apology, assurance that they would fix the issue and the estimated time to repair.

    • Wilson Chua

      PDI_only, they were fast. Fixed at around 10am today

  • Wilson Chua

    Thanks to you all that shared and commented, traceroute data shows PLDT is now directly routing their subscribers to pregi.net using the fiber link as of 10am today. That was fast! Kudos to the guys at PLDT for this.

    https://uploads.disquscdn.com/images/3821467d405058be352f9d95beef1aa4c33f761723b17d5b5d79c33ed5dbeabc.jpg

    • PDI_ONLY

      would this mean that all of the local-based traffic sent and received directly? it would have been nicer had they been more proactive next time. i’m sure they’ve got monitoring tools to detect these.

      • Wilson Chua

        Yes PDI_ONLY. At least for PLDT subscribers to and from the government networks. But not to the rest of the other networks.

        that is another uphill battle that needs to be fought down the line.

    • sassan

      i hope we check it regularly… next thing we know it is back to how it was…
      but i am giving them the benefit of the doubt for now… maybe it was overlooked.LOL

      • Wilson Chua

        Looks like it was overlooked, judging by the speed at which the resolution took place.

      • Hopefully!

        • sassan

          sad to say though that the fix may only reflect on PLDT subscribers to and from the govt networks… other networks are still worth looking into. [see @wilsonchua:disqus’s reply to @disqus_Edj7GzJXpj:disqus below]

      • ting victoriano

        tama, all marketing and PR stunt lang. then ibabalik din.
        sad to see there’s no oce policing them. At least now, more people are aware.

        • sassan

          OR… since it was noticed and published and they realized that the “questionable” configuration apparently affects the government, they fixed it for good..
          but then again, this does not mean there were changes made to the network where consumers’ traffic flow.
          -_- dapat may consumer watchdog ang pilipinas.

    • Antonio Bautista Ibañez Jr.

      https://uploads.disquscdn.com/images/426635d8d82c63a962c2cf9e131355a0d5a7676fb26053efcfaa35c87b51c878.png

      I got below from a non-PLDT provider but I am not sure if you guys could see it a better connection. :)

      • Wilson Chua

        The IP is from converge. Showing off their higher speed low latency link to pregi.net,

        • Raimi

          I am dissatisfied with converge with its poor unstable connection with its undisclosed burstable speeds. I am currently 6 months locked in and wanting to deactivate it without having to pay the expensive disconnection fee. Can you help sir?

          • JamesVMM

            What do you mean by ‘undisclosed burstable speeds’? I’m planning to get their fiberx plan.

    • Sadly our upload rate and disconnection in the internet is still happening here in zamboanga. It’s been 3 weeks already and we feel like we just wasted money on a whole month of disconnection and very unsatisfied internet connection. Dunno if this is also the cause of this “fix” they’re doing or if it’s another problem here in zamboanga or anyone else in the country. PLDT doesn’t really notify it’s consumers and they ought to do it as to help alleviate the anger of it’s consumers.

    • Mikkel Georgsen

      If only they didn’t just change that one route but actually obeyed the law in Philippines, imagine the possibilities.

      http://forum.lol.garena.ph/showthread.php?50649-ISPs-in-Philippines-the-real-facts-and-why-you-should-care

    • Tagpi Neko-Sensei

      For us gamers, this isn’t new. We have been continuously blocked somewhere whenever PLDT makes an upgrade therefore forcing us to use VPNs to connect to our games. Via our traceroute report, GMs always tells that the problem lies with our Internet Service Provider.

      The fix might be true to FIBR users (or should i say this problem never existed at all) but to myDSL customers this is a recurring unfixed problem.

      • Sapu

        Fibr won’t totally help you.

    • GenWill Alava

      good sir, pls. do monitor this telco. we the consumer deserve the best services.

    • Ma RIan

      hahahha. You’ve got them. !!!! You should check it regularly or ask the Philippine Competition Commission, National Telecommunication Commission and Department of Information, Communication and Technology about this Problem. This is in competitive, unreliable lies of telco bringing pain in the ass for the past 20 years. National Telecommunication Act of 1994 or 1995 is the another problem. Well in that law, there is no specific fine so tNTC must use the Public Service Act long way back which fines the public service provider who violates with just a 250 pesos!! So outdated. It’s better to violate the law than lost millions of profit. Overpricing, slow, low coverage, high latency, worst of all. Pls not juts look in the IP address but also to the other aspects like pricing, internet bandwidth, cellsite density and other,s

    • krisaquinomalandimarquez

      Kudos to the guys at PLDT?you stupid?

      • Wilson Chua

        No. Not stupid, just showing appreciation where it is warranted.

      • Wilson Chua

        However, having said that, Krisaquinomalandimarquez (ang haba ng nic handle mo)….you may be on to something. the latency from hop 10 to 12 are not normally associated with a direct fiber link. Thank you for the *hint*. I am just waiting for Pregi.net’s looking glass to be active to confirm a suspicion i have regarding this PLDT fix.

  • GustoKoHappyKa

    lol routed sa NSA/CIA

  • Bungbung Marcos

    Reminds me…
    Anyone remember Globe Tattoo traffic being re-routed to Israel for compression? Or that one time when all of PLDT’s (?) consumer traffic gets routed to Hong Kong first before coming back here?

    • Wilson Chua

      Hmm i didnt know about the compression thingy. Can to share ?

      • Bungbung Marcos

        Globe Tattoo used to have this “myAcceleration” script that shows on your browser in the form of a toolbar (that was making my Anti-Spyware crying all the time).

        Because of that script, any image viewed on the web comes from a specific IP address range. xyz.com/abcd.jpg becomes http://**.*.*.***/xyz.com/abcd.jpg

        Of course, I wasn’t the only one who tried to trace the scary-looking IP address. Turns out it points to Tel Aviv, Israel and is constantly used for image compression.

        • Wilson Chua

          Thanks for that bit of trivia Bungbung.

  • Rene Canlas

    Thanks for the article, Wilson.

    For years now, it’s been well-known to online gamers that performance (ping times) suffers when you have players from Smart connecting to a Globe game server or vice versa. This is because of what you mentioned in your article — no direct interconnection between Globe and Smart so the traffic has to leave the country before it can come back.

    I had high hopes for PH OpenIX when it opened, and was sad to note that PLDT resisted joining for years, claiming that most of their traffic goes out the country anyway so connecting to a local peering exchange was of little benefit, but then they had a “change of heart” and decided to interconnect. I got to talk to a DOST insider, and the reason for this 180-degree turnaround was the fact that PLDT wanted to participate in the National Free Wi-Fi Project of the DOST and one of the terms in the TOR for the project is that peering with PH OpenIX, so it was crafty of DOST to put that clause in the TOR to get PLDT to peer with them. As your article implies, they may be limiting the traffic going through the exchange.

    I suspect the main motivation for this practice is that PLDT needs to provide a certain volume of bandwidth to upstream providers to meet their desired tiered pricing. Obviously the more bandwidth you send over, the lower the transit price per Mb. We can do the math from there. Hope PLDT can clarify their plans for PH OpenIX.

  • praetoriosneil

    additional income nga naman yan sa pldt kung ibebenta nila sa us ang ating online info… personalized advertisements for each pldt sibscriber… galeng!!

  • Te Re

    Just to share my analysis here:
    The high latency resulted must be first confirm whether the forward traffic from PLDT or the return traffic causes the RTT of 399ms. Traceroute would be the best diagnostic tool to determine that.

    Second, PLDT-PH Open IX peering must verify the route policies applied on advertising and receiving the routes.

    • Wilson Chua

      Hi Te Re. Yes, with regards to PLDT-PH OpenIX peering, the route policies were reviewed. And we took a screenshot of the command [sh ip bgp] to make sure that we have documentation.

  • tsioh

    anong BASS uncovers? lumang tugtugin nato dati nato ginagawa ni PLDT since time immemorial. simpleng dos cammand tracerrt lang yan malalaman na.

  • Diane Roger Naldo

    I think they are selling user information without our consent. the law maker should protect us from this company.

  • Fernando Miguel de Cordova

    It’s a US intelligence surveillance project to keep an eye on the country’s political situation, economy and person of interest. It’s not surprising since the US is conducting signal intelligence analysis for years.

  • ferdie san pedro

    same old scenario ….only money needed …damn networks in the Philippines still greed players

  • Hisashi Mitsui

    kaya pala malakas na pldt ngayun dito samin. liit na ng ping. tinalo pa minsa globe lte ko

    • Wilson Chua

      Glad to hear that!!

  • tsioh

    @wilsonchua:disqus try traceroute from PLDT to a local Globe IP. PLDT will bring you first to US then back to Philippines.

    • Wilson Chua

      Hi tsioh. PLDT did say that PLDT to government ONLY. they did not say anything about other networks. Sigh. in a perfect world, PLDT should have been doing multilateral peering at PHOpenIX.

    • Wilson Chua

      PLDT confirms that globe and pldt are directly peered

  • Harvey V

    There should be an independent group that monitors these folks, full time. Thinking of the
    possible reasons that they cited maybe true.

    And I think they have lost most (not all) of their good pool of talents either (retired), moved to a better company (abroad) or as the result of sub contracting to external sources. The ph is full of bright minds that is why foreign companies love doing bussiness with us in terms of IT “coz we know our sh!te” well done guys thanks for checking! Cheers to you! from a #nonITguy

    • Wilson Chua

      That’s why volunteers formed BASS. Check it out.

  • Mikkel Georgsen

    This isn’t exactly new information, it was all detailed here http://forum.lol.garena.ph/showthread.php?50649-ISPs-in-Philippines-the-real-facts-and-why-you-should-care

    This could easily be fixed by the government growing some balls and threaten pldt to either interconnect freely as required by the law or loose their frequency bands…

  • Jay Jimenez

    @wilsonchua:disqus

    I just want to cite a minor correction on the information that you posted above

    “fetch a file at DOST servers in PHOpenix.”

    The site mirror.pregi.net (202.90.159.172) falls under Autonomous Systems 9821 or AS 9821 under the prefix advertised as 202.90.128.0/19 and not on the same network as PHOpenix which is using AS 4779. PLDT is not peering with AS 9821 and it has to check against BGP peers of the shortest possible AS path which should be the PHOPENIX.

    Upon checking AS 9821(DOST where pregi is hosted), none of its peers are being used as transit by PLDT that could reasonably justify 1 AS PATH away from PLDT to DOST(AS9821) therefore the AS path that must have taken might be more than 1 path away to reach DOST from PLDT.

    Can you share the screenshot please? It wasn’t posted in your article.

    The first thing that I have in mind if it really took a different AS PATh (the US route as you mentioned above) instead of using the shorter PHOPENIX route (assuming that DOST has properly advertised its route to PHOPENIX) is through PREPENDING AS PATHS being advertised by PHOPENIX or maybe some policy route configuration.

    On AS path prepending:

    Assuming there are 2 AS PATHS away from PLDT to DOST via international transits and only 1 AS path between PLDT and DOST via PHOPENIX, technically it should route via PHOPENIX because it’s the shortest AS PATH. But due to prepending the AS path to PHOPENIX, it could make it appear that AS paths between PLDT and DOST can be 3 or more AS PATHS away hence the least choice.

    BGP flap:

    Come to think also that there was a bgp flap between DOST(as9821) and phopenix(as4779) causing the prefix 202.90.128.0/19 where pregi is seated not advertised to PHOPENIX during the time of testing.

    just my 2 cents

    Cheers,
    Jay

  • mojo76

    wow thanks for this diagram…..PLDT anong reaction nyo dito

  • Jay Jimenez

    From PLDT to PREGI

    #1 The long distance path
    AS 9299 (pldt) –> AS 701 (verizon US) –> AS 3257 (gtt) –> AS 9821 (pregi)

    #2 The short distance path (the right path)
    AS 9299 (pldt) –> AS4799 (PHopenIX) –> AS 9821 (pregi)

    Why did PLDT route to LONG DISTANCE PATH? because it’s the PHILIPPINE LONG DISTANCE TELCO! LOL!

    Possibilities..

    1. PLDT forgot to add the DOST prefix such as 202.90.128.0/19 (where PREGI is seated) to allowed prefix being advertised by DOST through PHopenIX.. Their BGP policy configuration with phOPENIX is somehow pessimistic (meaning they only add what are allowed prefixes – mostly government networks as per MOA)

    2. AS PATHS to either 4799 or 9821 were BGP prepended (make it appear even longer than #1 above)

    Least possible. DOST forgot to advertise the prefix 202.90.128.0/19 to PHopenIX where PREGI is seated hence routed to GTT (AS 3257)

    • Wilson Chua

      I also did a reverse traceroute from Phoenix. The results seem to show that pldt did not announce their full routes

  • Tony Gennaccaro

    No matter which telco you talk about in the Philippines. From my experience as a foreigner, contacting my family in a major city in Leyte is a nightmare. When I am lucky enough to get through, the system is so slow and the signal so weak the connection drops out continuously. it’s hardly worthwhile. It’s about time a major player such as Telstra joined the competition to wake your pathetic telcos up. Filipinos are slugged way too much for inferior telecommunication services. I recently visited the country and I couldn’t even get internet in my 2nd floor hotel room. The orovider was unfortunately PLDT. Hopeless!

  • DJohnson

    I have been suspecting the routing errors for years now — from a 1990s ccie who could have diagnosed and fixed it in half a day!!!@@

    • Wilson Chua

      Honored to have a feedback from a CCIE. DJohnson!

  • dutdust

    Di match bandwith nyan saka packet size .

  • Shawn

    sorry for noobish curiousity but can you test the other broadband Telecom as well? (Globe & Smart)

    • Wilson Chua

      The bass app can also test the other networks. Just use your mobile to connect to your wifi

  • Ace Jamias

    Isumbong kay president Duterte na yan!!

  • Ma RIan

    Yes

  • Joed Sanchez

    Jeez.. So that’s why I was having a hard time to register my SIP phone to our server the whole damn day yesterday with my pldt connection. I dunno what gives. I was to call pldt tech support but thought it would be just a waste of time. I just tested it now after reading this post and, voila! My softphone is now able to register. Glad it got sorted right away. Good job, you guys.

  • Orlando Solano

    So nobody in Pldt checks the speed of tbeir inernet service regularly?

  • sagbotgamot

    It’s a ‘business decision’ hatched by the oligarchs. Accept it… ’till Duterte finds out!

    • Wilson Chua

      sagbptgamot. Judging from the speed at which they resolved this issue, i think it was an honest mistake. Of course, unless there is evidence to say that PLDT already knew about it long before the article came out.

  • Rabbit

    Hmm. if the server is from US. then the route should be from PH to US then back to PH. i think there is nothing wrong with the said routing.

    the main problem why the PH speed is too slow is because of how far is the server they want to access is. if there is a server near PH then the speed will also increase.

  • Gaudz Pacot

    What I can think of is SOMEBODY IS MAKING A KILLING OUT OF THE RE-ROUTING.
    It was unnecessary and stupid. And the “dummy” knows everything that’s going on…

    • Wilson Chua

      Upvote kita Gaudz. Maybe they will find out who?

  • Xlit
    • Wilson Chua

      Thanks for this reference Xlit!

  • Felipe Soriano

    Just be careful with PLDT especially their promos. They appear to not know what good service is. I quit PLDT line service after more than 20 years of engagement because of bad service and appearance of a better option. If you are home based most of the time, cable DSL if available is better/cheaper than PLDT’s line DSL.

  • JamesFelizardo InstructionalSt

    there’s a possibility that PLDT employs this roundabout (and may use overseas servers) because these may be cheaper in price (and more profits for them). thanks BASS for the exposition. Let’s create more noise online about this.

  • Shoppie Online

    PLDT service is such a disappointment. Home based worker here and the latency is such a burden since we need to connect to foriegn clients and because of that it affects the quality of the audio and video of the medium that we use to connect with our clients. We get low satisfactory evaluation because of this. What hurt most sometimes if we cannot connect due to the poor network condition we lost clients and that is a big loss as well where PLDT cannot even reimburse us because of this very bad service that they have.

  • Wilson Chua

    PLDT’s statement making corrections to this article is refuted by DOST. Read their official statement here: http://www.asti.dost.gov.ph/press-room/asti-statement-phopenix