DynStatus

Degraded Performance in Chicago

Starting at approximately 1530 UTC today we experienced routing issues with one of our bandwidth providers in our Chicago datacenter. Our Operations staff took measures to work around the issue until it could be resolved. This had an impact on Dynect DNS services as some traffic was routed to datacenters in other regions.

By 1630 UTC, regular service was restored.

Degraded DNS Resolution from Hong Kong

Due to a routing issue with an upstream provider, which began at approximately 1725 UTC, users may notice a degraded performance in DNS resolution from our Hong Kong data center. We apologize for any inconvenience, our network operations group is communicating with this provider in order to resolve the issue.

Update: As of 1830 UTC, routing issues upstream have been resolved.

Degraded Performance in Hong Kong

Due to a routing issue with an upstream provider, which began at approximately 1725 UTC, users may notice a degraded performance in DNS resolution and monitoring from our Hong Kong data center. We apologize for any inconvenience, our network operations group is communicating with this provider in order to resolve the issue.

Update: As of 1830 UTC, routing issues upstream have been resolved.

mydyndns.org DDOS

We are currently experiencing a distributed denial of service (DDOS) against the MyDynDNS nameservers. This attack began at approximately 0900 UTC and our operations team is working on mitigation.

Update: Mitigations are in place. Operations reports that at no time did DNS resolution fail.

Update: It has come to our attention that mitigation efforts have caused resolution issues in some cases. We are working with our upstream provider to resolve this. DNS from mydyndns.org nameservers continues to resolve as traffic over our other providers is not affected.

Update: We have worked with our upstream provider to address the previous issues. For approximately 40 minutes a filter applied to assist with the mitigation was too restrictive causing DNS to not resolve on one of our IP links. We continue to monitor the situation.

MailHop IPs on Spamhaus XBL

Some of the IP addresses for our MailHop Relay, Forward, and BackupMX servers have been added to the Spamhaus XBL blacklist because they appear in the CBL blacklist. We have requested delisting from CBL and the are in the process of doing so. Once Spamhaus checks the CBL again, the servers should be removed.

Unfortunately we were given no warning from CBL and have been given no information as to why our servers appeared on their blacklist. We are working to get that information so we prevent this from happening in the future.

Update: Immediately after the original status posting, Spamhaus removed the IPs from their XBL. We will continue to investigate and try to work with CBL to prevent this from happening in the future.

1520 UTC Update: Currently, we are still listed in the CBL via the XBL and are continuing to work with CBL to get our addresses delisted; however our attempts to contact them via e-mail have not yielded a resolution. Please note that if you are a BackupMX or Relay customer and are experiencing issues with this DNSBL, you should use a whitelist to guarantee delivery from our mail server’s IP addresses.

1720 UTC Update: We are awaiting a reply from Spamhaus regarding the situation with being listed in the XBL from the CBL and as of yet, have received no reply from the maintainers of CBL.

DynDNS Customer Support System Maintenance Window

On October 29, 2009 between 2200 and 2359 UTC, we will be making a service-affecting upgrade to our customer support ticketing system. This will prevent us from processing email requests bound for various dyndns.com addresses. Phone-based Premier Support updates will be routed to live support during this period. Email requests will not be lost, but delayed as they are held on another server until the upgrade is complete then processed when the upgrade procedure is completed.

Update at 0002 UTC: We are extending the window period by an additional hour to continue with our maintenance.

Bandwidth Provider Maintenance in Palo Alto, CA

On October 30, between 0600-1400 UTC, our bandwidth providers will be performing maintenance on our connections in our Palo Alto, CA data center. As a result users may experience degraded performance as connections that would normally come in over this provider are routed over other paths. For Dynect traffic management customers with US West regional definitions a small percentage of responses may return your global pool. We do not recommend any changes to your GSLB regional settings.

MailHop E-mail Delivery Delays

Due to an extremely high amount of deferred e-mail being queued on our MailHop Forward, Relay, and BackupMX products, we are seeing intermittent delays for some customer e-mail. We are taking steps to correct the problem, and will update this post at regular intervals with status information. No customer e-mail has been lost, everything will be delivered. This does not affect MailHop Outbound, nor any of our other products.

We apologize for any inconvenience this has caused.

Update: As of approx. 0000 UTC, the queued mail is continuing to drain. We are continuing to monitor the situation, and expect all backlogged mail to be delivered before morning.

Update 2: As of approx. 0300 UTC, the backlogged mail has been delivered and everything is working as it should. Again, we apologize for any inconvenience.

Bandwidth Provider Maintenance in Newark, NJ

On October 18, between 0300-1100 UTC, our bandwidth providers will be performing maintenance on our connections in our Newark, NJ data center. As a result users may experience degraded performance as connections that would normally come in over this provider are routed over other paths. For Dynect traffic management customers, changes to your US East site definitions are not necessary as this traffic will be routed to other anycast sites.

Dynect DNS and monitoring services at this site will continue to function.

Internet Routing Issues

At approximately 1915 UTC today routing issues involving AS 701 (MCI Communications Services, Inc. d/b/a Verizon Business) resulted in some of our sites being unavailable to part of the Internet. The problem appeared to be localized to the northeastern United States. Due to the diversity of our connections to the Internet and hosting locations, we successfully routed around the outage.

As of approximately 1950 UTC, the issue appears to have been resolved, and connectivity restored.

Internet Routing Issues

At approximately 1915 UTC today routing issues involving AS 701 (MCI Communications Services, Inc. d/b/a Verizon Business) resulted in some of our sites being unavailable to part of the Internet. The problem appeared to be localized to the northeastern United States. Due to the diversity of our connections to the Internet and hosting locations, we successfully routed around the outage.

As of approximately 1950 UTC, the issue appears to have been resolved, and connectivity restored.

Dynect Platform Monitoring Degradation

As of 0900 UTC, we’ve been made aware of degradations of service with our Dynect Platform Monitoring service. Our teams are actively investigating the issue.

Update: During a DDOS, our monitoring agents were unable to properly report their results. This resulted in some monitoring services reporting incorrect results. After mitigation of the attack we were able to resynchronize the monitoring services.

As of 1015 UTC all monitoring services have returned to normal. We apologize for any false negatives that may have been reported during this time.

Update 2: During a DDOS, from approximately 0840 UTC to 0930 UTC DNS responses from 2 of our nameservers were impacted. Our other nameservers were not impacted at all, and responded flawlessly during this time.

All DNS responses are back to normal at this time.

MyDynDNS DDoS

We are currently experiencing a distributed denial of service against the MyDynDNS servers. We are working on mitigating the the attack.

Update: Our teams are actively mitigating this attack. At this time, ns1.mydyndns.org and ns2.mydyndns.org are offline.

Update: At this time, all nameservers are back online. Our teams have the situation under control but continue to mitigate the attack.

Statistics Reporting Maintenance Window

On Monday, Sept 28, 2009 from 1330 UTC to 1400 UTC we will performing the final stage of our Statistics and Reporting upgrades. During this maintenance window there will be a brief period of time that graph retrieval via the web interface will be unavailable.

We apologize for any inconvenience.

Bandwidth Provider Maintenance in Ashburn, VA

On October 1, between 0700-1000 UTC, one of our bandwidth providers will be performing maintenance on their router in our Ashburn, VA data center. As a result users may experience degraded performance as connections that would normally come in over this provider are routed over other paths. For Dynect traffic management customers, changes to your Ashburn, VA site definitions are not necessary as this provider does not handle anycast traffic.

Dynect DNS and monitoring services at this site will continue to function.

Bandwidth Provider Maintenance in Palo Alto, CA

On September 28, between 1000-1400 UTC, one of our bandwidth providers will be performing maintenance on their router in our Palo Alto, CA data center. As a result users may experience degraded performance as connections that would normally come in over this provider are routed over other paths. For Dynect traffic management customers, changes to your Palo Alto, CA site definitions are not necessary as this provider does not handle anycast traffic.

Dynect DNS and monitoring services at this site will continue to function.

Emergency Server Maintenance in Palo Alto, CA

On September 21 at 2100 UTC we will be performing emergency server maintenance in our Palo Alto datacenter. This will partially impact our DynTLD services, DNS resolution of our corporate domains, and the checkip service. These services will continue to be served by our other datacenters. Expected downtime is less than 5 minutes. All other services will be unaffected.

Update: As of 2102 UTC the maintenance has been completed and all affected services are operating normally.

Dynect Traffic Management Interface Upgrade

During our regular maintenance window on Tuesday Sept 22, 2009 (0900 UTC) we will be releasing an updated version of our Traffic Management user interface which makes a change in the ruleset definition interface.

We will be replacing the ruleset definitions from location specific to regional definitions. The new user interface will display the following regions.

• US East (combines Newark and Ashburn rulesets)
• US Central
• US West
• Asia
• EU West
• EU Central
• EU East

What does this mean to you? If you have defined your Traffic Management Service with just a single Global Ruleset you will see no changes in the way the current service works.

For the regional location which will be combining two existing city locations the results will be that those rules will be combined into a single ruleset. If both cities have matching rulesets you will not notice any change in the way DNS is handled.

This modification to the Global ruleset definitions will provide you with easier ruleset definitions as we add more individual locations to our network.

Routing Reconvergence in Amsterdam 2

At approximately 23:45 UTC, a routing event in our Amsterdam 2 point of presence caused BGP routing re-convergence, creating a 1 minute degradation of service for ns2.pXX.dynect.net and ns4.pXX.dynect.net. Our Operations team quickly investigated the issue and isolated the problem to upstream service provider maintenance, and has mitigated the issue. Traffic in this region was gracefully handled by our ns1.pXX.dynect.net and ns3.pXX.dynect.net nodes.

Dynect Statistic Reporting Maintenance Outage

Between 1300 and 1900 UTC on September 14, 2009, we will be performing maintenance on the servers for our statistics reporting. During this time reporting and graphing of Dynect statistics will be unavailable.

We will continue to be collecting the data for statistical reporting, there will be no loss of query information. For a short period following this maintenance window, statistical reporting will be delayed as the server processes the backlog of queries from that time.

No other services will be effected during this event.

Update: Data transfer for this maintenance window is taking longer than expected. We are extending this Statistics Reporting Outage until 0000 UTC Sept 15, 2009.

Update 2: As of 22:40 UTC Sept 14, 2009 this maintenance window has closed. The statistics server is available again. There will be a short time over the next several hours while the query logs are processed that the data presented will be incomplete.