DynStatus

Bandwidth Provider Maintenance in Palo Alto, CA

On November 25, between 0800-0845 UTC, one of our bandwidth providers will be performing maintenance on their router in our Palo Alto, CA data center. As a result users may experience degraded performance as connections that would normally come in over this provider are routed over other paths. For Dynect traffic management customers, changes to your US West site definitions are not necessary as this traffic will be routed to other anycast sites.

Dynect DNS and monitoring services at this site will continue to function.

Bandwidth Provider Maintenance in Newark, NJ

On November 22, between 0400-1200 UTC, our bandwidth providers will be performing maintenance on our connections in our Newark, NJ data center. As a result users may experience degraded performance as connections that would normally come in over this provider are routed over other paths. For Dynect traffic management customers with US East regional definitions a small percentage of responses may return your global pool. We do not recommend any changes to your GSLB regional settings.

Bandwidth Provider Maintenance in Palo Alto, CA

On November 19, between 0700-1500 UTC, our bandwidth providers will be performing maintenance on our connections in our Palo Alto, CA data center. As a result users may experience degraded performance as connections that would normally come in over this provider are routed over other paths. For Dynect traffic management customers with US West regional definitions a small percentage of responses may return your global pool. We do not recommend any changes to your GSLB regional settings.

Dynamic DNS Domain dyndns.ws Offline

Due to some troubles with the registration of the domain “dyndns.ws”, this zone is currently not resolving in Dynamic DNS. We are working with our registrar to resolve the issue.

Update: The delegation for “dyndns.ws” has been repaired. The domain is now resolving.

Degraded Performance in Chicago

Starting at approximately 1530 UTC today we experienced routing issues with one of our bandwidth providers in our Chicago datacenter. Our Operations staff took measures to work around the issue until it could be resolved. This had an impact on Dynect DNS services as some traffic was routed to datacenters in other regions.

By 1630 UTC, regular service was restored.

Degraded DNS Resolution from Hong Kong

Due to a routing issue with an upstream provider, which began at approximately 1725 UTC, users may notice a degraded performance in DNS resolution from our Hong Kong data center. We apologize for any inconvenience, our network operations group is communicating with this provider in order to resolve the issue.

Update: As of 1830 UTC, routing issues upstream have been resolved.

Degraded Performance in Hong Kong

Due to a routing issue with an upstream provider, which began at approximately 1725 UTC, users may notice a degraded performance in DNS resolution and monitoring from our Hong Kong data center. We apologize for any inconvenience, our network operations group is communicating with this provider in order to resolve the issue.

Update: As of 1830 UTC, routing issues upstream have been resolved.

mydyndns.org DDOS

We are currently experiencing a distributed denial of service (DDOS) against the MyDynDNS nameservers. This attack began at approximately 0900 UTC and our operations team is working on mitigation.

Update: Mitigations are in place. Operations reports that at no time did DNS resolution fail.

Update: It has come to our attention that mitigation efforts have caused resolution issues in some cases. We are working with our upstream provider to resolve this. DNS from mydyndns.org nameservers continues to resolve as traffic over our other providers is not affected.

Update: We have worked with our upstream provider to address the previous issues. For approximately 40 minutes a filter applied to assist with the mitigation was too restrictive causing DNS to not resolve on one of our IP links. We continue to monitor the situation.

MailHop IPs on Spamhaus XBL

Some of the IP addresses for our MailHop Relay, Forward, and BackupMX servers have been added to the Spamhaus XBL blacklist because they appear in the CBL blacklist. We have requested delisting from CBL and the are in the process of doing so. Once Spamhaus checks the CBL again, the servers should be removed.

Unfortunately we were given no warning from CBL and have been given no information as to why our servers appeared on their blacklist. We are working to get that information so we prevent this from happening in the future.

Update: Immediately after the original status posting, Spamhaus removed the IPs from their XBL. We will continue to investigate and try to work with CBL to prevent this from happening in the future.

1520 UTC Update: Currently, we are still listed in the CBL via the XBL and are continuing to work with CBL to get our addresses delisted; however our attempts to contact them via e-mail have not yielded a resolution. Please note that if you are a BackupMX or Relay customer and are experiencing issues with this DNSBL, you should use a whitelist to guarantee delivery from our mail server’s IP addresses.

1720 UTC Update: We are awaiting a reply from Spamhaus regarding the situation with being listed in the XBL from the CBL and as of yet, have received no reply from the maintainers of CBL.

DynDNS Customer Support System Maintenance Window

On October 29, 2009 between 2200 and 2359 UTC, we will be making a service-affecting upgrade to our customer support ticketing system. This will prevent us from processing email requests bound for various dyndns.com addresses. Phone-based Premier Support updates will be routed to live support during this period. Email requests will not be lost, but delayed as they are held on another server until the upgrade is complete then processed when the upgrade procedure is completed.

Update at 0002 UTC: We are extending the window period by an additional hour to continue with our maintenance.

Bandwidth Provider Maintenance in Palo Alto, CA

On October 30, between 0600-1400 UTC, our bandwidth providers will be performing maintenance on our connections in our Palo Alto, CA data center. As a result users may experience degraded performance as connections that would normally come in over this provider are routed over other paths. For Dynect traffic management customers with US West regional definitions a small percentage of responses may return your global pool. We do not recommend any changes to your GSLB regional settings.

MailHop E-mail Delivery Delays

Due to an extremely high amount of deferred e-mail being queued on our MailHop Forward, Relay, and BackupMX products, we are seeing intermittent delays for some customer e-mail. We are taking steps to correct the problem, and will update this post at regular intervals with status information. No customer e-mail has been lost, everything will be delivered. This does not affect MailHop Outbound, nor any of our other products.

We apologize for any inconvenience this has caused.

Update: As of approx. 0000 UTC, the queued mail is continuing to drain. We are continuing to monitor the situation, and expect all backlogged mail to be delivered before morning.

Update 2: As of approx. 0300 UTC, the backlogged mail has been delivered and everything is working as it should. Again, we apologize for any inconvenience.

Bandwidth Provider Maintenance in Newark, NJ

On October 18, between 0300-1100 UTC, our bandwidth providers will be performing maintenance on our connections in our Newark, NJ data center. As a result users may experience degraded performance as connections that would normally come in over this provider are routed over other paths. For Dynect traffic management customers, changes to your US East site definitions are not necessary as this traffic will be routed to other anycast sites.

Dynect DNS and monitoring services at this site will continue to function.

Internet Routing Issues

At approximately 1915 UTC today routing issues involving AS 701 (MCI Communications Services, Inc. d/b/a Verizon Business) resulted in some of our sites being unavailable to part of the Internet. The problem appeared to be localized to the northeastern United States. Due to the diversity of our connections to the Internet and hosting locations, we successfully routed around the outage.

As of approximately 1950 UTC, the issue appears to have been resolved, and connectivity restored.

Internet Routing Issues

At approximately 1915 UTC today routing issues involving AS 701 (MCI Communications Services, Inc. d/b/a Verizon Business) resulted in some of our sites being unavailable to part of the Internet. The problem appeared to be localized to the northeastern United States. Due to the diversity of our connections to the Internet and hosting locations, we successfully routed around the outage.

As of approximately 1950 UTC, the issue appears to have been resolved, and connectivity restored.

Dynect Platform Monitoring Degradation

As of 0900 UTC, we’ve been made aware of degradations of service with our Dynect Platform Monitoring service. Our teams are actively investigating the issue.

Update: During a DDOS, our monitoring agents were unable to properly report their results. This resulted in some monitoring services reporting incorrect results. After mitigation of the attack we were able to resynchronize the monitoring services.

As of 1015 UTC all monitoring services have returned to normal. We apologize for any false negatives that may have been reported during this time.

Update 2: During a DDOS, from approximately 0840 UTC to 0930 UTC DNS responses from 2 of our nameservers were impacted. Our other nameservers were not impacted at all, and responded flawlessly during this time.

All DNS responses are back to normal at this time.

MyDynDNS DDoS

We are currently experiencing a distributed denial of service against the MyDynDNS servers. We are working on mitigating the the attack.

Update: Our teams are actively mitigating this attack. At this time, ns1.mydyndns.org and ns2.mydyndns.org are offline.

Update: At this time, all nameservers are back online. Our teams have the situation under control but continue to mitigate the attack.

Statistics Reporting Maintenance Window

On Monday, Sept 28, 2009 from 1330 UTC to 1400 UTC we will performing the final stage of our Statistics and Reporting upgrades. During this maintenance window there will be a brief period of time that graph retrieval via the web interface will be unavailable.

We apologize for any inconvenience.

Bandwidth Provider Maintenance in Ashburn, VA

On October 1, between 0700-1000 UTC, one of our bandwidth providers will be performing maintenance on their router in our Ashburn, VA data center. As a result users may experience degraded performance as connections that would normally come in over this provider are routed over other paths. For Dynect traffic management customers, changes to your Ashburn, VA site definitions are not necessary as this provider does not handle anycast traffic.

Dynect DNS and monitoring services at this site will continue to function.

Bandwidth Provider Maintenance in Palo Alto, CA

On September 28, between 1000-1400 UTC, one of our bandwidth providers will be performing maintenance on their router in our Palo Alto, CA data center. As a result users may experience degraded performance as connections that would normally come in over this provider are routed over other paths. For Dynect traffic management customers, changes to your Palo Alto, CA site definitions are not necessary as this provider does not handle anycast traffic.

Dynect DNS and monitoring services at this site will continue to function.