Thinktel Support Notifications

UPDATE - One-way and No Audio Issues

Thu, 17 May 2012 19:32:40 MDT

Our switch engineers have identified a potential issue with our media gateway DSPs and have restarted the suspect adapter. Existing calls were unaffected during the reboot as they would have been picked up by the backup card. As of yet, test calls have been unable to reproduce the issue. We will be contacting our switch manufacturer to further investigate the cause of the issue to ensure that this issue does not resurface.

Please report any new one-way and no audio occurring after 9PM EDT tonight (May 17).

Thank you for your patience as we worked to resolve this critical issue.

Sincerely,

ThinkTel Technical Support

One-way and No Audio Issues

Thu, 17 May 2012 14:27:34 MDT

We are currently experiencing intermittent one-way and no audio issues. Calls affected are those that involve our Toronto media gateways and is not specific to any one proxy or rate centers, however, calls within Eastern Canada are likely to be more affected. Calls to our support numbers may also experience one-way audio.

Our engineers are actively working on this issue and there is no estimated resolution time available.

If you would like to report this issue, please call us at (866) 928-4465 or use our support forms located on our support website. Be sure to include relevant call examples with your trouble report.

We are sorry for any inconvenience that this may be causing.

Sincerely,
ThinkTel Technical Support

Scheduled Maintenance - Support Site

Tue, 15 May 2012 17:29:52 MDT

Date/Time: Wednesday, May 16, 2012 00h01 ET to 05h00 ET

Description of Maintenance: As per our previous email communication advising you of our support site maintenance and redundancy upgrades, we will be performing subsequent maintenance tomorrow morning.

Customer Impact: During the maintenance window our support website, email responses and knowledgebase will be unavailable. All emails received will be queued and you should receive an auto-response following the conclusion of the maintenance. Certain web forms will also be unavailable, for example the online porting form.

We apologize for any inconvenience during this activity. We are still available by telephone at (866) 928-4465 should you need to report a trouble issue.

Sincerely,

ThinkTel Communications Ltd.

ThinkTel Advisory - Email Issues Update

Thu, 10 May 2012 10:29:52 MDT

Our email server issues have been resolved. As such all new email, and queued email, is now being received and processed.

Thank you for your patience as we worked to resolve this issue.

-ThinkTel Technical Support

ThinkTel Advisory - Email Issues

Thu, 10 May 2012 09:50:28 MDT

Dear Distributel/ThinkTel Customers:

This advisory is to notify you that we are currently experiencing issues with our mail system. As a result, support tickets sent to us via email are not being received at this time. Auto-responses are also not being sent if you are reporting issues by filling out our support form on our support website.

Alternatively, you can report an issue to us by filling out the form at http://support.thinktel.ca/index.php/contact or by calling us at (866) 928-4465.

This is a temporary issue and will be resolved shortly. All emails that have been sent are cached and will be processed once our IT department resolves this issue.

We apologize for any inconvenience and appreciate your understanding.

Sincerely,

ThinkTel Communications

Scheduled Network Maintenance

Wed, 09 May 2012 19:52:51 MDT

Network Notification – Emergency Maintenance Activity

Date/Time: Friday, May 11, 2012 02h00 ET to 04h00 ET.

Expected Outage Time: 5 minutes during the maintenance window.

Description: We will be moving the TorIX link from the existing router to our new core router.

Services Affected: Internet traffic routing through the TorIX hub.

Customers Affected: Commercial and residential customers.

Please be assured that all efforts are being made to minimize downtime during the scheduled activity. If you have any questions or have issues that extend beyond the maintenance window, please contact our technical support at 877-564-7660 (english) or 877-540-9869 (french).

Update: Scheduled Maintenance - Ticketing System

Sun, 06 May 2012 07:16:50 MDT

Our ticketing system and support sites are now back online. There are additional maintenance activities related to the system that will be conducted at a future date. We will advise again once that date has been determined.

We thank you again for your patience.

Sincerely,

ThinkTel Communications Ltd.

Scheduled Maintenance - Ticketing System

Fri, 04 May 2012 17:34:59 MDT

This communication is to advise of system maintenance that will be taking place on Sunday, May 6th from 00:01 to 12h00 ET involving our ticketing system and related sites.

An email notification with all of the details has been sent to all ThinkTel customers.

Another notification will be posted to this RSS feed when the system is brought back online.

We thank you for your understanding and patience during this activity.

Sincerely,

ThinkTel Communications Ltd.

Scheduled Maintenance - Toronto CLEC Interconnection

Mon, 23 Apr 2012 17:02:08 MDT

We will be performing scheduled maintenance to increase the capacity of our Toronto CLEC interconnection circuit.

Date: 4/27/2012

Start Time: 00:01 ET

End Time: 06:00 ET

We do not anticipate any outage during this maintenance. However, as traffic fails over to backup links there will be several switching hits.

Please contact our support line at (877) 564-7660 or support@thinktel.ca if you experience any issues outside of the maintenance window.

Scheduled Maintenance - Toronto Metaswitch

Mon, 23 Apr 2012 16:57:50 MDT

As part of our action plan and continuing efforts to increase the capacity and robustness of our network, we will be performing scheduled maintenance to our Toronto Metaswitch and Metasphere platforms.

Date: 4/26/2012

Start Time: 00:01 ET

End Time: 06:00 ET

No outages are expected during this maintenance. However, as traffic fails over to backup facilities and interfaces there could be minor delays associated with call setup and post-dial delay (PDD).

Please contact our support line at (877) 564-7660 or support@thinktel.ca if you experience any issues outside of the maintenance window.

Network Maintenance - Increased capacity to some LIR CLEC regions

Mon, 23 Apr 2012 16:44:01 MDT

Please be advised that we will be performing maintenance to upgrade capacity in some of our LIR CLEC regions.

Date: 4/25/2012

Start time: 00:01 ET

End time: 06:00 ET

The affected rate centers that will be impacted during the upgrade are as follows:

Windsor, Chatham and Sarnia will experience a 10 minute outage affecting incoming calls.

Please contact our support line at (866) 928-4465 or support@thinktel.ca if you experience any issues outside of the maintenance window.

ThinkTel Support Site Maintenance

Mon, 23 Apr 2012 16:41:18 MDT

As part of our action plan to enhance and improve specific support resources we will be performing maintenance on our support website on Tuesday, April 24 starting at 00:01h EDT (midnight). This maintenance period is being used to prepare for adding redundancy to the support site to reduce the chance that it is unavailable due to internal network issues.

The expected downtime is 1 hour and our support website will be back online at 1:00AM EDT. No other support resources should be affected during this maintenance period.

If you experience any issues with our support website once this maintenance has concluded, please call our support team at (866) 928-4465.

Update to Important Changes: New audio IPs

Fri, 13 Apr 2012 17:10:55 MDT

Dear Customer,

In response to feedback that we've received, we have decided to shift the changes scheduled to RTP IP addresses from April 13th to April 27th. Early next week, we will provide you with a simplified list of IP addresses that will need to be allowed by your company's firewall. Keep in mind that these changes are necessary in order to improve service reliability, redundancy and performance.

We thank you for your understanding and apologize for any inconvenience. If you have any questions or concerns, please contact your Account Manager or our Technical Support line at 1-866-564-7660.

Thank you,

ThinkTel Technical Support

Update to Important Changes: New audio IPs (Action Required) / Changements Importants: Nouvelles adresses IP pour l'audio (Action Requise)

Fri, 13 Apr 2012 16:47:59 MDT

Dear Customer, In response to feedback that we’ve received, we have decided to shift the changes scheduled to RTP IP addresses from April 13th to April 27th. Early next week, we will provide you with a simplified list of IP addresses that will need to be allowed by your company’s firewall. Keep in mind that these changes are necessary in order to improve service reliability, redundancy and performance. We thank you for your understanding and apologize for any inconvenience. If you have any questions or concerns, please contact your Account Manager or our Technical Support line at 1-866-564-7660. Thank you, ThinkTel Technical Support --------------------------------------------------------------------------------------------------------------------------------------------- Cher Client, En réponse aux commentaires reçus, nous avons décidé de déplacer la date de changement des adresses IP RTP du 13 avril au 27 avril. Au début de la semaine prochaine, nous allons vous faire parvenir un liste simplifiée des adresses IP qui devront être permis par le Pare-Feu/Firewall de votre entreprise. Ces changements sont nécessaire afin d’améliorer la stabilité, la redondance et la performance de nos services. Nous vous remercions pour votre compréhension et nous nous excusons pour les inconvénients encourus. Si vous avez des questions ou si vous éprouvez le moindre souci, s’il vous plait contactez votre Gestionnaire de Dossier ou notre ligne de Support Technique au 1-866-564-7660. Nos remercîment les plus sincères, Thinktel, Support Technique

Important Changes: New audio IPs (Action Required) / Changements Importants: Nouvelles adresses IP pour l'audio (Action Requise)

Mon, 26 Mar 2012 16:35:03 MDT

Subject: Important Changes: New audio IPs (Action Required)

Dear ThinkTel Customer,

This important message is to advise of upcoming enhancements to RTP addresses within our network that may impact your service as it relates to the treatment of audio. The change is required to improve and accommodate increases in audio handling.

Two new DS3 blades are scheduled to be activated in the coming weeks which will require the activation of new RTP IPs. As a result of this change, you will be required to ensure that your firewall(s) are open to allow future traffic and routing from the new IPs.

Effective Friday, April 13th, 2012, please ensure that your firewall(s) allow the following range of IPs:

- 159.18.161.96/27 (159.18.161.97 - .126)

- 208.68.17.0/25 (208.68.17.1 - .126)

- 206.80.250.96/27 (206.80.250.97 - .126)

- 206.80.250.0/27 (206.80.250.1 - .30)

Note: Customers currently peered to our EICO proxy (eico.thinktel.ca/208.68.17.144) may experience a greater impact as a result of the change. Users of EICO are likely to see audio from any of the four IP ranges mentioned above.

A reminder closer to the date will be posted on our support site (http://support.thinktel.ca) and RSS feed (http://support.thinktel.ca/index.php/announcements). We encourage you to subscribe to http://ucontrol.thinktel.ca/rss to ensure that you receive the update along with future advisories.

Please direct any questions or concerns to ThinkTel Support by submitting a ticket at http://support.thinktel.ca or by calling 1-877-564-7660.

Thank you,

ThinkTel Communications

Sujet : Changements Importants : Nouvelles adresses IP pour l’audio (Action Requise)

Chers client ThinkTel,

Ce message important est pour vous aviser des améliorations à venir pour les adresses RTP au sein de notre réseau qui pourraient impacter vos services, en ce qui a rapport au traitement de l’audio. Ce changement est nécessaire à fin d'améliorer notre service et de répondre à l’augmentation de demande pour le traitement audio.

Deux nouveaux serveurs DS3 seront activés dans les semaines à venir, ceci nécessitera l’activation de nouvelles adresses IP RTP. En conséquence de ce changement vous devrez vous assurer que votre pare-feu est ouvert pour permettre la circulation et le routage à partir des nouvelles adresses IP.

À compter du vendredi le 13 avril 2012, s’il vous plaît vous assurer que votre pare-feu permet les plages d’adresses IP suivantes :

- 159.18.161.96/27 (159.18.161.97 - .126)

- 208.68.17.0/25 (208.68.17.1 - .126)

- 206.80.250.96/27 (206.80.250.97 - .126)

- 206.80.250.0/27 (206.80.250.1 - .30)

Note : Ce changement pourrait avoir un impact plus profond sur les clients actuellement connectés à notre « proxy » EICO (eico.thinktel.ca/208.68.17.144). Les clients connectés à EICO verront probablement de l’audio provenant des quatre plages d’adresses IP mentionnées ci-dessus.

Un rappel sera affiché sur notre site de soutien (http://soutien.thinktel.ca) ainsi que notre flux RSS (http://support.thinktel.ca/index.php/announcements). Nous vous encourageons à vous abonner à http://ucontrol.thinktel.ca/rss pour s’assurer que vous recevrez la mise à jour ainsi que des avis futurs.

S’il vous plaît adresser vos questions ou préoccupations au soutien technique de ThinkTel en soumettant un billet via http://soutien.thinktel.ca ou en appelant 1-877-564-7660.

Merci,

ThinkTel Communications

Update: ThinkTel Hosted Lync and Hosted Exchange Service Interruption / ThinkTel Interruption des Serivce de Lync Hébergé et Exchange Hébergé

Tue, 06 Mar 2012 08:21:03 MST

Dear ThinkTel Customers,

The issue experienced with Hosted Lync and Hosted Exchange earlier has been resolved. Resolution occurred at approximately 9:45am EST.

We apologize for any inconvenience and thank you for your understanding.

If you have any questions, please do not hesitate to contact us by calling our support department at 1-866-928-4465 or by submitting a ticket for follow up at http://support.thinktel.ca/index.php/contact.

Sincerely,

ThinkTel Communications

=====================================================================

Cher Clients ThinkTel,

Le problem que nous avons rencontrés avec les services de Lync Hébergé et d’Exchange Hébergé devrait maintenant être résolu, depuis environs 9h45 HNE.

Nous sommes désolés pour tout inconvénient et nous vous remercions de votre compréhension.

Si vous avez des questions, n’hésitez pas a nous contacter en appelant notre département de soutien au 1-866-928-4465 ou en soumettant un billet pour suivi a http://soutien.thinktel.ca/index.php/contact.

Sincèrement,

ThinkTel Communcations

Interruption des Serivce de Lync Hébergé et Exchange Hébergé

Tue, 06 Mar 2012 07:38:04 MST

Cher clients ThinkTel,

Nous travaillons en ce moment pour résoudre un problème qui affecte nos clients utilisants notre service de Lync Hébergé ainsi que notre service d’Exchange Hébergé.

Les symptômes sont : impossibilité d’établir une connexion au service, affectant la capacité de placer et recevoir des appels avec Lync.

Nous avons isolé la source du problème et travaillons pour rétablir le service aussitôt que possible. Le problème devrait être résolu dans 30 à 45 minutes.

Nous sommes désolés pour tout inconvénient.

ThinkTel Communications

Hosted Lync and Hosted Exchange Service Interruption

Tue, 06 Mar 2012 07:37:21 MST

Dear ThinkTel Customers,

We are currently working on resolving an issue affecting Hosted Lync customers including Hosted Exchange.

Symptoms consists of the inability to make and receive calls.

We have isolated the source of the problem and working to get it resolved ASAP. Expected ETA is within 30-45 minutes.

We apologize for any inconvenience.

ThinkTel Communications

ThinkTel Service Interruption - Service Restored

Fri, 24 Feb 2012 08:13:24 MST

Dear Customer,

The network outage has been resolved as of 7:28AM MST and all services restored. We sincerely apologies for the inconvenience.

If you would like a detailed report of the incident, please submit a request by visiting http://support.thinktel.ca/index.php/contact and filling out the Request RFO form.

Thank tyou for your patience as we worked to resolve the issue.

Sincerely,

ThinkTel Communications

ThinkTel Service Interruption – Inbound and Outbound Calling Affected

Fri, 24 Feb 2012 07:20:14 MST

Dear Customer,

Please be advised that we are currently experiencing a network outage resulting in call completion failure on (Inbound &/or Outbound).

Our technicians are actively working to identify and resolve the issue. No ETA is available at this time.

Please visit our support site (http://support.thinktel.ca/index.php/announcements) for updates on this problem. Updates will be posted every 15 – 30 minutes until the problem is resolved.

We apologize for any inconvenience and thank you for your understanding.

If you have any questions, please do not hesitate to contact us by calling 1-866-928-4465 or by submitting a ticket for follow up at http://support.thinktel.ca/index.php/contact.

Sincerely,

ThinkTel Communications

Update: Hosted PBX No Audio Issue

Thu, 16 Feb 2012 08:58:03 MST

The issue was resolved at approximately 10h28 ET and two way audio has been confirmed.

Please note that the incident did not have any affect on Hosted Lync customer.

We sincerely apologies for the inconvenience.

If you would like a detailed report of the incident, please submit a request by visiting http://support.thinktel.ca/index.php/contact and filling out the Request RFO form.

Thank you for your patience as we worked to resolve the issue.

Sincerely,

ThinkTel Communications

Hosted PBX No Audio Issue

Thu, 16 Feb 2012 08:03:29 MST

Dear ThinkTel Customers,

Please be advised that we are currently investigating the issue of no audio on our hosted pbx platform. We apologies for the inconvenience.

No eta for resolution is available at this time. We're expecting that the matter will be resolve shortly.

A status update will be provided within 15-30 minutes

Sincerely,

ThinkTel Communications

Calgary/Edmonton TPIA Network Activity Maintenance Advisory

Mon, 13 Feb 2012 18:36:43 MST

Dear ThinkTel Customers,

Please be advised that we'll be conducting maintenance in our Edmonton Data Center tomorrow morning (Feb. 14th) between the hours of 8h00 - 08h30 MT (10h00 - 10h30 ET.

The maintenance being performed involves moving our Edmonton and Calgary TPIA (third party internet access) fibre links from our Cisco router to Juniper MX960 router and is not expected to have any impact on voice services.

Some TPIA customers will experience downtime of no more that 5 minutes during the maintenance. Based on the time performed, the effect will be minimal as traffic is very low at that time of the morning.

We apologize for the short notice and for any interruption in service that might affect any of our TPIA customers.

Please do not hesitate to contact our support line if you have any questions.

Sincerely,

ThinkTel Communications

Western Canada Service Update

Thu, 09 Feb 2012 12:20:08 MST

Though the major part of the earlier service disruption was resolved at 10:05 MST, we continued to receive a problem where inbound calls to the following rate centers received fast busy.

Winnipeg, MB
Sherwood Park, AB
Calgary, AB

The issue is now resolved and was the result of a problem on the Edmonton MetaSwitch gateway triggered by the earlier OC-3 failure.

We again sincerely apologize for this latest occurrence.

Please contact us if you continue to experience any further issues.

As per the previous message, an Incident Report will be available within the next few business days. Please visit http://support.thinktel.ca/index.php/contact and submit a request if you would like to receive one.

We thank you for your continued patience.

Sincerely,

ThinkTel Communications Ltd.

Service restored

Thu, 09 Feb 2012 10:59:36 MST

Service has been fully restored as of 10:05 AM MST. The issue was due to faulty cables to the OC-3 in our Edmonton Data Center.

A post-mortem incident report will be made available in the next few business days.

Please submit a RFO/Incident Report request by visiting http://support.thinktel.ca/index.php/contact if you would like to receive a copy of the report.

Please contact our support department if you continue to experience service-related issues. We thank you for your patience as we worked to resolve the problem. We apologize for any inconvenience that may have resulted due to this incident.

Sincerely,

ThinkTel Communications Ltd.

Service Update

Thu, 09 Feb 2012 09:46:43 MST

We've isolated the source of the problem and working to resolve it ASAP.

ETA to fully restore service is 30 minutes - 1 hour.

Winnipeg, MB is not affected as was previously thought.

Another update will be provided shortly.

Update

Thu, 09 Feb 2012 08:59:32 MST

[10:48 AM EST] A NOC technician is on-site working to isolate the source of the issue.

Service interruption in the following areas:

Thu, 09 Feb 2012 08:07:30 MST

[8:52 AM EST] We are currently experiencing an outage in the following areas:

Grand Prairie, AB

Medicine Hat, AB

Abbotsford, BC

Fort Mc Murray, AB

St Albert, AB

Fort Sask, AB

Sherwood Park, AB

Winnipeg, MB

Our NOC is working to restore service as quickly as possible and will provide periodic updates.

Services are fully restored

Tue, 07 Feb 2012 12:18:11 MST

All services are now operating normally.

The Toronto Metaswitch gateway was restored and passing traffic at 12:25 PM EST. This was the major catalyst that caused the degraded service and caused our Shaw transit to shut down. The subsequent network issues with the Shaw transit did not have a direct impact due to BGP routing redundancy through Bell's network.

Separate investigations are in progress and an incident report will be available in the next few days. Please submit a RFO/Incident report request by visiting http://support.thinktel.ca/index.php/contact if you would like to receive a copy of the report.

Actual times for degraded services affecting call traffic :

Start time: 10:20 AM EST

End time: 12:25 PM EST

Please contact our support department if you continue to experience service-related issues. We thank you for your patience as we worked to resolve these issues and apologize for any inconvenience that may have been caused.

Sincerely,

ThinkTel Communications Ltd.

Status update

Tue, 07 Feb 2012 11:07:18 MST

We have been advised of two issues that caused the service interruption this morning:

Metaswitch: the errors that were seen in the switch logs prompted our engineering department to contact the switch manufacturer emergency support team. The vendor was able to resolve the issue by performing a soft reset. No calls would have been affected during the reset.

Edm-Tor Transit: We are continuing to work with the service provider for our transit link. There is no eta to resolution for this issue.

All traffic is being routed through the redundant network links and calls should be completing normally. We're still investigating and working to resolve a problem with intermittent audio quality and one way audio that's being experienced on a very small percentage of calls.

Please continue to check for updates and report any issues that you're still experiencing to our support department.

Sincerely,

ThinkTel Support

Update

Tue, 07 Feb 2012 10:23:29 MST

Please be advised that we are still experiencing intermittent issues.

We have determined that the source of the issue is with our Metaswitch. Our engineering department has contacted the vendor and are working with them to investigate further.

Another update to follow.

-ThinkTel Support

ThinkTel Support Center

Tue, 07 Feb 2012 10:03:37 MST

As a result of the network issues this morning our main support line is also affected and ~30% of the inbound calls are not completing. There are also intermittent reports of one-way audio.

If you are attempting to contact us by telephone, please retry if you receive a "not in service" message as the majority of calls are completing. Many callers are finding that the 2nd or 3rd attempt gets through to our support.

Another update to follow in 15 minutes.

- ThinkTel Support

Support Update

Tue, 07 Feb 2012 09:47:05 MST

We are still experiencing intermittent call completion and audio quality issues. Our transit link provider has been contacted and NOC and Engineering are working to resolve issues that are discovered within our network.

Another update will be provided in 15 minutes.

Sincerely,

ThinkTel Technical Support

Continued intermittent service interruption

Tue, 07 Feb 2012 09:24:45 MST

Dear customers,

Our NOC and engineering departments are actively working to isolate the cause of intermittent call completion issues. This started at approximately 10:20AM EST. At this time we are investigating our internal network and the transit link between our Edmonton and Toronto data centers.

Sincerely,

ThinkTel Technical Support

Service Interruption - Tuesday, February 7th. 2012

Tue, 07 Feb 2012 08:45:54 MST

Dear Customers,

We're investigating a service disruption that at this point appeared to have lasted approximately 2 minutes which occurred between 10:20am and 10:30 ET. Since being restored some calls are still failing to complete. Our NOC and Engineering teams are investigating.

Additional information will be provided once we have more details. ETA on update is within 15 minutes.

We apologies for any inconvenience this disruption has caused. Please call our support line or select the contact us tab to log a ticket.

Thank you for your patience,

ThinkTel Communication

Action Plan for Network & Support Enhancements & Q & A with ThinkTel CEO Dave Damer

Fri, 03 Feb 2012 08:48:48 MST

Action Plan for Network & Support Enhancements

Q & A with ThinkTel CEO Dave Damer

You are invited to participate in an open question and answer period with ThinkTel CEO Dave Damer regarding the Action Plan we have tabled to address recent service-impacting issues.

When: Friday, February 3rd

3 p.m. ET / 1 p.m. MT

How: In order to participate fully in the event, we ask that you click on the link below to join the online component of the meeting as well as join by phone using the access information provided.

Join the online meeting

https://meet.thinktel.ca/dave/QFJMKMK6

Join by phone

+1-877-509-3914 Toll Free

+1-778-724-1400 Vancouver; English, French

+1-780-809-8075 Edmonton; English, French

+1-613-317-1182 Ottawa; English, Francais

+1-613-317-3601 Ottawa; Francais, English

+1-647-258-7462 Toronto; English, Francais

Find a local number (https://dialin.Distributel.ca)

Conference ID: 251238

First online meeting? (http://r.office.microsoft.com/r/rlidOC10?clid=1033&p1=4&p2=1041&pc=oc&ver=4&subver=0&bld=7185&bldver=0)

A message from our CEO regarding recent network-impacting issues

Fri, 27 Jan 2012 15:07:47 MST

To our customers:

In light of the frequency of network impacting issues in the last 6 months I have called an emergency meeting with our company owner as well as the CTO. The purpose of these meetings is to determine the specific actions, both immediate and ongoing, necessary to permanently resolve the issues with network stability and customer communications.

We have identified a number of items that we are currently in the process of changing or fixing; other items have already been fully addressed. We understand that your confidence can only be restored by our delivering consistent and uninterrupted service going forward and hope that you will give us the opportunity to deliver.

A detailed report is being prepared that will outline the steps that we’re taking and the dates targeted for completion of those steps. This report will be completed on February 3rd. Following the report’s release at 10 a.m. MT, I will be hosting a live conference call at 1 p.m. MT to address any additional questions and concerns. After February 3rd, weekly updates on the completion of the steps we’re taking will also be posted to ensure we’re staying on track and being held accountable. Between now and then, our entire team is available to address any immediate questions or concerns.

My apologies to any of our affected customers, resolving this situation is our highest priority.

Sincere regards,

Dave Damer
President & CEO | Président et chef des opérations
ThinkTel Communications
t. 780.809.8002
sip:dave@thinktel.ca

Service Restored

Wed, 25 Jan 2012 18:10:29 MST

Please be advised that all services were restored at 3:45PM EST. There continued to be audio quality issues that were resolved at 4:20PM EST.

If you would like to receive an incident report please visit our support website contact page and select Request RFO (Incident Report).

We apologize for this interruption of service.

System Status Update

Wed, 25 Jan 2012 13:05:34 MST

12:05 PM PST - Jan. 25, 2012

The network impacting issue that we just experienced has been stabilized. During our investigation is was determined that the cause did not originate on the ThinkTel network.

We are waiting on confirmation from our NOC and engineering departments that all services are up and operating normally.

We apologize for the inconvenience and will keep you posted on the situation.

Service Advisory

Wed, 25 Jan 2012 11:50:26 MST

10:30AM PST - January 25, 2012

We are currently experiencing severe packet loss with one of our IP transit providers. This is adversely affecting audio quality and call completion. Our engineering and NOC departments are investigating this issue and we will provide periodic updates. At this time there is no estimated time to resolution.

Emergency Maintenance Update - Switch replacement: Thursday, January 19th, 2012

Thu, 19 Jan 2012 05:36:52 MST

Good-day ThinkTel Customers,

We are pleased to advise that the Switch replacement has been completed successfully. Any disruption of service experienced during the transition would have lasted no more than 5 seconds.

Please contact us at 877-564-7660 if you have any questions.

Thank You,

ThinkTel Communications

Emergency Maintenance - Switch replacement: Thursday, January 19th, 2012

Wed, 18 Jan 2012 17:18:17 MST

Dear Customers,

Please be advised that we will be conduct maintenance tonight to replace one of our switches involved in last week Thursday's service interruption. During the maintenance, voicemail, music on hold and mail services will be affected. Anticipated downtime is between 5 to 30 seconds before returning to normal.

Date: Thursday, January 19th, 2012

Time: 02h00 EST through 03h00 EST

Please contact us at 877-564-7660 if you have any questions or experience an issue following the maintenance window that you believe might be related.

Thank you,

ThinkTel Communications.

Maintenance Update

Wed, 18 Jan 2012 08:47:39 MST

Dear ThinkTel Customers,

Maintenance for the new 1 Gig-E link between our two new core router connecting between Toronto and Edmonton was completed successful this morning. The Toronto router that was the source of intermittent service disruptions between Friday night and Monday afternoon has been removed and there's no traffic currently going through it.

We were unable to complete changing out of the Switch involved in the service disruption which occurred on Thursday, January 12th during last night's maintenance and will be scheduling another maintenance window to have this completed. Another update will follow to advise when we will be completing this activity.

Please contact us at 877-564-7660 if you have any questions.

Thank you,

ThinkTel Communications.

Emergency Maintenence Update

Tue, 17 Jan 2012 06:40:27 MST

Dear ThinkTel Customers,

Unfortunately we were unable to complete the maintenance work on the 1 Gig-E link between Toronto and Edmonton and the replacement of the switch in Edmonton scheduled for earlier this morning due to unforeseen reasons, but have made progress towards completing both tasks.

Another maintenance window has been set for tomorrow morning (Wednesday, January 18th. to begin at 00h01 EST.

No services are expected to be affected during this time. We'll continue to keep you updated.

Thank you,

ThinkTel Communications

Emergency Maintenance: Scheduled to being Tuesday morning at 00h01 EST

Mon, 16 Jan 2012 14:36:27 MST

Dear ThinkTel Customers,

We have several emergency maintenance activities scheduled for tonight in order to deal with last Thursday's outage, and another problem with a router in Toronto that this weekend intermittently crashing and again this afternoon at approximately 4:15pm EST.

1. We are bringing in to service our 1 Gig-E link between Toronto and Edmonton. This will remove the problem router in Toronto and its existing routing setup between our Toronto and Edmonton network. The new configuration will also address issues of redundancy and service reliability.

The activity will start tonight at 00h01 EST and should be completed by 02h00 EST. There will be switching hits / however, no packet loss or outage time is expected.

2. We will be replacing a Switch with in Edmonton switch from 02h00 EST (00h01 MST) and 05h00 EST (03h00 MST) to address and resolve the cause of the outage experienced last Thursday.

No voice services are expected to be affected, however, there will be some brief outages on the mail cluster affecting some customers.

We are taking all necessary steps to avoid customer affecting outages, and minimizing degraded services during these two key activities. We have thought out and planned the steps to minimize any fallout. We will monitor the network after the activities have been completed.

If you have any questions or experience an issue following the maintenance that you believe is related to the activity, please contact us immediately at 877-564-7660.

Thank you,

ThinkTel Communications Ltd.

Service Interruption - Thursday, January 12th. 2012

Thu, 12 Jan 2012 21:04:08 MST

Dear ThinkTel Customers,

We experienced a service impacting network issue this afternoon/evening from approximately 6:00pm to 8:50pm EST which affected the ability to make and receive calls to and from our network. Call quality during this time might have been affected as well during this time for calls that we're completing

Our Network technicians first noticed the problem at the time the problem occurred and started investigating immediately.

We are still completing our investigation and working on a detailed RFO that will be available in the upcoming days. Please select visit the contact us page of our support site (support.thintkel.ca) and select "Request RFO" and proceed to fill in the request information if you wish to receive an Incident Report.

Please accept our sincere apologies for the disruption this has caused to your business

If you have any questions or concerns, please contact us at 877-564-7660

Best Regards,

ThinkTel Communications

Scheduled Maintenance - Dec. 29, 2011 3:00AM EST to 5:00AM EST

Wed, 28 Dec 2011 11:16:19 MST

Please be advised that we will be performing scheduled switch maintenance on Thursday, December 29, 2011 between 3:00AM EST and 5:00AM EST. All efforts are being made to minimize the impact of this maintenance and no downtime is expected during this event.

Should you experience a service-impacting issue please call our support line or open a support ticket.

uControl Emergency Maintenance

Tue, 13 Dec 2011 14:58:19 MST

On December 13, 2011 uControl required emergency maintenance and was inaccessible from 10:45AM MST to 11:45AM MST. While this maintenance was being performed the online portal was unavailable to our customers. This was NOT a service-impacting issue.

We apologize for any inconvenience.

Scheduled Maintenance

Fri, 09 Dec 2011 17:50:07 MST

Date: Saturday, December 10, 2011

Start: 00h01 EST

End: 02h00 EST

Summary: Network operations will be restarting our Toronto proxy server to resolve a media issue. During the 2 hour maintenance window there will be approximately 1 minute where calls may not complete.

System Outage Update

Tue, 06 Dec 2011 07:26:02 MST

The issue has been resolved and identified as being related to one of our proxies (eico.thinktel.ca).

The outage lasted from approximately 7:05am EST to 9:03am EST on December 6, 2011

System Outage

Tue, 06 Dec 2011 06:49:23 MST

December 6, 2011 7:05am EST - Please note that we are currently experience system outage affecting the ability make and receive calls. The problem is suspected to be with one of our proxies. We're continuing to investigate and will have an update for you as soon as possible. An update will be provided within the next 15 minutes.

Service restored to the primary WAN circuit

Mon, 05 Dec 2011 21:10:56 MST

Service was restored to the primary circuit at 12-05-2011 21:55:23 EST and traffic is routing normally.

If you experience any further audio related issues, please contact our support.

Edmonton/Toronto WAN circuit service degradation

Mon, 05 Dec 2011 18:47:11 MST

At 19:40:37 EST on December 5, 2011 we experienced an issue with our primary WAN circuit. The backup circuit provided fail-over redundancy, however, due to the volume of traffic there may be service-affecting issues such as voice quality degradation.

We apologize for any inconvenience. This issue is being investigated and another update will be posted once resolved.

Service-affecting issue with our switch

Fri, 02 Dec 2011 02:11:22 MST

We experienced a service-impacting issue this morning from approximately 2:30 to 3:15 AM EST. The problem is now resolved however we are still in the process of our root cause analysis to determine why it occurred. More details will follow as soon as more information becomes available. An incident report will be provided as soon as possible.

Service-affecting issue with our switch

Fri, 02 Dec 2011 02:06:49 MST

Nov. 24, 2011 3:00AM EST - MAINTENANCE NOTICE

Wed, 23 Nov 2011 18:45:16 MST

On November 24, 2011 at 3AM EST our NOC will be performing an upgrade to our core switch. Due to the redundancy of the switch there is no downtime expected.

Nov 23 @ 2:22 PM EST - SERVICE NOTIFICATION UPDATE

Wed, 23 Nov 2011 12:23:40 MST

As we continue to investigate the G729 codec issue we have confirmed that this is only affecting inbound calls. Outbound calls have remained unaffected. A suggested workaround until this issue is resolves is to configure your equipment to fallback to G711 as a temporary solution.

Nov 23 @ 11:30AM EST - SERVICE NOTIFICATION

Wed, 23 Nov 2011 11:19:28 MST

At around 11:30AM EST this morning we started to receive reports of calls failing when using the G729 codec. The issue appears to be with our switch not offering that codec in the SDP and not accepting it when offered by our customer's equipment. This is currently being investigated by our NOC and Engineering teams.

November 8, 2011 - MAINTENANCE ALERT

Mon, 07 Nov 2011 17:00:00 MST

Shaw has advised that there will be emergency software maintenance performed just after midnight MST between the hours of 00:00 MST to 6:00 MST.
We have individuals from Level 2 Support, NOC and Engineering on standby during the maintenance window looking for any signs of impact to our network. Should you experience any downtime during or after the event, please contact our L2 Support line at 877-564-7660 immediately.

Nov 7 - 2:24PM EST - Update

Mon, 07 Nov 2011 14:24:00 MST

Our NOC and engineering department is continuing to investigate this issue. At this time the proxy servers are confirmed to have been unaffected and not the cause of the outage. There was a network issue that was caused by emergency maintenance performed by Shaw that caused an outage of our OC3 circuit. Traffic automatically failed over to a backup circuit. We are currently investigating the amount of time that was required for that fail-over to occur.

Nov 7 - 12:17PM EST - SERVICE ALERT

Mon, 07 Nov 2011 12:17:00 MST

We received multiple reports of an issue that occurred at 4:43AM EDT this morning with our vp.thinktel.ca proxy. Service was automatically restored in approximately 5 minutes.
Our NOC department has been notified and they are currently investigating the cause of the outage.
Another notice will be posted here once the root cause has been identified.

Nov 2 - 12:10PM EDT - vp issues are resolved

Wed, 02 Nov 2011 12:01:00 MDT

Issues with vp are resolved.

Nov 2 - 11:00PM EDT - Reports of one way audio with vp.thinktel.ca

Wed, 02 Nov 2011 11:00:00 MDT

Reports of one way audio for customers on proxy vp.thinktel.ca

Nov 1 - 7:30PM EDT - Engineering confirming all issues are resolved

Tue, 01 Nov 2011 19:30:00 MDT

Engineering reported that the proxies were stable and the issues shows to be resolved.

Nov 1 - 7:00PM EDT - All services are now operational and stable

Tue, 01 Nov 2011 19:00:00 MDT

Service appears to be stable. We're continuing to monitor the situation and will update with any development or to confirm that the problem is fully resolved. Please report any related problems that you're currently having to support.

Nov 1 - 6:00PM EDT - Reports of one way audio and dead air

Tue, 01 Nov 2011 18:00:00 MDT

We had re-occurrence of the problem at around 5:50pm EDT. The problem lasted for approximately 5 minutes. There have been some reports of one way audio and dead air. Please report to our support team if you're experiencing one-way audio issues/no audio issues. Another update will follow soon.

Nov 1 - 4:56PM EDT - All services are now operational

Tue, 01 Nov 2011 16:56:00 MDT

All of our proxies are now operational. The vp.thinktel.ca proxy is considered stable but there is a known issue with specific signalling that may cause minor interruptions for particular calls, however, this is not affecting all customers on that proxy. Another update will be released once that final issue has been resolved. All other proxies are now stabalized.

Nov 1 - 2:44PM EDT - Temporary outages with the cluster server hosting our proxies

Tue, 01 Nov 2011 14:44:00 MDT

As our engineers are working to resolve issues with our proxy cluster the proxies that were up and running are experiencing temporary outages. At this time we are aware of intermittent issues with our eico.thinktel.ca proxy.

Nov 1 - 2:08PM EDT - Continuing issues with vp

Tue, 01 Nov 2011 14:08:00 MDT

We are continuing to experience issues with our vp.thinktel.ca proxy server. We appreciate your patience as we work to resolve this priority issue.

Nov 1 - 1:02PM EDT - Remaining service to be restored: vp.thinktel.ca proxy

Tue, 01 Nov 2011 13:02:00 MDT

Our vp.thinktel.ca proxy is the last service to be restored. All other services are confirmed to be back online and stable. Another update will follow once this remaining issue with vp has been resolved.

Nov 1 - 12:31PM EDT - Most services restored at 10:55AM EDT

Tue, 01 Nov 2011 12:31:00 MDT

Most of our services were restored at 10:55AM EDT. At this time the OC3 circuit is stable and the router has been brought back online and is broadcasting correctly to the network.

Nov 1 - 10:45AM EDT - Continuing issues

Tue, 01 Nov 2011 10:45:00 MDT

Currently we're still working on resolving the issue with Engineering.

Unfortunately we have no eta at present but all hands are on deck to resolving the problem.

An update will be provided within 30 if still unresolved.

Nov 1 - 8:35AM EDT - SERVICE ALERT

Tue, 01 Nov 2011 08:35:00 MDT

Please be advised that we experiencing a network outage resulting the failure to complete calls (inbound and outbound). The problem started at approximately 8:25am EDT and was thought to be the result of a router failure. Investigations are being conducted.