Things to Think About Before Ordering VoIP...

'Researching VoIP' Tech Bulletin

Updated:  7/3/10
-------------------------------

WHAT'S THE WORST WAY YOU CAN GET SCREWED?

Absolute #1:  Your VoIP phone line provider goes belly up... this could happen to you if you use VoIP for incoming calls:

• In June of 2009, CentricVoice, a rather expensive business class VoIP phone company gave their customers about a week's notice before they turned off all of their customer's phones lines. In the message on their web site to their customers, they said "The regular porting process usually takes 20 - 30 days, therefore, you will need to state that this is an Emergency port in order to have it done sooner."
  
  What would you do if your phone company told you they were going out of business in a week?
   

• In January of 2009, Verizon Voicewing, the VoIP provider division of Verizon, told their customers they have 90 days to find new service - they're just shutting it down on March 31st, 2009. Hey, you're just a stinkin' customer. Their attitude is simply "Screw you and the camel you rode in on."  Is it possible that a real phone company (utility), formerly part of the Bell System, would turn off phone service provided by a non-regulated subsidiary?  Yes.
   
• In July of 2008, AT&T Callvantage, the VoIP provider division of AT&T, stopped taking new customers. In April of 2009, they announced they're discontinuing the service some time in 2009. Then they gave notice that it will be turned off on 11/17/09.
  
  This is the most notice any VoIP phone company that's gone out of business has provided its customers to-date. Pretty classy compared to how other phone companies have closed their doors, but still amazing when you consider it's a part of a real phone company turning off their service. They don't seem to understand that by screwing their subscribers now, they're screwing themselves and the whole telecommunications business over in the future. They know it's not like closing a restaurant or gas station, but they don't care.  Is it possible that a real phone company (utility), formerly part of the Bell System, would turn off phone service provided by a non-regulated subsidiary?  Yes.
  
  In an interesting note, in December of 2009 AT&T announced that they want to shut down POTS analog phone lines because they aren't profitable - everybody is going to VoIP (so if everybody is going to VoIP, why did they shut down the AT&T VoIP Callvantage service?).
  
  AT&T says: "Congress’s goal of universal access to broadband will not be met in a timely or efficient manner if providers are forced to continue to invest in and to maintain two networks."
  
  This would be fine except that VoIP isn't ready for prime time, and POTS is in fact the only communications service that's likely to be left working after many disasters. VoIP and the Internet doesn't work without power, and cell service goes down pretty quickly from overload and lack of power when the backup batteries run down before power is restored.
  
  Why does the government allow regulated utilities to own non-regulated companies? I've been asking that since 1992.
   
• In July of 2007, Sunrocket, a VoIP provider with around 200,000 subscribers, permanently closed their doors and turned off their system. This left everybody with a Sunrocket phone number, or a number that was ported to Sunrocket from a real phone company, with NO PHONE SERVICE. (fast busy when you called it, and you couldn't dial out).
   
• In July of 2004, Norvergence, a VoIP provider that preyed on around 10,000 businesses (no home phone service), permanently closed their doors and turned off their system. This left everybody with a Norvergence phone number (which also included Internet access and cell phones), or a number that was ported to Norvergence from a real phone company, with NO PHONE SERVICE.
   
• In July of 2007, Google bought a VoIP phone service called Grand Central, who advertised that you get a "phone number for life" (they don't advertise that any more). Google almost immediately stopped signing up new customers. Some of the customers who had a "phone number for life" must have been murdered, since it turns out that Grand Central wasn't really able to deliver on the claim that the phone number would work for a lifetime (I don't know if the customer, or the phone number went dead first).
  
  In March of 2009, Google announced they are starting a new free VoIP phone service, called Google Voice. You can only get one line and you have to use a headset on your PC sound card to make calls, so it's not an option for business phone service. They do have some interesting features that a "One Number Service" charges money for, like ringing multiple phones at one time to find you when someone calls the number Google gave you. Google says it's not a real phone company, so they may not complete calls to certain phone numbers. You have to have other phone numbers from a phone company to be able to use this service.
  
  In November of 2009, Google announced that the "phone number for life" Grand Central service they bought is shutting down at the end of 2009. They say you better get any voice mail messages off it before they close it down.
  
  In November 2009, Google also announced they are buying a crummy VoIP phone service called Gizmo5. When I tested Gizmo5 a few years ago, it wasn't very good. I emailed them to stop billing me, but they refused. Then they started sending me hate email that my credit card expired, and I owe them big bucks for their service that I don't want, and I only made maybe three calls on to test. Anyway, now Google is a phone company.
   
• In March of 2009, Skype announced a new beta trial SIP telephone service for businesses (their regular service primarily works from computer to computer). This is just a test, so if you try the beta - your service could stop if they decide Skype for business isn't worth messing with.
  
  Skype for SIP (for businesses) allows "computer to phone" calls to businesses, for free, from any computer in the world with a broadband connection, to a business equipped to handle SIP VoIP trunks on their phone system. Skype says that if the trial works out, they may offer the service later. A trial is obviously not a guarantee of service (nothing is guaranteed by any VoIP phone company, since there's no government regulation of them), so you shouldn't count on it continuing after the trial.
  
  There are expensive boxes out there that convert Skype to POTS lines, so they can be used on regular trunks on a phone system. If Skype actually deploys Skype for SIP, it will probably hurt the companies making those expensive boxes (which won't be needed any more).
  
  Ebay (who owned Skype) recently sold 70% of Skype to private investors, so the service might get better. Or not.
  
  In December of 2009, Skype opened their Skype for Business beta to all businesses. All you need is a SIP capable phone system or a SIP ATA, and some money.
  
  Our standard method for testing a SIP service is to program a Linksys SPA3102 for that service (I use the SPA3102 like a VoIP butt-set), and connect it in exactly the same place into our router with the same patch cord as our control VoIP vendor that's been working fine (we used Junction Networks as our control for testing).
  
  The control panel or whatever they call it for setting up Skype for Business was very confusing, but it is a beta service, so that's OK. I had to use Paypal to add about $60 to various accounts to start it up - just to get one ATA working.
  
  It sounded terrible. Literally at the bottom of the barrel of SIP providers. I tried it at a second location with another SPA3102 ATA as a sanity check. Still terrible quality. Probably OK for talking to a relative across the ocean, but certainly not useful for business in the US.
  
  It may be OK if you have a lot of people in other countries calling you on your phone system from their Skype account (for free). We get a lot of calls from other countries, so I thought it would be neat for people to be able to call us free using Skype. Users of Skype are used to inconsistent quality for free phone calls, but just about anybody would take issue at paying for inconsistent quality.
  
  I cancelled the account and asked Skype for a refund. Out of the $60, all they said they would refund is $10. I was out $50, and a lot of labor setting-up and trying their service. I sent a complaint to their president's office, and finally got a refund of the $50 about a month later (the person responding to my email from the "president's office" definitely sounded like someone in India who didn't have a clue). There are new owners at Skype now, so maybe things will change for the better in the future?
   
• The most terrifying VoIP story ever?  On April 2, 2009 the FBI had a search warrant for a data center, where they suspected the owners of fraud. A data center is where lots of companies either rent servers, or co-locate their own servers. A data center is like an apartment building for computer servers. They have multiple fast connections to the Internet, probably from different companies so there is no one source of failure. They have heavy duty air-conditioning, battery backups for all the servers, a generator and high security. Unfortunately, the security wasn't enough for about 50 customers of the data center who the FBI put out of business, at least temporarily.
  
  Some companies have servers running in two data centers, just in case something really bad happens at one of them. There's some chance that it would be the same owner at both data centers, which happened in this case.
  
  The FBI backed up trucks and took everything including backup tapes that could have helped the data center customers get a server setup at another data center.
  
  It's pretty obvious after the fact that the FBI and judge who signed the search warrant had absolutely no knowledge of what a data center is. The FBI guy or judge probably had no clue that they would be shutting off many innocent company's web sites, their mother's church's web site, credit card processing, on-line ordering and tracking, and VoIP phone calls (they just go through a regular computer server).
  
  Considering the stink the government makes about 911 calls from VoIP lines, it's interesting that all the telephone customers of the VoIP service(s) in those data centers couldn't make any phone calls, including to 911, after the FBI (government) threw all the equipment, maybe 200 servers where each server could be shared by multiple companies, into trucks.
  
  I've done a lot of big telecom and IT projects. Nothing bothers me because I just do everything in a methodical manner, all planned out. I've never had a cutover go badly. The thought of trying to sort out the pile of servers and get all those companies back up in a timely manner is totally overwhelming to me, no matter how many technicians I had available.
  
  It's unlikely that if the FBI was investigating the owners of a shopping mall, they would get a search warrant for the mall management office, as well as every store in the mall... and then empty every store in the shopping center into trucks in case the items in each store were somehow related to the case against the shopping mall owner.
  
  This could easily happen to any VoIP provider. The only place it probably couldn't happen is at a real phone company (utility). Every other CLEC (a fake phone company that competes with the real Phone Company), VoIP provider or otherwise fake phone company can go away just like this (just like any other business or even a church in the US).
  You can read a Wired article about it HERE.

If the above examples were the only phone service a business had,
they'd be in big trouble!

They would have to get new phone service, and hope they can port the number they had with their old VoIP provider to the new service - which would probably take a few days or more. If the VoIP company went belly up overnight with no notice, and their business depended on incoming calls, they'll lose quite a bit of business (outgoing isn't a big deal since we can all use our cell phones).

A little farther down in this Tech Bulletin, I mention more details of the Norvergence disaster, and the best VoIP provider I've found so far, Junction Networks.

Why can VoIP service just go away like that?  Because it costs very little to get into the VoIP provider business. You can start your own VoIP service for very little money:

• Just get a few servers for the calls and the billing (buy them used on ebay, or rent them from an ISP/Data Center).
   
• Connect the servers to the Internet (just rent some rack space very cheaply at an ISP/Data Center - a third world country would be fine).
   
• Get some backend software to run the business.
   
• Contract with one of many companies selling VoIP call terminations by the minute (their business is to connect local phone lines to the Internet at various points around the country and the world).
   
• Find a computer geek to help you setup and maintain the servers.
   
• Rent local phone numbers from one of the national companies that "rents" phone numbers to VoIP phone companies, very cheaply.
   
• Rent a 911 call center service from a third world country, where they will manually transfer one of your customers who dials 911 to the local police department (maybe?).
   
• Hire a third world Indian company (in the country of India) to answer the phone, to make your customers think they are getting support (the Indian support person simply emails your computer geek about a problem... At best "Robert" in Inda can try to read English from a script you provide them).
   
• Alternatively, don't offer any phone support. Just put a form on your web page to contact support (which you never even look at). As long as you call yourself a "Phone Company," your customers won't have any question in their mind that they'll get the same support they get from the real phone company in their area (which is often just as bad as when it's coming from a third world country).
   
• Start rolling around on the floor in all the money you collect. You're a "Phone Company!"
   
• Stop paying the Data Center bills, the VoIP call termination bills, the rented local phone number bills, the computer geek, the third world Indian phone bank company, and pocket all the money you save until the business folds, leaving your customer's businesses without telephone service.
   
• There are no regulators, no regulations, and only civil courts to deal with. The customers aren't going to sue you when their service stops. Oh, and by the way, if the ISP or the VoIP termination company goes belly up, your service is down whether you like it or not - to say nothing of what happens if the computer geek gets sick, dies, or even goes to jail.

I found a mention of a small one-man band VoIP company on the Internet, and got a line from them to test hoping that they would be better than the big boys. Afraid not.

I called Jim Jackson at Binary Telecom. He answered the phone right away, spoke English, was knowledgeable, and answered all my questions. When I started testing the line a couple of months later, it couldn't be called. I could make outgoing calls, but incoming calls went to another company. That company wasn't one of his subscribers, but a customer of the wholesaler he was buying the lines from.

He fixed that within a day or two. The new number he gave me didn't work because he changed the initial password. He changed it back. It worked. I set it up in our office for people to make outgoing calls from for a while.

His control panel was very weak compared to most. There was no way to forward all the calls from the control panel (or change the displayed Caller ID number to our main number), so I asked him to forward all incoming calls to our main number (he obviously has a more powerful control panel from his supplier). This is important when using a line just for outgoing in a business, since lots of people just call the number that displays on their Caller ID, not the phone number you leave in the message. The Binary Telecom VoIP line wasn't programmed to ring in on our system.

After a few days, nobody in the office wanted to make calls from that line. Calls took a long time to connect, if they connected at all. There was no ringback, so we didn't know the call was actually ringing. We kept calling people over and over, and it turns out their phone was ringing even though when they answered it there was nobody there. When I sent an email to complain (couldn't get through by phone), Tim (the President) said he was switching to another wholesale provider, and the problems should be fixed within a week:

"I am making arrangements to move all Binary customers to a new termination (outbound calls) system. I expect this change to be completed this week. The performance should improve dramatically."

At some point, we couldn't make any calls - they just never connected. No problem, we were just testing the line anyway. A week later, I tried to make a call and I didn't get dial tone. I reset the ATA. Still didn't work. I tried to go into the control panel, and it wouldn't let me in. The performance may have improved, but there was no way for me to know since it didn't work at all.

He obviously changed the password again, and I had no way of knowing what it was. When I called the office, there was no answer - just voice mail no matter when or how many times I called. So I gave up and asked him to cancel the line via email, and for a refund of the charge he had just put through. He said no refund, I had to give him a month's notice to cancel the line (even though it didn't work at all, and I couldn't log into the account). So I got ripped off for $30. Live and learn. There's a lot of learning to be done with VoIP.

Although he was quite forthcoming with the problems with the line until I cancelled, when I told him his service was terrible, he replied with this email:

"We have a switch that performs flawlessly for thousands of customers, terminating and originating MILLIONS of calls every day. You had service that worked for what should have been ample time to see if it worked on your Local Area Network."

If you've played with VoIP, it probably sounds familiar. Exactly what an Indian guy in tech support would read off a script when someone called to complain to most VoIP companies today. In other words... It's your network, not our service!

If it's your home phone line you're playing with, so what?  For a business, it's going to get expensive in lost business - especially if you're using the line for incoming calls (a bad idea with any VoIP service).

I'm not telling you this to scare you away from using VoIP

It's here to give you a sense of the current reality of the phone business. Make your decisions so you have a Plan B. Don't put all your eggs in one basket. Like I say numerous times in this Tech Bulletin, just because someone calls themselves a Phone Company, doesn't mean they will always be here like a local Phone Company (even if they have Verizon or AT&T in their name) - who are utilities that essentially can't go out of business (they just keep wasting money, and raising rates to make up for it!).

A VoIP Provider is NOT a real phone company. If you know that
going in, you should be able to successfully implement VoIP
at your company... Because you'll have realistic expectations.

You don't have to use VoIP phone lines with most VoIP phone systems. You can keep your regular analog phone lines for the real phone company, or use a real voice T1 (not a  Data T1) if your phone system has that option. A voice T1 is also known as a channelized T1 (or PRI - Primary Rate ISDN line), where the line is separated into 24 voice channels (a PRI is 23 voice channels, and 1 data channel).

Getting a Data T1, where you share the T1 with Internet access and VoIP phone lines, is pretty dangerous if all of your incoming phone lines are on that Data T1. There is the same probability that the company you bought that T1 from is going to go belly up as there is from a pure VoIP phone line company. If you keep some regular analog POTS lines for incoming calls, the shared Data T1 can be a pretty good deal, and it won't be the end of the world when the company providing the Data T1 goes belly up.

The bottom line is that it's perfectly safe and can really save money using VoIP for outbound calls, since if it goes down you can use your cell phone, your home phone, or even go to a bus station, airport or the local jail to use their pay phone (probably the only places left with pay phones). It's not safe to have all your incoming calls coming in via VoIP or on a Data T1, no matter what anybody tells you (especially the guy trying to sell it to you!).

-------------------------------

One more word of warning: You know you have to be very careful ordering anything on the Internet, but here's an example of what I saw when I tried to order a couple of  VoIP ATAs to replace the Sunrocket ATA we used for testing before they went belly up:

The $79.95 magically becomes $132.85 when you go to checkout:

It turns out that Technical Support and a 1 year guarantee is automatically added to the checkout (for over $50!), with no apparent way to remove it. As you can guess, I didn't order these from this company. In this busy world, it's easy for a company to use "bait and switch" like this. They almost got me.

It's dangerous buying anything on the Internet. Even with 'more or less honest' companies who will actually ship you what you ordered, the #1 way of ripping you off is tacking on $5 or $50 for warranty, support, extended download, or whatever. These crooked (or 'more or less honest'?) companies automatically mark the box for you to pay this extra $5 or $50. You have to uncheck the box manually to remove the bullshit charges before you buy the thing.  Keep that in mind as you travel the Internet!

By the way, when I called the company to mention their price discrepancies, they said "Their customers prefer it that way." I forgot to take my Stupid Pills that day, or I would have believed them.

Why was I ordering the Linksys SPA3102NA?  Because it's a great tool for testing VoIP, and it's cheap (cheaper than when I tried to order it above).  Think of it as a VoIP butt-set.  Program in the local network router settings and the SIP settings for a VoIP provider's line, and you're testing in a matter of minutes. Because the 3102 also has an FXO port, you can extend an analog station port from a phone system to another location (OPX).
 

So here's the VoIP Research Tech Bulletin...

-------------------------------

If you're thinking of getting VoIP to save money, do a little research before ordering it. It could save you several bottles of Tums, some hard cash, and some lost business.

If you're thinking of getting VoIP because your business has multiple locations, that's where VoIP really shines today - but you still have to do your homework. Like any business with multiple locations using a central PBX or Centrex, you have to be very careful in dealing with 911. If you screw up and 911 doesn't work correctly from one of the locations and someone dies, your job and your whole business are at stake.

Because our company sells all kinds of gizmos to fix strange telephone problems, we hear about an incredible number of problems implementing VoIP, T1 and even POTS lines from CLECs  (Competitive Local Exchange Carrier, or "fake phone companies") every week. We also hear lots of problems with POTS lines from real Phone Companies, but they're easier to solve. VoIP can save money and/or help your business run better, but you should dip your toe in first to see what it's like!

Look BEFORE You Dive Into VoIP!

People who buy phone equipment today assume that the stuff is dependable, mainly because phones and phone service has been pretty dependable in the past. Not now.

People who get phone lines from a company who says they're a Phone Company assume that the phone lines they're ordering will work as well as the ones they've gotten from their local Phone Company. Afraid not.

Just because someone is selling you a telephone device or telephone service doesn't mean it will actually work, especially when it's connected to a particular piece of equipment.

In the old days, all telephone equipment was essentially compatible. These days, there's some chance that it just won't work in your application - and you don't find out until after you've spent a lot of money on new equipment that won't work right, or you can't make or receive phone calls... and you're losing business.

At companies who decide to have their IT guys make decisions about and install the telephone equipment, there's an even bigger danger since their time is split between many projects, and they often don't have experience dealing with telephone equipment (so they don't know the pitfalls to watch for).

VoIP lines generally don't work with alarm equipment, modems, credit card authorization terminals, or satellite/cable set-top boxes (VoIP telephone audio is compressed so it won't take up much bandwidth, which makes everything other than plain voice a crap-shoot).

Faxes often don't work well on a VoIP line.

VoIP stuff and T1s generally don't work when the power goes out (a UPS will help until it runs down, and a generator helps until it runs out of fuel or breaks down/never starts up).

The technicians installing and servicing VoIP equipment and T1s for businesses are sometimes clueless, and may not even be able to communicate while on-site with the company who's actually providing the VoIP/T1 line (sometimes they're forced to call a third-world country for support themselves, or they have to email support with the problem - a bad thing if you have no working phone lines).

If a technician can't fix your phone system or phone line,
and needs to email someone to get support themselves, you
 know you screwed up buying that system or phone line!

If you buy a phone system or phone equipment, make sure you vendor stocks the parts to fix the system locally - so that they don't have to go order it and your phone lines are down until the equipment gets shipped in. Sometimes you need spares just to determine what's wrong, and if the vendor doesn't stock spares it may be very difficult to diagnose the problem.

When you order VoIP phone lines yourself, since you didn't purchase the VoIP phone service from your phone system vendor, and probably didn't even call them to see if they thought it would work on your system before you ordered it, they can't help you much when the old phone lines go down and your new service isn't working.

You should seriously consider talking to your phone system vendor before making any decision, which is a good way to learn from others' mistakes. They've probably seen it all by now.

If you order the stuff yourself, and you don't do your homework and contact your phone system vendor, you're basically a test pilot. Do you want to put your business at risk, testing stuff to see if it works?

Blaming your old phone equipment for your new VoIP phone lines not working right is stupid. You're likely to pay T&M for a lot of troubleshooting, and in some instances the VoIP stuff is junk, so it will never work no matter what your phone system vendor does.

When the Chinese or third world country designs the VoIP hardware, they don't do much testing... they're going for the "low hanging fruit" where it goes in and works at maybe 75% of the places it's installed right away. They don't care about the other 25%, since it's not profitable for them to care about it.

When you order a VoIP line from the salesman who's telling you how much money you can save, there's usually nobody to tell you the things to check for - you find out after you install it and it doesn't work. After that, tech support from some third world country is useless. All of these things can cause some pretty big problems if you get rid of your regular phone lines before doing your homework.

There's no reason you can't order the new lines and see if you're happy with them before you drop the old ones. Knowing that there's a reasonable chance that the new lines won't work as expected makes it easy to just get rid of them and try a new vendor. That's part of the process with VoIP that you didn't have to worry about with the real Phone Company.

Be realistic in your expectations, and a switch to VoIP will be a lot less stressful.

NEVER EVER EVER EVER GET A BUSINESS PHONE NUMBER
FROM A VoIP PROVIDER!

The phone numbers that most VoIP providers will give you are a special breed of number. They won't belong to you, and you can't keep them if you switch VoIP providers or go back to the Phone Company for real phone lines. The numbers don't even belong to the VoIP provider.

VoIP providers needed a way to get local phone numbers throughout the country quickly, so they could become a "national" phone company. Most actually "rent" these phone numbers from companies who are in the business of renting out blocks of phone numbers.

The local phone number "rental" business started up in the mid 90's with the popularity of the Internet. The ISPs (Internet Service Providers) that offered dial-up service needed a local phone number just about everywhere, since nobody wanted to pay big bucks to the phone company for a toll call to surf the Web for hours. Just at the time that broadband was killing the dial-up ISPs a few years ago (fewer local phone numbers were needed to dial into the Internet), VoIP companies came along needing phone numbers in virtually every city in America. They went ahead and rented blocks of these numbers everywhere, and became overnight "national" phone companies.

Imagine the surprise you'll get if you publish the VoIP phone number you get, and later decide you could get a better deal somewhere else, or try to go back to a real Phone Company because of quality issues. You'll never be able to use that number with another phone company, and if that VoIP provider goes out of business, you may not be able to get that number from any other VoIP company (they may deal with a different phone number rental company). If you need a new phone number and you really want to use VoIP service, get a line installed from the Phone Company, and then get it ported to the VoIP provider (and disconnect the Phone Company line). If you have multiple lines that hunt, you really only need to port over the main number that you publish.

If the VoIP provider promises you that the phone number will be yours to keep forever, they're not telling the truth. If the company they're renting the phone number from gets out of the business, or goes out of business, and the number can't be ported, you'll lose the number forever. It's impossible for anybody except the real phone company to promise you that you'll have the number forever, and even then you could lose the phone number in rare cases.

Once you port a number away from the Phone Company, you may no longer be listed in the White Pages or Information, and you may have a problem getting into the local Phone Company's Yellow Pages?

I personally would never port our incoming local numbers, since our company would be out of business without them. We don't use 800 numbers at our company because it's possible to have the number hijacked by an 800 service provider. While this doesn't happen often, it's possible that the 800 number you've used for many years could be taken away from you and given to another company. You don't own an 800 number, and many of the 800 service providers have been through bankruptcies.

While a real Phone Company (a Public Utility) won't disappear into the night, a VoIP provider or CLEC could close their doors leaving you high and dry. The most publicized case was Norvergence (besides Sunrocket, mentioned earlier), who offered customers local and long distance, 800 service, Internet, and cell phones on one low monthly bill - until everything stopped working one day because Norvergence didn't pay their vendors (look up the sad story on Google). That left over 10,000 businesses with no way to communicate, scrambling to get back into business. I should have bought stock in Tums that day!

Let me stress this again...

NEVER EVER EVER EVER GET A BUSINESS PHONE NUMBER
FROM A VoIP PROVIDER!

If your dial tone is coming from some kind of box (instead of a line from the Phone Company), and you're using something other than an old fashioned single line phone to make and receive calls, you may have problems that you didn't have when you were using POTS lines (Plain Old Telephone Service) from the Phone Company.

VoIP phone lines were originally used to make outgoing calls cheaply - mainly from home with a regular single line phone or using a headset attached to a computer. While the quality wasn't as good as the real Phone Company, the savings, particularly on International calls, were substantial enough to put up with the quality issues. The savings on outbound International calls was even more significant for business.

Because VoIP worked well for outgoing calls, companies started to use it for incoming calls - which was the start of the problems. Some of the VoIP phone companies started offering unique features on incoming calls like inexpensive 800 service, foreign exchange (phone numbers from multiple cities ringing in to a single VoIP device), and external call transfer. These features make it very attractive to just go ahead and switch to VoIP, but  just because a VoIP provider says their features work doesn't mean they'll work in your application. If you don't do your homework, I'd start buying those Tums at the warehouse club.

Companies start using VoIP lines for incoming calls, and find that it doesn't work with their particular phones or phone system (but it works OK with a standard single line telephone).

The reason that most companies consider switching to VoIP is simply to save money. You can get almost all of the features that VoIP service offers, but it will cost you a lot more from a real Phone Company. I hear the craziest stuff from companies who have switched to VoIP without doing their homework, and are in deep stuff - looking for a solution to get them out of the hole they dug for themselves.

Most VoIP phone systems come with licensing fees. In the past, primarily the larger legacy phone system manufacturers charged licensing fees on a per feature basis. It was usually one time fee when the system was purchased.

With legacy phone systems you bought a cabinet that was big enough to hold enough the station and CO line cards needed to run your company when you bought the system. If you needed more stations or trunks, you bought more cards, and then maybe an expansion cabinet or migrated to the next size up system cabinet. You bought as many proprietary phones as you needed, as you needed them.

VoIP phone systems generally don't have station or trunk ports. One box that fits in a 19" rack could run hundreds or thousands of phones and lines. Some will work with any cheap Chinese VoIP phone. If there were no licensing fees, a guy with 10 phones would spend the same thing as the guy with 200 phones for the phone system. The VoIP phone system manufacturer would never be able to make money, because every system would have to sell for the same thing that a 10 phone system would cost.

I don't know that there's a better way to handle VoIP phone system pricing than licensing fees, but if you're a legacy phone system owner who's never had to pay licensing fees, this might be tough to handle. If you have multiple offices or just want to allow workers to answer the phone from home, the benefits of a VoIP phone system can make the licensing fees seem cheap.

If you will never need the features that only come on a VoIP phone system (like off-premise workers), it's stupid to pay the licensing fees. There are plenty of perfectly good new and used legacy systems that don't require licensing fees. Some legacy phone system manufacturers have add-on VoIP features to their legacy phone systems, which may be the best of both worlds and save money on licensing fees.

There are now many companies offering a "hosted PBX."  You don't buy a PBX , just industry standard VoIP phones (that they normally sell to use with the PBX). It's a cheap way to get a phone system, especially when you have workers spread out throughout the local area, the country, or the world.

The inexpensive VoIP "phone system" acting as the "hosted PBX" can be connected to the Internet just about anywhere in the country, and it's shared by as many subscribers as they can get. Like having your own VoIP phone system in your office, the bandwidth to the Internet of the "hosted PBX," both up and down, is critical.

As with any VoIP solution, the calls won't all sound wonderful since the packets with the voice are traveling over the public Internet. The phones won't be as friendly to use or have as many features as a 25 year old legacy phone system, but this can be a reasonable solution if you make sure there's enough bandwidth for the calls on each Internet connection (and local network) that has a VoIP phone working off that "hosted PBX"... Which is the real secret of making VoIP calls sound good in general.

Keep in mind that that all companies advertising a "hosted PBX" aren't alike. Some will be operating out of a garage where they keep the pizza that was just delivered warm with the PC running Asterisk (the free VoIP phone system), and some will be trying to provide good service and support. There will be a difference in quality based on whether the company has the "hosted PBX" server in their office, or in a rack located at a real data center somewhere, with batteries, a generator, climate control and a big pipe to the Internet. You will never get the voice quality or dependability of having your own phone system, but in some cases you might never otherwise be able to afford to start and run an innovative company without using a "hosted PBX", at least in the beginning.

Like with any VoIP solution, try it before you jump in!

NOTE:

We've used AT&T's Callvantage (which AT&T closed down) for outgoing calls for a few years at our company. When we finally settled on Callvantage years ago, we had gone through almost all of the VoIP providers out there. The difference in quality was incredible.

I've called AT&T numerous times to ask them why the quality of their service is so much better than everybody else's, but I could never get an answer. I couldn't even talk to someone who would admit knowing anything about Callvantage, or who ran it within AT&T. I also tried to reach Centillium, the company that made the ATAs for Callvantage which worked unbelievably well, but they didn't respond to my requests for information.

I definitely have a question about why Callvantage's calls are better?  The Centillium ATA is locked down, so I can't look at the programming. Is it the ATA itself, the codec used, some other magic? We may never know why it's so much better than the rest. AT&T (formerly SW Bell) sure as heck doesn't want to tell anybody how to make VoIP sound better!

On the other hand, I suppose it's possible that the Linksys SPA3102s and PAP2s are total garbage (we have lots here that all sound the same), but I did try some of the VoIP providers on a couple of  Cisco 7960 phones (setup for SIP) as a comparison. They sounded like garbage on that phone, as well as on the ATAs.

Callvantage was setup by the "old" AT&T, before Southwest Bell bought them. Being a real phone company (which AT&T was not before SW Bell bought them), it's apparent that SW Bell decided that it wasn't  good idea to own Callvantage, a VoIP company that was going to eat SW Bell's children (real phone lines). What's really strange is that about the time that AT&T closed down Callvantage, they also told the FCC they want to turn off all the POTS lines in their areas because they can't afford to maintain them.

With the imminent demise of Callvantage, I went searching for another VoIP provider for our outbound calls. Essentially nothing has changed in the last few years.

While Callvantage had a problem with maybe 1 in 200 calls, the other VoIP providers we recently tried had a problem in 1 in 10 or 20 calls. To make sure we were doing a valid test, we simply unplugged AT&T's Centillium ATA and put in a Linksys SPA3102 or PAP2 ATA in it's place.

Some of the problems were garbled voice, occasional calls that wouldn't go through after they were dialed, intermittent one-way audio (only on some calls), intermittent busy signal on incoming calls (only on some calls), no Caller ID sent, strange Caller ID sent, no dial tone or a busy signal when we went to make an outgoing call  (only on some calls), DTMF digits in the middle of a call ("talk-off" caused by frequencies in some people's voice), etc.

The best I tested was Junction Networks' OnSIP Hosted PBX service (I tried it on a Linksys
PAP-2). Junction Networks is cheap, requires no contract, is easy to setup if you understand how to program SIP information into a VoIP phone/ATA or a SIP trunk (even with DID), and they have a free 30 day trial. When I tested it, maybe 1 in 100 calls had quality problems, and we did notice talk-off (random DTMF digits caused by voice frequencies by one of our employees, maybe caused by the Linksys ATA and not the service). The quality was about as good as I've heard on VoIP (not quite as good as Callvantage), but not always as good as a real phone line from a real phone company.

None of the problems I had with the VoIP lines were resolved by resetting the power to the ATA (unplugging it for 10 seconds and then plugging it back in), although most VoIP providers love to tell their users to reset their ATA. I've always wondered why anybody keeps a phone service where you have to reset the phone line on a regular basis to make it work, but I guess Microsoft has gotten us all used to rebooting stuff (and less than the near 100% dependability of a real Phone Company, because a real CO essentially NEVER goes down).

I've been installing electronic phone systems since the first ones in 1980 (before that, they were electromechanical). I guarantee you that the companies I've installed phone systems in didn't take kindly to the suggestion that they reset their phone system to solve a problem (even old electromechanical crossbar PBXs needed to be powered down once in a while). Doing that during the day can be pretty traumatic at a busy company.

I was once locked in a psych ward at a hospital by the nurses until I fixed their phones, none of which were working when I arrived on the service call on a Saturday. All they had to communicate with the outside world was a single Motorola hand-held radio that security gave them. Many of the early releases of electronic phone systems did require an occasional reset until the bugs were worked out by the manufacturer. The Stromberg-Carlson DBX at that large hospital in 1981 was a real mess. Any kind of outage with their previous electro-mechanical system was extremely rare. Then came stored program electronic phone systems, and problems soared (keeping pace with all the new features that were available once the phone system was computer controlled).

Many of today's phone systems are much worse, requiring resets on a much more frequent basis to resolve strange problems (but then most of today's phone systems are first generation, even if they are sold by a legacy phone system manufacturer - because they are using third world country engineers with no experience to save money when designing new systems). If it's a VoIP phone system where each phone is plugged into an Ethernet port like a computer, most require frequent resets of lots of stuff... the phones, VoIP server, Internet router, etc.

Every VoIP service is going to go totally down occasionally since it's just a bunch of PCs (servers) somewhere providing the talk path, but some that I tested were much worse than others. Most of them allowed me to program in another phone number to direct incoming calls to if the service went down. I never tried it, since we don't receive incoming calls on the outbound lines. You really want to have this ability with any VoIP service you get for incoming calls!

There were some VoIP providers who were obviously screwed up as soon as we programmed the ATA, so we didn't even bother to put them live on the phone system. There were a couple of VoIP providers where we never got them to work at all, even after defaulting the ATA (that had worked with another provider) and using their published setup procedures (including port forwarding in our router).

One thing that surprised me was how callers depend on Caller ID when they answer a call from us. While they were used to seeing Mike Sandman in the past with Callvantage or our real AT&T (SW Bell) phone lines, they were totally confused when it said something like Illinois Call or even Out of Area on their Caller ID box. None of them said PRIVATE on outgoing calls, but many allow you to set a flag for permanent blocking or dial *67 before the phone number to block the Caller ID.

As our customers know, we have screen pops based on Caller ID on our POS (Point of Sale) system. Using the screen pops, we can get their order and get off the phone - usually in under four minutes. If someone calls us and it says PRIVATE, and they have never bought from us, we generally don't even bother to talk to them since we deal primarily with phone companies and phone system vendors (who generally don't try to block their calls). If you get a call from someone wanting to buy something and their Caller ID says PRIVATE, they're probably a scammer of some sort.

Some VoIP providers, primarily the slightly more expensive business grade VoIP providers who often offer "VoIP PBXs" or "Hosted VoIP PBXs" (many of which seemed as bad as the others), let you put in a Caller ID name and number to display from a control panel for the service (Junction Networks OnSIP service mentioned above lets you do that). That would be particularly useful for us, since we don't have a way to answer the lines that we make the outbound calls on, and a lot of people these days just push DIAL on their cell phone to return a call.

Out of maybe 20 different VoIP providers we tried (and another 20 or 30 I called that I decided not to test), the only one we found even close to the quality of Callvantage was Junction Networks. There are other business class VoIP providers who may also have reasonable quality, but the ones I called all required a contract with a minimum time and dollar amount each month. Some required us to put in a dedicated T1 just to try their service (probably the only right way to get VoIP is on a dedicated pipe to the Internet with the same up and down speed, so their service was probably pretty good?), which is a very expensive proposition to see if their service was good or bad. One company was willing to give me one line to test, but I had to guarantee that I would take their service for a minimum dollar amount each month after that. Right.

As a phone man I may be pickier than most, but we've all answered calls from people on cell phones or VoIP lines where we couldn't understand what they heck the person was saying (over 10% of the incoming calls we get on our real POTS lines are like that these days). I sure don't want to project that image when we call out. I can live with 1 in 100 calls being bad to save some money. I can't live with 1 in 10 or 20 calls being bad to save money.

You can try out a single line from almost all of the consumer grade VoIP providers. If you don't like it, you're out maybe $50 including signup fees. Spending a couple of hundred dollars trying a service before you actually commit to using it can save you a ton of money and grief later. Maybe you can put up with the quality issues to save some money on VoIP, but will your customers?

As you might imagine, as a company selling strange solutions to strange phone problems I personally get more phone calls about all kinds of VoIP problems than maybe anybody else in the country. Even if they're not calling me to solve the problem, they seem to have a need to mention it to me because it's my fault - I'm a phone man.

In one day, I got three calls from guys who were calling about something else, but mentioned in passing their VoIP problems. I think there are good days and bad days for everybody, with everything (I'm glad I don't have to fix nuclear reactors!).

The first guy mentioned that during calls a DTMF digit is heard randomly from time to time by the person at the far end of a call, on a VoIP phone system. When I asked him if this was one person, or all the users on the system, he said it was one person. I told him that it was talk-off, a common problem in both the old days and still today with voice mail systems, where a persons voice just happens to sound like one or more DTMF digits. The result is that they are cut-off when leaving a message on a voice mail system because the voice mail system always has a DTMF decoder sitting on the line, listening for digits to control the voice mail. That doesn't happen on a real phone line, since DTMF decoders are only on the line while the call is being dialed. After the connection is made, there's no reason to have a DTMF decoder on a real phone line.

On a SIP VoIP phone system, because DTMF digits are clobbered in the compressed voice packets, almost everybody has "out-of- band" signaling setup on the VoIP device (called RFC 2833). That means that rather than sending the audio for the touch tone to the other end of the call, which may be a voice mail or telephone banking IVR that really needs to hear the correct DTMF digits, the actual DTMF digit is split off the line and a data packet is sent to the other end, telling the VoIP device connected to the PSTN (public phone network) on the other end to make a real DTMF digit onto the phone line.

This works pretty well, but since the VoIP device has no way of knowing whether you have to send DTMF digits on the call, the DTMF decoder has to be listening all the time for a DTMF digit to be sent (on a regular non-compressed phone line the actual tones from the phone/phone system are heard on the other end just fine).

Another problem with RFC 2833 is that although the VoIP gizmo is supposed to split the line and not send the DTMF audio to the other end of the line, sometimes it lets a little bit of that DTMF audio down to the other end of the line. So if the local person dials 7, the other end hears 7 7 (two sevens) - because it gets enough of the actual DTMF digit audio to recognize it, and also the regenerated DTMF digit from the VoIP device connected to the PSTN on the other end. Double digits can be a common complaint with RFC 2833.

When you have a VoIP talk-off problem, only the person at the far end of the line (on a real phone line) will hear the DTMF digit. The person on the end of the call with the VoIP line will usually hear a little click or maybe a little silent period while the DTMF digit is sent, but the local VoIP gizmo line will never send the DTMF back to the local person (because they are supposedly using their own phone to make the DTMF digit in the first place).

I didn't have a solution other than to put duct tape on the users mouth, or maybe kill the user?. The guy called back later in the day to tell me that when they gave her a cordless Plantronics headset, the audio quality was bad enough so that her voice was no longer recognized as a DTMF tone (cordless headsets also compress the voice to some extent, and introduce some distortion). He said the user refused to wear the headset. Like I said, kill the user.

The second guy who called said he was calling from his cell phone because his Internet and phone service was down. He probably told me that because I make fun of him quite often because nobody answers his 800 number, or he sounds horrible (and my POTS lines never go out!). He did mention that he had one POTS line that his T1/VoIP provider was sending all his calls to. The receptionist was answering the line, taking a message, and giving it to people to call back on their cell phones. A pretty reasonable solution if you really must use VoIP for incoming - but you have to make sure the VoIP provider will be responsive enough to forward the calls when requested, or even automatically when they detect the VoIP line is down (not all VoIP providers will do either).

The third guy to call was a friend who is a programmer, and an IT guy from way back. At his current company, all he's doing is programming. Other geeks handle the network. He said everybody has been complaining for a long time how slow the network was, and today was terrible. Later in the conversation he went on to say "That guy you know who put in our phone system just can't get it to work." Since I'm a phone man, everything that goes wrong with phones is my fault.

Now this is a smart guy. Really smart. I asked him if I remembered correctly that he had a Toshiba VoIP phone system? He said he didn't remember. I asked him if the phones were connected to his network, or wired directly to a KSU. He remembered that they all were patched into the network switches. Ah, it's a VoIP phone system.

So I asked him why he thought the phones would work right if his computers won't work right, if they are all on the same network?

I could see the light bulb go off - even over the phone.

Then I asked him if there were two cables from the wall, one to the computer and one to the phone, or if there was one cord from the wall, and the computer was plugged into the back of the phone (most VoIP phones have an internal Ethernet 10/100 switch that lets you run both the phone and PC off one CAT5 cable). He said there are two separate cables coming from the jacks on the wall, one for the computer and one for the phones. I told him that's great.

Why is that great? Because it's easy to patch all the cables going to the phones to their own Ethernet switches, creating a separate Ethernet network for just the phones. No big files to compete with, no bad PCs that are throwing garbage onto the network, just no problems unless there's a bad phone, bad network cable, or bad switch - and you can find those easily by process of elimination.

Even if he fixes the problems with the phones by putting them on their own network, if the switches are plugged into the same router as the computers, he may not have enough bandwidth to the Internet to make many calls outside of the building (his company deals with big databases).

The correct solution if you're going to make calls through the Internet is to have a separate T1 or T3 (a T3 is equal to the bandwidth of 28 T1s). T1s are now pretty cheap, and a T3 can be had pretty darn cheaply these days (for the incredible bandwidth you get). Using a a T1 or T3 is important because both the download and upload bandwidth is the same. DSL and cable Internet are almost always asynchronous... the upload speed is much slower than the download speed. Since VoIP is using the same bandwidth for up and down, VoIP goes to hell pretty quickly on lines that have slower upload speed than download speed.

Another question is whether the T1 provider has enough bandwidth to their router in the CO? If they have 3 customers with T3s, and 40 with T1s, and only one T3 to that router, there are times where none of the T3 subscribers can possibly get the full bandwidth they are paying for since all the T1 provider's customers on that router are sharing the single T3 (equal to 28 T1s).

Keep in mind that in the old days if a T1 went down it got a real priority getting restored. Now, there are so many T1s that you're lucky if they fix it the same day (week?). No matter what, you'll never get as good a voice quality consistently using the public Internet as you would using POTS lines. If you have remote workers, the quality tradeoff might be well worth it for the conveniences of the remote workers using VoIP phones to connect to your office from anywhere in the world?

I'm surprised when people don't see the problems themselves, but I guess I have a gift (or curse?) that lets me see the "system" end to end in my mind when someone describes it to me correctly. Then I start at the beginning thinking of what can be wrong each step of the way. It only takes me a few seconds. I always ask questions in the order I see them in my mind (I don't skip around), the same way I would handle it if I was on the service call myself. I don't think anybody can troubleshoot these types of problems without breaking them down into smaller pieces. When you think of it as a whole, it's overwhelming!  I can't do it.

Here's a typical VoIP related phone conversation:

CALLER: Hello. I'm having trouble with my phones since I switched from the ABC Phone Company to XYZ VoIP service. I've called XYZ, and they say their VoIP lines are fine. It must be my phone equipment. My phone equipment vendor checked the system, and said it's fine, especially since it worked OK last week with the Phone Company Lines.

MIKE: OK, what kind of problem are you having?

CALLER: When someone calls, there's nobody there when I answer the line. If the caller calls a second time, sometimes I can answer it and I hear them, but I can always answer it and hear them when they call back a third time.

MIKE: Sounds like you should switch back to the ABC Phone Company.

CALLER: Oh no. I would never do that. XYZ VoIP is only $199 a year per line, with unlimited local and long distance calling.

Here's another typical conversation:

CALLER: Hello. I just replaced my three phone lines with lines from ABC VoIP service a few days ago. Since then, I can make outgoing calls fine, but when someone calls here, the call is answered by the Automated Attendant on our Toshiba phone system, the caller hears a click, and the call is dropped. Once I figured it out, I turned off the Automated Attendant, but I really need it to answer the lines. We can't always get to them ourselves.

MIKE: What happens if you answer the VoIP lines with a regular single line phone plugged directly into the VoIP box?

CALLER: It works fine.

MIKE: Is your Automated Attendant plugged into single line analog station ports, or is it integrated into your phone system digitally?

CALLER: It's on analog station ports.

MIKE: Then unplug the Automated Attendant and plug the regular single line phone into the analog station port, and see if the phone rings - and you can answer and talk.

CALLER: OK, I just did that and the phone rings and I can answer an incoming call and talk just fine.

MIKE: There seems to be something on the VoIP line that makes your phone system think that the line has hung up as soon as it's answered. There may be an extra CPC (Calling Party Control) signal coming from the VoIP box as the line is answered, which is detected by your phone system, which then sends in-band signaling (like a DTMF D tone) to the analog station port to tell it to hang-up. You're going to have to call your phone system vendor to figure it out, or switch back to lines from the phone company.

CALLER: I can't switch back to the phone company. I'm saving $100 a month with the VoIP lines!

And another typical conversation:

CALLER: My customer is a doctor's office with a small Panasonic phone system. They just switched to Eggplant Phone Service (not their real name), a CLEC in our area. Since they switched, every time the customer hangs-up from a conversation, the phones ring back right away. They answer the line, and there's nobody there. They hang-up, and the call rings in again. I replaced the KSU, but it's still doing it. I took all of the readings on the Telephone Line Diagnostic Table on your web site, and they were all OK except one.

MIKE: What were the readings?

CALLER: The On-Hook AC voltage from Tip to Ring was 106VAC.

MIKE: 106 Volts? Is it a really cheap meter?

CALLER: No, it's a Fluke. I checked my meter by checking an AC outlet, and it seems OK.

MIKE: That's really strange, but it explains why you'd be getting false ringing. Regular ringing is only 90VAC, so the phone system should see the 106VAC as ringing, which it does! There should be less than half a volt of AC on the line.

CALLER: What can I do to fix the problem?

MIKE: Luckily, phone company specs say that the phone company has to reduce the AC on the line if it's above 50VAC, which is because anything higher would be dangerous to a phone man working on the lines (not because the phones or a phone system wouldn't work right). You just have to call Eggplant and tell them you measured 106VAC on the line from tip to ring, and they need to bring it down. 

CALLER: I called Eggplant, but I can't get anybody to call me back. I told the customer to switch back to the Phone Company so I can talk to repair, but he said he's saving too much money to do that.

MIKE: You can take a look at our Longitudinal Imbalance Tech Bulletin where we list the name of a company who makes gizmos to reduce the AC on a phone line. It might be expensive, especially since it's really Eggplant's job to reduce the AC.

And another typical conversation:

CALLER: We're getting our incoming 800 service, our local phone service, and our Internet from one company who installed a box that lets us dynamically share a T1 between voice and the Internet. For the past six months, we've been getting calls that ring once, and when we answer the line, there's nobody there. Our customers keep telling us that they've been trying to call us, but the line rings once, they hear a click, and that's it. They finally reach us later in the day, but I think we're losing business. We didn't have this problem before we switched to ABC Telephone Company.

MIKE: Do you have the T1 going directly into your phone system, or do you have a Channel Bank that breaks the T1 out into separate analog lines that go into your phone system?

CALLER: We have a Channel Bank in-front of our Panasonic key system.

MIKE: It could be a problem in your phone system, or with ABC Telephone Company. Did you call your phone system vendor and ABC?

CALLER: I called both of them, but they both say their equipment is OK.

MIKE: The easiest way to determine what's broken is to bridge regular single line phones onto the lines in-front of your phone system. They will ring at the same time as the phone system does. When you notice the problem start happening, answer the call on the single line phone to see if someone is there. If there's always someone there on the single line phone, but they're not always there when you answer on your phone system, your phone system is probably broken or not compatible with your Channel Bank. If there's nobody there when you answer the single line phone, the problem is with ABC Telephone Company. You may have a bad Channel Bank, it might be programmed wrong, or they may have a problem with the programming on the T1 or a problem in their CO switch. They should be able to figure it out right away by connecting a data scope to one of the lines in the CO, and watching the data as the bad calls come in to you.

Here's one more typical conversation:

CALLER: I need to order some Loop Current Boosters.

MIKE: OK. What kind of problem do you have.

CALLER: Our customer ordered XYZ cable phone service for his hotel. After the cable company installer connected the trunks to their phone system and left, they couldn't make outgoing calls from the system.

MIKE: OK. What was the loop current on the lines?

CALLER: I don't know. When we put the phone system back on the AT&T lines, which are still live, we could make outgoing calls.

MIKE: Well I don't think a Loop Current Booster is the first thing I would try, especially without determining what the loop current was. How did you decide to order the Booster?

CALLER: We sent a technician there and he couldn't get dial tone on the lines.

MIKE: The phone man didn't get dial tone with his butt-set?

CALLER: No, he doesn't have a butt-set. We're a computer and networking company. We don't know anything about phone stuff. He couldn't get dial tone on the PBX console.

MIKE: What kind of phone system is it?

CALLER: A Mitel SX200.

MIKE: Why didn't the customer call the company that maintains the Mitel?

CALLER: The customer just bought the hotel, and we are doing the computers for him. He doesn't know who maintains the phone system. He figured we could do it for him.

MIKE: OK. I think I would take the on and off-hook voltage readings, and the loop current readings for all the lines. You can find our Telephone line Diagnostic Table on our web site, which will let you record the readings and then try to figure out what's different from the old lines and the cable company lines. Oh, wait a minute. Are the lines for the Mitel loop start or ground start?

CALLER: I don't know. What is ground start?

MIKE: OK, I think that's your problem. Most or all Mitels are setup for ground start lines, which are used to try to prevent crashes or glare where somebody trying to make an outgoing call answers an incoming call. I don't remember whether the SX200 uses different trunk cards for loop and ground start, uses jumpers for the card, or is programmable from the console. You're going to need to check whether the AT&T lines are ground start by putting a phone on the pair to see if you get dial tone. If you don't get dial tone, you're going to need to attach a wire to ground like the screw on an AC wall plate, and touch the ring side of the line for a moment to see if you get dial tone.

CALLER: OK. We can do that.

MIKE: If it is ground start, you need to ask the cable company if they can make their lines ground start, or find someone to see if they can make the SX200 loop start. Until then, they won't be able to make calls, and won't be able to receive calls on the AT&T lines that still work for outgoing, since they've already ported the numbers.
 

These problems are happening many times a day all over the country.  Just because a VoIP provider or CLEC says they have a feature doesn't mean it will work the way you expect, work the way it did from the real Phone Company, or that you'll be able to get any support if it doesn't work.

Just because you can buy a phone or phone system, doesn't mean you will be able to make all of the features work.

Some of the worst offenders are the expensive "phone systems" they sell at the Office Biggie store. They have four line corded or cordless "systems" with tons of features, but there's a pretty good chance all of the features won't work right. Sometimes they'll work OK on a real phone line, but when used with a VoIP phone line a lot of features don't work (these types of phones usually communicate on frequencies over the normal voice range, on Line 1). If you buy this stuff, make sure you save the boxes and can return it. A lot of companies thought they could save money by getting this stuff, ended up buying a real phone system, and then sold the expensive junk on ebay (or it's still sitting in a closet).

KSUless phones from the Office Biggie Store are JUNK!  Expensive junk.

VoIP phone systems can be a real surprise. Some features which both users and Interconnects take for granted on a regular phone system are missing or crippled on IP phone systems. I got an email from an Interconnect who was surprised to learn that All-Call Page on the VoIP system they just installed didn't quite work the way it did on most other phone systems - it would page a maximum of three phones. They wanted a good way of getting paging to the other 97 phones they had just installed. Even if a VoIP phone system vendor says they have a feature, it might not be implemented at all (in the current release), it might only be partially implemented, or it might just plain not work right. Will the manufacturer fix the problem? Eventually, but maybe not in time to keep the customer happy, or from returning the phone system.

Most VoIP sets require power to work, either from a power cube at the desk or using PoE (Power Over Ethernet). PoE may be difficult and/or expensive to implement (but it's getting a little cheaper and easier). The power to run the phone comes right from the Ethernet Switch, and it's a very neat installation.

Everybody should probably buy Switches that provide QOS (Quality of Service priority for VoIP packets) and PoE for any VoIP sets they may buy in the future - even if they don't need it right now.  PoE Switches aren't cheap, and some only offer PoE on a limited number of ports, not on all the ports. PoE is a much better idea than having every phone plugged in using a power cube (if they do use a power cube, hopefully they plug it into a battery backup at the workstation). Every part of an Ethernet network should be battery backed, including the switches both inside and outside the computer room.

Some businesses feel compelled to use the all of the features on their new phone system. Everybody wants to get their money's worth from a purchase, but using features just because they're there is not a good business practice. I've had customers actually put their company out of business because they had to use their expensive new Automated Attendant and Voice Mail - when the main thing the company did was incoming sales over the phone. If you give employees the ability to hide behind technology, many will take the opportunity. Unfortunately, that translates to the bottom line when sales start going to other companies who don't hide behind technology, and eventually everybody at the company loses their job when the company goes under.

If you want your company to stand out from the others, use real people to answer the phones!

I'm still amazed when I call a company and the greeting tells me to "listen carefully." I simply dial "0" for the Operator because I just don't have time to screw around with this stuff. Some companies are goofy enough to either not give a caller the ability to dial "0", or they send "0" calls to some company mail box during the day. That's when I call another company.

Some VoIP phone system manufacturers and dealers tell you that you can use their system to make your small company sound like a big company. Sounds like a good way to make your company go out of business, to me.

I can't believe how many calls I get about cordless phones. Sometimes they just don't work because of the construction of the building, interference from other stuff using the same frequencies (like wireless cameras or Wi-Fi), or they're just garbage. The only way to know if it will work is to try it in your particular location. Make sure you can return this expensive stuff if it doesn't work! Ask your phone system vendor if they have one you can borrow for a couple of days before ordering it (you'll probably have to pay for installation).

Cordless, VoIP and cell phones are coming together in a new generation of wireless phones. You can get Wi-Fi cordless phones that work over the Internet instead of a phone line (even at a coffee shop), or a combination cellular and Wi-Fi phone that will switch over to Wi-Fi when you get to a hotspot, like your office or a coffee shop. I'd either wait a while to adopt this new technology within a company until it's ready for prime time, or start buying bigger bottles of Tums.

It seems like "Buyer Beware" is important when purchasing telecommunications these days.

How can an otherwise smart business person
get themselves (and their company) into this?

An awful lot of people are going for the lowest cost deal without trying it, checking references, or asking their phone system vendor about it. Phone service is phone service, right?  NO!

The  Phone Company has been the poster child for bad service for decades (remember Ernestine the Operator on Saturday Night Live?). It's probably worse today than ever, because they've laid off everyone who knew or cared, and now you talk to people who are as close to clueless as possible. Next time you feel a need to ask your local Phone Company something, ask your dog instead. The answer might be better than you get from the people they hire today either here in the US, or in India.

If the first person you talk to at a Phone Company is an idiot, just hang up and call back. Keep doing it until someone answers who you can deal with. Not everybody who works for the Phone Company is an idiot, so you'll eventually get someone who is fairly sharp, and who cares. Call the Phone Company back several times in the next few days to have them read the order back to you (there's a pretty good chance it will be wrong the first time). Have them fax you a copy of the order you just placed!

If this is what you get calling the real Phone Company (who's guaranteed a profit because they're a utility), what do you think will happen when you deal with a VoIP provider? Even if you get wonderful service from a VoIP provider or CLEC, they're probably using the Phone Company's copper to get the data to your building, and they'll  have to deal with the Phone Company if something goes wrong with that copper. Some VoIP providers and CLECs just don't offer the same level of service as the Phone Company, no matter what. They have a limited menu of products, services, and support that they offer. Just because you can get a particular level of support from the real Phone Company doesn't mean you can get it from a VoIP provider or CLEC (most find that out the hard way).

I would suggest that you keep some lines from the real Phone Company. Unless phone calls aren't very important to your business, you need to have a Plan B. If you're crazy enough to get your incoming calls from someone other than the real Phone Company, make sure you have a disaster plan so that they'll re-route your incoming calls to a pre-determined number, like your cell phone. I'm not just talking about a hurricane. I'm talking about losing power to your building, a water leak onto your phone system or computers, a cable cut outside, or even a fire. Cell phones are a good choice for outbound calls if your phone service goes down, as long as the cell sites are still working in your area. Since almost everybody has cell phones these days, outbound calling in a disaster at your company isn't much of a problem. If it's a community disaster, the cell phones are unlikely to work because the cellular system is overloaded.

Having some real phone lines from the Phone Company also lets you do some troubleshooting. When everybody is pointing fingers at each other, the easiest way to determine who's at fault is to replace a VoIP or CLEC phone line with a real phone line (even a fax line would be handy in that case). If the problem goes away, the problem is with the VoIP provider or CLEC. If the problem is still there, the problem is with the phone equipment.

That single fax line might end up doing double or triple duty, also being used for the alarm system, postage meter, and maybe even the water meter's modem.

Many of the phone system vendors tell me that they only find out that their customer has switched to VoIP or a CLEC after  the customer has ordered the lines, dropped their old lines, and things aren't working.

The customer then expects their vendor to make the phone system work with these new lines one way or another. This is stressful for both the customer and the phone system vendor, but many VoIP providers and CLECs don't care - they figure it's not their problem, and they know they've got you because you don't have any real phone lines, and you've signed a contract for a zillion years to get the lowest rates. Some of the VoIP providers and CLECs will go out of their way to help their customers. Without doing some research, maybe trying a line or two to see if it works with your equipment, and checking some references before you jump in, you're really behind the 8-Ball...

Call your phone system vendor  before  you make a switch in who provides your phone lines!

Some VoIP providers specifically target simple residential or small home office customers (the low hanging fruit), but a business will order VoIP service from them because it's so cheap or they want a particular feature. In many cases the VoIP provider just doesn't have the level of support needed for a real business customer. It's like buying a used Yugo or a Chinese car to drive from New York to LA, and expecting a gas station to be able to fix it when it breaks down.

There's nothing inherently wrong with VoIP.  It's just a way to digitize and transport the digitized voice from one place to another. There's no reason the quality has to be any worse than a traditional voice T1, which is digitized using a different technique (but VoIP is more prone to echo). Besides the fact that the equipment that most VoIP providers use is not "Carrier Grade" like the real Phone Company uses (which makes VoIP somewhat less dependable), the biggest difference is the bandwidth that's available to send the voice packets from Point A to Point B.

One of the problems with the modern telephone network is that a call starts out as analog from the mouth of a guy on a telephone handset (which is always analog). It then gets digitized and un-digitized many times before the sound gets to the ear of the guy using a handset on the other end of the call. It's very unlikely that a call would be analog from end to end these days. Phone system manufacturers use one method of digitization, the Phone Company uses another method of digitization, and VoIP stuff uses still another method of digitization.

As analog voice is digitized, some of the information is lost. When it's changed back to analog, a little more information is lost. The analog to digital and back sequence can happen several times on a single call, resulting in quite a bit of lost voice on the other end of the call. Most of us are used to this new mechanical sounding voice, so we don't seem to care. When analog touch tones go through that process, they can get clobbered and the voice mail or whatever may not work right on the other end of the call.

VoIP devices usually hear the analog touch tone on the sending end, and covert it to data. The VoIP device at the receiving end sees that data, and makes a new touch tone right out of that device - which allows most devices like Automated Attendants and Voice Mail systems to work correctly.

Other digitization methods (like TDM and PCM - not VoIP) theoretically have enough bandwidth reserved so the touch tone audio should make it to the other end, the touch tones can get distorted as they are digitized up and back via the different methods various phone companies and phone systems use. In some cases, the DTMF tones that are dialed just won't be recognized dependably if any part of a call is VoIP - on either end. Do a lot of testing with real calls before making a system live!

One other result of several digital to analog conversions is echo. Echo is caused by several factors, the main one being that a digitized telephone call is four wire - with separate transmit and receive. A traditional phone line is two wire, with both sides of the conversation on the single pair. A device called a hybrid transformer is used to convert four wire to two wire, or back. The hybrid transformer is simply some windings of wire around a metal core - but it's not simple. No matter how well the hybrid is made, it's not 100% efficient, which means that some of the transmit and receive audio gets mixed and sent back to the other side of the hybrid transformer. That's called sidetone.

On a regular analog phone call, the sidetone caused by the inefficiency of the hybrid can't be heard because there's no delay. The sidetone created by the inefficiency is increased if there's an impedance mismatch between the analog line and the hybrid, but again you don't notice it because it's pretty far in the background, and there's no delay.

When the conversation is digitized, there's a slight delay, which means that sidetone is heard a fraction of a second later - which is why you hear an echo. Most VoIP equipment has echo cancellers, but there's a limit to how much echo they can cancel. Echo is increased if you have two analog to digital conversions inside the phone system, like an analog phone line going into a VoIP phone system, and then someone talking on a call from that line from an analog station port. Now you've got twice the echo to deal with, coming at you from two places within the phone system. It would be better to give users digital phones, instead of old style analog phones that require an extra D to A conversion. The fewer analog to digital conversions you can do on a call involving VoIP, the better!

Some day well into the future, voice will be digitized and transmitted as digital data from end to end on a phone call, where it's converted back to analog just once in the handset (until we get digital ears?).

One of the things that companies have had to deal with as they ordered digital lines is that they can no longer easily record their employees' conversations. There are some very expensive devices that you can hang on a T1 line, and devices that will work with some models of phone systems that decode the digital voice and send it to a voice logger (recorder). If you don't use one of these devices, the only place you can get analog audio on a digital phone system using digital phones, is at the handset. That requires a little gizmo be attached to the handset or headset jack on the phone, which sends the analog audio back to the voice logger on a spare pair of wires.

If you're rewiring your office, it would be a good idea to put in two CAT5 cables to each desk. Having an entirely separate CAT5 network in your office for a VoIP phone system will give you a lot less grief than trying to run both voice and data on the same network.

Some business class VoIP providers insist on your getting a T1 just for their lines. While you could conceivably put the T1 router on your network and segment it so the VoIP phones get their packets from the T1 router used for VoIP, it's much simpler and more dependable to have two totally separate networks. You'll appreciate this advice when nothing works right and your computer guy can't figure out why.

There are real benefits to not having voice compete with the data traffic on your network, like while you're downloading a large file, large database, youtube files or porn (gosh, I'm sure that would never happen?!?).

If you insist on running just one network, most VoIP phones have a built-in 10/100 switch with an extra CAT5 jack on the back for you to plug the PC into - so you don't need two CAT5 cables at the workstation. Then they're both running on the same network.

Do you have an enemy in business that might want to turn off your phones? They can do it if you're using VoIP. All they need is the IP address of your DSL, T1 or whatever you're using to connect your network to the Internet. DoS (Denial of Service) attacks are pretty commonplace. The web sites of the White House, Microsoft and lots of businesses across the world have been literally shut down for hours or days, and it could happen to you.

While there are lots of ways to screw up your business' computer network to the point that it slows down (you've probably noticed it before), which will make VoIP calls so garbled that even if they go through the conversation will be unintelligible, the easiest way is to "rent" some home computers that are infected with a Trojan Horse from a Russian underworld character. They program all the PCs they've taken over to try to reach your IP address at the same time - and you're out of business.

The most famous use of Russian etc. "botnets" is during a popular sports event like the World Series or Superbowl, where the Russians extort the owners of online sports betting sites. They tell them that unless they pay $X00,000 up front, they'll send all their botnet PCs to the betting site for a day or two before the game, so nobody can reach the site. For the betting web sites, the protection money is just a cost of doing business. Can it happen to you? There's no way for you to stop it if you use VoIP phone lines.

Even if everything works OK, you might have voice quality issues with VoIP:

The frequency bandwidth of a traditional telephone conversation is around 3,000 cycles per second (from around 300 to 3300 cycles per second). It's not like listening to a radio or TV station, but it's OK for carrying on a conversation. When those 3,000 cycles are digitized for a voice T1, they end up taking about 64Kbps of data. You can fit exactly 24 of those digitized voice conversations on a traditional T1, which reserves the bandwidth for each conversation so it's always the same quality.

Now days, everyone wants to (has to?) stretch everything to get the most bang for the buck. The lowest price wins out, even if it's not the smartest solution. Even if it doesn't work... "If there's a problem we'll deal with it later. I'll get my bonus this quarter."

When cellular phones came out, it became OK to have bad sounding telephone conversations. While the original analog cell phones had the same 3,000 cps reserved for voice, they sounded bad because the analog radio signals would break up with static or noise.

When the cell companies decided to digitize voice (analog cell has been turned off in most of the US), they realized that we were already putting up with crummy voice quality so they used their new digital system to compress some of the traditional 3,000 cps of audio to something that sounds pretty bad. The more conversations they can carry, the fewer times the customer will see "Call Failed" (even if the calls that don't "fail" sound like garbage). Sprint is famous for that, with most calls sounding garbled.

It's not just telephone traffic that gets compressed. Anything that can be digitized can be compressed. You've probably noticed the effects of compressing a satellite TV channel, or a satellite radio channel. You can certainly see the effects of compression when you watch a video on the Internet, where the picture is small and the quality low (but the audio usually sounds pretty good because it takes the least bandwidth).

The reason that all this stuff is compressed is simply so you can fit more of it into a single pipe of limited bandwidth. Amazingly enough, VoIP calls don't take up a lot of bandwidth. You can make many internal calls on an office computer network (LAN) and they'll sound great. You can make some calls on a broadband Internet connection, as long as you have the same bandwidth to the Internet as from the Internet.

ADSL lines, the type of DSL that the Phone Company usually provides, limit the upload bandwidth but give you a pretty big download bandwidth. That works fine for downloading movies or music, but since telephone calls are two-way, making several simultaneous VoIP calls on an ADSL line often  works badly.

The real problems show up when you try to make VoIP phone calls on a broadband connection that's also being used to surf the web or for email. At that point, the voice calls are competing for the limited bandwidth with web pages and email. You'll hear the quality of the VoIP call get pretty bad as someone starts to download (or upload) email, music or a web page. Modern Ethernet Switches can use QOS (Quality of Service) rules to give VoIP packets a priority, but few companies have the equipment, or have it set-up properly.

Even if the VoIP packets make it through to the Internet on your broadband connection, you have no control over how those packets get to the other end of the call. With a web page, if you see the bottom or the page before the top for a second, nobody knows or cares. With a VoIP call, if the person on the other end of the call hears the second half of your words first, that ain't good. Luckily, there's a lot of bandwidth available at most ISPs and on the backbone of the Internet, so that's not the main source of bad sounding calls today. In the future, the entire Internet might have QOS so that VoIP packets will get priority no matter where they go, and calls will sound quite good?

Rather than using your regular Internet connection for VoIP, there are companies who will give you a T1 on their private digital network where the packets don't have to compete with regular Internet traffic (often called an ATM network - which is not Cash Machines). The voice quality on that network, or even on your own private T1 between branch offices, will be as good as a regular phone company POTS call (unless you purposely limit the bandwidth of calls to stuff more on the pipe).

If you're getting VoIP strictly to save money, you won't like the private network solution because it's expensive. If you're getting VoIP for the features, and you have a business need for the features which will either allow you to save money, make more money, or provide better service to your customers, a connection to a private ATM network might be just what you need to get your company into this leading edge technology.

Some cable companies are offering telephone service that they call VoIP. It's "hip" to use the word VoIP these days. Although the cable company is digitizing the telephone call using IP packets, which makes it VoIP, they aren't sending the packets out over the public Internet, and they're prioritizing the packets as they leave your premise so that TV programs or Web surfing won't affect the quality of the call. Although the quality of the voice will be about the same as the real Phone Company, you're still going to be dealing with a box or features that may or may not work well with the particular phones or system you have connected to it. The box also needs electricity to run, so you need a good battery backup/UPS to be able to use the phones during a power failure.

Remember that if you're using VoIP over your Internet connection, if the Internet goes down, you won't have telephone lines, OR email and the Web.

It's an insane idea for most businesses to switch 100% to VoIP. If your business is telephone intensive and you're selling something, it would be a good idea to use real phone lines where the quality of the call won't become an issue in the sales process. Just do your homework before committing to any of this leading edge technology - so it doesn't become bleeding edge for your company.

A VoIP Provider is NOT a real phone company. If you know that going in, you should be able to successfully implement VoIP at your company... Because you'll have realistic expectations.

A little background is necessary to understand what you're getting into by being a first adopter of VoIP. This is still early in VoIP's development. You're participating in a big experiment.

In the beginning, there was the Phone Company. Only one Phone Company. You got your phone lines from them, and you rented your telephone equipment from them.

They weren't perfect, but it was all we had, and we could make and receive calls just fine. As a matter of fact, they sounded pretty good compared to today's cell and VoIP calls.

The phones that the Phone Company rented out were as tough as nails. The 500 set (rotary dial), and then the 2500 set (touch tone), became the standard by which all other phones were judged. When the Phone Company finally came out with Decorator Phones (like Mickey Mouse, or a fancy phone in a box), the innards always had the same components as a regular 2500 set.

The 2500 set is still the gold standard for both the Phone Company and VoIP providers. They have the same attitude... If the line works with a standard 2500 set, then the line is fine and a problem you're having with your phone system is your problem (click). Unfortunately, nobody actually uses 2500 sets these days (except my Mother?). They're using all kinds of electronic telephone stuff. Even the expensive multi-line feature phones you buy at the Office Biggie store don't work on every phone line (but hopefully you can return them).

When the Phone Company was broken up into a lot of smaller parts, it became legal for other companies to sell you telephone equipment. Eventually, you were also able to get your phone lines from other companies. The first generations of telephone equipment that were sold to the public were still tough as nails. A lot of the designs were licensed from Western Electric, who designed the original equipment. Today's telephone equipment is not tough as nails (it's closer to spaghetti). Almost everything is designed as throwaway - figuring the maximum life may be a few years (months?).

The engineers who design today's telecom equipment, the factories who produce it, and the companies who sell it are so far removed from the quality standards of the Phone Company that there is no comparison in the equipment... But, for some reason (that I can't figure out) consumers and companies assume that every piece of telephone equipment that's made is still up to the standards of the Phone Company of yesterday. No way!

Purchasing phones or phone service can be pretty stressful!  Or maybe I should say what happens after you buy the phones or lines is stressful.

The CPE (Customer Provided Equipment, which is phones or phone systems) that's available today is essentially throw-away. Phones that were rented in the old days would often last decades. While some of today's equipment can be fixed if it breaks, most of it can't be fixed economically because of the way it's made (surface mount components that are hard to replace). It's just cheaper for the manufacturer to send you a new gizmo than to try to repair the bad one. If you do send your old one in for repair, they'll often throw it in a pile and sell it for scrap, or maybe send it back to China to be refurbished in the future.

How often telephone equipment breaks isn't that big a deal these days. If it works out of the box, it's usually pretty dependable and won't just out and out break until you throw it on the floor.

In some applications phones don't work right out of the box. The engineers who design today's telephones, phone systems or VoIP stuff come from a digital world of 1's and 0's. The way 1's and 0's act in a digital device is always the same if the device isn't broken. Analog is much more complicated - closer to an art than science.

Almost all of the telephone instruments made these days are made to be as "hip" looking as possible. Most manufacturers have odd shaped phones, odd shaped buttons, and odd shaped handsets. While in the old days you could get an amplified, noise cancelling, or push-to-talk handset for just about any phone, most phones made today are incompatible with anything else on the market, and the manufactures don't even think about the special stuff that their customer may need, like an amplified handset.

Companies buy new telephone systems, and forget that they had special stuff like loud ringers or amplified handsets attached to some of the phones, and the same stuff isn't available for the new system.

For many years, the "key system" has been the most popular type of phone system. In a key system, there's a button and a light for each line, and a hold button. While you can intercom someone and tell them to pick up a line, a lot of people just yell over and tell someone to "Pick up line 4."

Larger companies with a lot of lines use a PBX. Since you can't get hundreds of buttons on a telephone in a larger business (and even if you could there would be too many to use), a PBX lets a user dial a code (like 9) to get an outside line, and automatically answer whatever incoming call that rings their phone (or it might ring a couple of "loop" keys). Users hate going from a key system to a PBX because they're forced to transfer every call to someone, instead of just putting a call on hold and telling them to pick up a particular line.

Most of the VoIP phones and systems sold today work like a PBX, where you can't see the status of every line. Maybe you can stick a couple of lines on most phones, but you usually can't make it work like the old familiar key system. Since there are a limited number of buttons, you don't get to have very many BLF/DSS (Busy Lamp Field/Direct Station Selection) keys that show you if someone at another extension is on the phone. If you've never had these features, you probably won't care. If you had them on your old system, you'll wonder what kind of idiot designed this new system.

A worse problem on a lot of the VoIP phones is the lack of sidetone. Sidetone is when you hear a little of your own voice coming back to you in your ear, as you talk on the phone. That gives you a nice warm feeling that the phone isn't dead. Dead  is the exact description of what you hear when you talk if a phone doesn't have sidetone. Some people can get used to it, and some can't (I can't), even on my cell phone.

You also lose the ability to record from the handset jack of that kind of phone, since you only hear one side of the conversation, the outside caller, if there's no sidetone. The nice thing about recording a call from the handset jack on a normal phone is that you hear both sides of the conversation, and the levels are usually pretty balanced so that the person inside isn't a lot louder than the person outside. If you don't have several users actually try a phone before you buy a lot of them, you could be in for a mutiny!

Telephone Systems are based on Standards

Since the beginning, all telephone equipment was installed to "standards."  The purpose is so that any phone man can come in and fix or add-on to the phones or systems. A phone man from Chicago could work in Podunk or New York the next day, and there would be no difference in how well he performed his job.

The standards are extremely simple to follow. If you can count to maybe 100 and can remember some colors, you're 90% of the way to becoming a phone installer. Programming is harder, but once you know a system it's easy.

Buying a phone system that's an odd-ball means that the installer/programmer is not familiar with it, and they will never get enough experience with it so the customer will be happy.

Unfortunately, many of the companies selling phone systems today don't employ phone men, don't know the standards, and don't care.  Does the stuff they install work?  Most of the time it kind of works initially.

The problem comes when the company installing a phone system makes their own rules for installing and programming it. If the system needs repair, it may be impossible to find the guy who installed and programmed it, and there may be no humans that could figure out what's wrong in a reasonable time.

The idea behind standards is that systems can be installed, programmed and repaired quickly enough so the company doesn't lose business - and there will always be someone available who's knowledgeable about the system - even if it's from another phone system vendor.

A good example of a very dangerous phone system is the Asterisk, or any of its decedents or copies. It's main benefit is that it's incredibly flexible, has a ton of features, and it can be hacked to do almost anything. It works reasonably well, but the uptime is never anywhere near a phone system made by a legacy phone system manufacturer. It's inexpensive compared to any other phone system (the software is free), but what you don't pay for in hardware is quickly made up by the maintenance contract or hourly fee to a geek who's stayed up nights for a year or more to figure the thing out. These guys are truly rare, and you don't know who's really an expert (or not) until you've invested a ton of money in the guy.

The main problem with the Asterisk phone system is that it's incredibly flexible, has a ton of features, and it can be hacked to do almost anything.

Even if you find a Linux geek who can program the system to do exactly what you want, if something happens to that geek you're just out and out screwed. It would take you days or weeks just to find someone who would be willing to try to figure it out, and then it would take them days or weeks to try to understand what the last guy did.

There are essentially no standards with an Asterisk system, unless it's just programmed to make and receive calls. I've been programming phone systems for 30 years, even in binary, octal and hexadecimal, and I couldn't figure it out in a reasonable time if there was a problem.

When you buy a phone system, the number one thing to do is check the references. If the vendor doesn't have similar systems, programmed in similar ways, used by similar types of customers are yours, and you still buy the system - you are really screwed.

Most VoIP telephone equipment is designed by engineers
with no experience with phones

Most of today's engineers who design phone systems (in a third world country) look at analog connections to a phone line or phone as easy stuff that's pretty much beneath them. They figure the analog telephone connection is simple, and they whip that out in no time. Problem is, the analog connection isn't simple. It's actually very difficult when you consider all of the possible combinations of old and new devices that can be connected together. Unlike a digital interface, which is simply trading 1's and 0's between two pieces of equipment, the analog interface deals with AC and DC voltage and current, db levels, impedance, resistance, noise, induced AC and imbalances.

Most of the VoIP boxes on the market today do a poor job of emulating a real phone line. One side of the coin is bad engineering, but the other side is cost. The manufacturers feel that they don't want to put an extra 25 cents worth of parts into a device if it's only going to be needed by 20% (or less) of the users. That works OK for the 80%, but it causes real headaches for the customers who buy the stuff expecting it to work, and it doesn't.

Most of the VoIP providers don't care. The 80% are giving them so much cash that they don't feel a need to start doing the much harder work of getting the other 20% working - so they don't. Most VoIP providers are perfectly happy to take the VoIP device back if it doesn't work in your application, which costs them very little. The problem is that the customer thought it would work OK, and they went through a lot of pain and expense to try to make it work.

Good VoIP works fine 99.5% of the time when the application is simply making outbound calls by a real person. It's when you try to do something as simple as putting an answering machine on the line when things start to go down hill.

People pick up a phone, make or answer a call, and hang up the phone when they're done. Automated Attendants, Voice Mail, answering machines and automated telecom devices expect to see cues that help the machine deal with the call.

Some VoIP devices (and even T1 Channel Banks) ignore the standard cues (protocols) that have been used on US telephone lines for decades. In today's global economy, the Chinese VoIP box that you bought here is probably sold in 50 other countries, some of which use different cues. The VoIP providers are looking to install as many lines as possible, and they don't really care whether the box is made to work well in England, China or the US. They just use it here in the US because they need a box that's cheap and easy to get.

When your telephone equipment gets the wrong cues it may not ring correctly (besides having the wrong ring voltage or not enough current, some phone system COs won't recognize ringing other than regular 2 seconds on, 4 seconds off), the voice mail may never hang-up at the end of a message, the volume may be low or high, or you may have echo.

Analog station ports on mainstream phone systems usually don't do a very good job of emulating a real phone line. If you're thinking of plugging a station port from a phone system into a VoIP gateway, the gateway may not recognize the station port as a real phone line because there's not enough talk battery, loop current, ring voltage or the wrong ring cadence.

While you may be able to fix these problems by adding voltage or current, changing the impedance, attenuating the audio, or getting a witch doctor... you may not be able to fix it even after playing with it for hours. That's not  a terrible thing if you want to be on the cutting edge of technology and really need the features or cost savings, but it's INSANE to just order the stuff and disconnect your old stuff - assuming everything will work fine.

Some VoIP providers will help you setup an ATA or Gateway so it will work with US type equipment. Unfortunately, most won't because they're scared of all the setting they can change (it's pretty intimidating!). Again, they just want to deal with the low hanging fruit that's not going to make them do much work for your money.

Some VoIP equipment manufacturers are touting HD VoIP, which is basically high fidelity voice over VoIP.  HD VoIP would work fine if there's enough bandwidth, but that's a real stretch if you consider that nobody can make regular VoIP calls sound as good as a real phone call, primarily because of bandwidth constraints.

The closest to delivering high fidelity VoIP for everyday use is Skype, which must be between two computers running Skype software (their new SIP service for business uses standard telephone quality bandwidth). If you use a good quality headset or microphone, the quality can be great (depending on the path the packets take over the public Internet). Better than AM radio quality. If you have enough bandwidth and a webcam, you can even see the other person you're talking to (I don't think they allow conference calls when using video).

There are companies making expensive ATAs for Skype that allow you to put a Skype phone line into your phone system, just like a line from a phone company (with standard quality audio bandwidth). That will be standard quality.

While that's not for everybody, it could really change the business model of an International company. Skype users already know Skype isn't all that dependable, so they may not hold you responsible for the bad sounding calls and cut-offs (like an American would)?

Even if Skype's voice quality is better overall on PC to PC calls, if you make a Skype call through a real phone system you're back down to the traditional limited bandwidth of your real phone system - 300 to 3500hz.

Troubleshooting VoIP can be difficult!

When dealing with an analog phone line, you can hear a problem. The problem is somewhere on a dedicated connection (for that call) between the two parties. If it's a local problem, that's fairly easy to troubleshoot by going to the demarc and listening for the problem. If it's still there, it's a phone company problem. If it's not there, it's an inside wiring or phone system problem.

With VoIP, there's nothing for a tech to listen to. A conversation is made up of packets from outside, which are then routed inside to the correct VoIP device. No direct connection... Just a few small packets of data to setup the call, and then many small packets of data for the voice. You have no way of knowing where those packets are going to or coming from, or where the problem is... Without some equipment that lets you look at those packets (a packet sniffer).

You can use a free packet sniffer program called Wireshark on a Windows laptop to look at the packets (Wireshark used to be called Ethereal). It has some amazing capabilities to filter the VoIP packets from everything else so you can see the call setup packets (RTP), and the voice packets (UDP) for a SIP call.

Wireshark also lets you "play back" a VoIP call you've captured through the laptop's speakers! You can hear just what the call sounded like.

It definitely takes a (brave) geek to troubleshoot VoIP packets on the Ethernet network, but it can be done. Training is available from Wireshark University:  http://www.wiresharktraining.com/

Laura Chappell is the geek who runs that company, and is probably one of the only people in the universe who likes diagnosing Ethernet problems by looking a the packets (but Wireshark's VoIP shortcuts make it a lot easier these days). Her on-line training is worth it, since it will save you a lot of time learning this stuff.

The problem with using a packet sniffer is that you need a way to attach it in-series with the VoIP phone, VoIP phone system, switch or router. To do that you need an Aggregating Ethernet Tap, which goes in-series with the phone or router's CAT5 10/100 network cable (not gigabit), and has an Ethernet port or USB connection for your laptop.

An Aggregating Tap takes both transmit and receive, and sends both streams of packets to your laptop on one Ethernet or USB connection. A non-aggregating tap will send transmit or receive, but not both. You would need two Ethernet NICs in your laptop to see both transmit and receive from non-aggregating taps, or use two PCs and merge the two trace files.

We do sell the EtherShark™ Aggregating Tap ($799.95, or 3+ at $749.95) to use with a modern notebook (with USB 2.0) and Wireshark software (free), if you'd like to try to learn this stuff (you could also run a softphone on your laptop with Wireshark to see if you understand it, and not use a Tap at all) . Watching Laura's 83 minute Wireshark 101 Training Video for $29 would be well worth it:

http://www.chappellseminars.com/s-wireshark101.html

Is the Perfect Storm coming to VoIP?

There may be a perfect storm coming for VoIP phone calls. There is a big push to allow people to download movies over the Internet. Movies, especially HD movies, are just plain HUGE files. Unlike phone calls, they can simply stop and buffer for a little while if the Internet is slow.

Comcast is touting a new very fast Internet connection over their cable service. AT&T's U-verse and Verizon's FiOS are close behind, or in-front of Comcast. There is almost no need for an Internet connection that fast, except  for downloading movies (main-stream and adult!).

While it's likely that many will stream movies at night, when you're probably not making business related VoIP phone calls, those with a slower Internet connection may start a download or two before they leave for work in the morning so the movies are downloaded and ready when they get home from work (an HD movie takes a while!).

To put it simply, there isn't enough bandwidth in the country if everybody starts downloading movies on a regular basis. In normal times, the Internet might get slow and choppy for a while until the owners of the backbone and the ISPs buy new bandwidth and the equipment to support it (normal growth pains).

There's already plenty of fiber in the ground to support the bandwidth increase, but there may not be any money available to buy/rent it, or all the very expensive (Cisco) routers needed to make use of it.

With the credit crunch, a perfect storm could be looming on the Internet, fueled by the popularity of videos and movies - but there may be no money to pay for it. VoIP phone calls and video conferencing are the two applications where you can't buffer any part... it has to happen in real time.

I have a feeling there's more money to be made in movies than VoIP, so VoIP may be in for some rough times in the future.

VoIP is easy for an ISP to purposely screw up

If you're planning to use VoIP between two locations (like an office and branch office or home worker), consider using a VPN (Virtual Private Network). A VPN will encrypt the SIP (VoIP) packets that carry the conversation (and any other data you send like email and spreadsheets, etc.). That encrypted connection is called a VPN tunnel.

VPN routers are fairly inexpensive and easy to setup these days. You'll need one for each end of the connection.

Since a VPN is encrypted, it prevents tapping your phone lines when they leave your building. It's very unlikely that someone would try to intercept the network traffic between your locations, and if they did it would be very hard to do unless they could get right onto your network or DSL/cable/T1 leaving your building.

The FBI does have a box at every ISP, including yours, that lets them grab all the packets from a particular IP address (like yours), which gives them the ability to monitor and record VoIP conversations (and everything else you send or receive over the Internet). I would imagine that they have the ability to break VPN encryption, but not in real time.

The real reason to encrypt your VoIP traffic is to prevent your ISP from blocking/degrading the quality of VoIP calls by messing with the SIP packets, which are easy to identify as they go through their routers.  Why would they do that?   Because they probably are or own  a phone company in addition to offering broadband. If your VoIP sounds terrible, you're more likely to switch to their phone service (which is almost always a better quality than VoIP, even if they call it VoIP themselves).

Some countries, especially China, monitor all of the traffic on the Internet - and have been known to automatically inject a BYE packet intermittently, which tells the other end of the call that the other party has hung-up.

If someone decides to hack the software on the Cisco routers that are used to route all of the packets on the Internet (which has happened), they could put a little bit of code to send a BYE in response to every SIP packet, which would bring down everybody's VoIP that isn't encrypted on a VPN (routers don't have the ability to break the encryption of a VPN on the fly).

The bitterness of poor quality is remembered long after the sweetness of low price has faded from memory.

Aldo Gucci, 1938... And he never used VoIP!

See the VoIP Checklist for a quick list of things to check before jumping into VoIP.

See the How To "Seem Like  Large Company"  Implementation Tech Bulletin

  

630-980-7710

Copyright © 2010 • Mike Sandman Enterprises, Inc.