Starent PDSN

We continue working to deploy new Wireless Data plans. This week we’ve stumbled on an issue in the edge router. It does not appear to be properly detecting the OSPF routes. This problem manifests itself when the 4th octet of the IP address assigned by the PDSN to a handset exceeds “128”. Whenever a higher number gets assigned all data calls fail. Doing a stare and compare with other IP address ranges in the router shows the difference. But so far the cause cannot be explained.

Nortel has proposed flushing the OSPF cache on the PDSN with this command:

clear ip ospf process

After the command has been entered, we’ll monitor use of the use of the erring address space to see if data sessions having an IP Address above .128 succeed.

SPAM, Sendmail and favicon.ico

I’ve been working on site administration for the last couple of days. To reduce the volume of SPAM-bots registering on the message boards I installed a modification in early June. Unfortunately I neglected to test the modification adequately and it had the unintended consequence of stopping not only the SPAM-bots but also legitimate users. My apologies if this frustrated any of your attempts to join for the last couple of weeks. Please do try again, registrations are again possible. Oh, and the SPAM-bots appear to be held at bay. For now.

I’ve also been working on site email for the last couple nights. What a wonderful voyage of Sendmail discovery this is becoming. The site is able to send mail from the domain teletips.net. Yet I still need to figure out Masquerading so that the hostname of the MTA is removed from the sent mail. Then some aliases need to be configured. Much more to learn about Sendmail.

Lastly I’ve been trying to sort out an issue with that pesky little favicon.ico. I use mod_rewrite in Apache version 2.0.55 to serve all the requests using a single GIF. Here’s that rewrite rule:

RewriteRule .*favicon\.ico$ /images/favicon.ico [PT,NS]

This rule will convert any request which ends with favicon.ico to same filename in the /images directory. The PT flag means Pass Through; NS means No Sub-requests. Except the desired result, the little 16×16 icon in the location box and in bookmarks is not displayed on every page as I would like.

Overview of E-9-1-1

I was recently asked to provide a parlor brief on Wireless E-9-1-1.

Wireless E-9-1-1 allows a handset to be located.

There are three phases of E-9-1-1:

  • Phase 0 provides the call-back number only, no location information.
  • Phase I provides the location of the cellular tower from which the 911 call was made.
  • Phase II provides the most accurate location, often within a few meters, depending upon the location technology deployed by the service provider. If for some reason a phase II “fix” is not possible during a given call, the system should fallback to Phase I location.

Location accuracy depends on which phase your service provider and the local Public Safety Access Provider (PSAP) have implemented. If phase II has been implemented the accuracy also will depend upon the wireless network technology (such as GSM or CDMA) and on the location technology used by the wireless service provider (such as TDOA, E-TDOA, NAGPS, AOA, etc.  Wikipedia has a good article.) PSAP generally are police departments, fire departments, and other emergency services.

When the PSAP and the service provider each deploy E-9-1-1 the lattitude and longitude of the cellular towers is configured in PSAP location software and equipment. The PSAP will add street address information, and usually both are overlaid on a map of the covered area.

All of the PSAP in a given area agree where the boundary between each PSAP should be drawn on the map. This information is shared with the service provider, or, more commonly, is shared with a wireless location service provider such as TCS or Intrado, who provides location services under contract to the wireless service provider.

At the time an E-9-1-1 call is made, the wireless service provider’s switch (the switch has many names: MSC, MTSO, MTX, Central Office Exchange, there are others) signals the wireless location service provider using SS7 (Signaling System 7, which someone with a poor grip on reality called “The Intelligent Network”.) This signaling provides a way for equipment at the wireless location service provider to determine the location of the caller, which is then compared to the previously agreed map of PSAP boundaries.

When the proper or most likely PSAP is determined then the wireless location service provider signals the switch to route the voice portion of the call to that PSAP, where a call taker will answer and the location of the caller is displayed on the map, and along with other information about the call.

There is much more. But this is just a high-level overview.

Deploying Wireless Data

For the last few days I’ve been working on defining new data products in a CDMA network. This is my first foray into wireless data so most of the actions are completely foreign. There seems to be no comprehensive procedural guide. Our vendor has Subject Matter Experts (SME) for each of the various components. But their expertise is defined by the platform for which the are responsible, and no one has been able to provide a cogent over view of the overall process for defining and implementing these product in the network. I plan to take a few posts to try creating a comprehensive guide.

First let’s cover the systems and hardware that’s involved. I’ll emphasize those that must be manipulated as part of the implementation process. We’ll start from the handset and work in towards the cellular switch and then outward towards the service application.

  • Handset. Must be capable of interacting with the service application.
  • The radio network, aka “Base transceiver Sub-system” (BTS.) Not generally manipulated on a per-product basis, mentioned here only for completeness. We’ll focus on a 1xRTT network for now.
  • The Packet Data Serving Node (PDSN.) Basically a router on steroids.  We have a message board on the Starent 16000 PDSN.
  • The Authentication Authorization and Accounting server (AAA.) IT folks will recognize this as a RADIUS server.
  • Core or Edge router, depending on your perspective. Fire-walled access to the Internet.
  • Mobile Telephone Exchange (MTX.) This is the switch, which includes the HLR.
  • Service Application. This can be the MMS-C, WAP Gateway, ASP, hosted content, or any of dozens of application servers. Which one depends on the actual data service being deployed.

In this series of posts we’ll detail the roles and configuration requirements of the PDSN, AAA Server and Edge Router. Others will be discussed in support of various examples.

Telecoms racks needn’t be orderly

We’ve started a forum on Teletips Network called Cable Conundrums for sharing pictures of telecoms racks. Our first post doesn’t actually look too bad; though the camera was uncharacteristically kind. In person the racks seem far messier. The pictures show these racks from the front. From behind the cabling looks far worse. Dangerously so. There have been several times when someone working in the rack accidentally bumped a cable and caused a hiccup to one of the applications running in that rack.

This has caused some discussion with my colleagues about what is the appropriate level of effort to expend dressing cables and in general attending to the cosmetic aspects of rack ownership. Given the relative rarity of such service affecting event, we generally agree that the time we saved by doing so little of this was probably worthwhile. To have spent more time neatening things as we installed them would’ve slowed us at a critical time and, in fact, has not seemed to have hurt us since. Maybe it is simply too soon to tell.

Something we have not agreed on is whether Ethernet switches should face the front or back of the rack. In the pictures they can be seen facing the front. But this has contributed to the cabling mess: cables exit the switch from the front of the rack and all of them must curl around to the server NICs at the back. If the Ethernet switch faced the rear of the rack then shorter CAT5 cables could’ve been used. This would’ve cluttered the rack interior less, and would’ve allowed for more direct cabling runs. We’ll try this with the next rack we install.

Write it down

Often it happens that an interesting or important detail crosses my desk for which there is no file or folder, or some other ready place to note the knowledge. These snippets come as emails, faxes, documents, meeting notes, conference calls or from my own notebook. It is far too easy to set aside or lose pieces of paper, and the volume of email is so huge even Google struggles to find things. The black hole of email, especially Microsoft Outlook email, is worthy of its very own blog. The flow of paper is a continual reminder how important it is to save the data somewhere and worry about the taxonomy later.

For each piece of equipment I manage I’ve created a separate text document or spreadsheet. In it I record the interesting and essential details for managing that equipment, ways to reduce event impacts, tips and tricks for getting reports and generally making the equipment do more of what I want it to do. I’ve heard this documentation called “paper brains.”

This unstructured input can become large and unwieldy. Eventually some of it may begin to lend itself to a classification scheme. If so, I like to structure it so it can be more easily accessed when it is needed. Either way, having it is better than not having because my memory is so horribly unreliable.

When an equipment outage occurs I try to document the problem and note the details of the repair. You’ll see many of those appearing on this site. This is how I communicate with my colleagues and add to the “corporate memory” for that equipment. Same when troubleshooting an individual subscriber complaint. If the front line customer care folks cannot resolve an issue and it gets escalatated to me, I document the results and share them among my peer group. The document goes into the the online repository for others to access. It is my hope that if I were to get hit by a bus someone else could take up my job behind me without impacting the business too much.