Problem Scenarios

   a. Insert one card and ensure that it properly downloads. If it does not properly download and come back into service, then permanently remove it (this could be the culprit card). Proceed to the next circuit card.

---

Note   If a card properly downloads and comes back into service, then wait two minutes after it comes back active before proceeding to insert the next card.

---

   b. Repeat the process until the card that you insert causes the system to reboot. This will isolate the cause of the periodic rebooting to one of service circuit or network circuit cards.

Step 6   To restore the system redundant operation, hard-select the AAC card to AUTO by moving the selector switch on the AAC card to AUTO.

Booting off Generic Diskette #1, or Optional Diskette, and an Error Occurs

The output will likely be an error "NonBootable Floppy Exists in A: Drive Replace it to Boot from A:" .

The following message sequence means that Generic diskette #1, or an Optional diskette, does not have the "/boot" directory on the diskette (but does have the correct volume label):

Copyright Motorola Inc. 1988 - 1997 All Rights Reserved MVME147 Monitor/Debugger Release 2.44 - 8/7/97 CPU running at 33 MHz FPC passed test MMU passed test COLD Start Onboard RAM start = $00000000, stop = $00FFFFFF 147-Bug> Searching for ROM Boot 147-Bug>G FFA0002C Effective address: FFA0002C SDS Initializing... Hard disk mounted successfully Hard disk dismounted successfully Reading boot file from disk device A:/boot/boot.sds NonBootable Floppy Exists in A: Drive Replace it to Boot from A: Trying C drive Reading boot file from disk device C:/boot/boot.sds Loading file C:/boot/globals.exe Loading file C:/boot/syswd.exe Loading file C:/boot/hostmgr.exe Loading file C:/boot/redmgr.exe Loading file C:/boot/netmgr.exe Loading file C:/boot/permgr.exe Loading file C:/boot/snmp.exe SW version loaded - Ver.Rev FSR : 5.1 002 Hard disk mounted successfully Hard disk dismounted successfully Loading File < syscnfg.TBL > ... Loading File < card.TBL > ... Loading File < port.TBL > ... Loading File < hostcnfg.TBL > ... Loading File < resgroup.TBL > ... Loading File < supvtmpl.TBL > ... Loading File < iprule.TBL > ... Loading File < oprule.TBL > ... Loading File < isdnsupv.TBL > ... Loading File < isdnmsg.TBL > ... Loading File < routesum.TBL > ... Loading File < routetbl.TBL > ... Loading File < exroute.TBL > ... Loading File < nfascnfg.TBL > ... Loading File < promptlb.TBL > ... Loading File < subrate.TBL > ... Loading File < motomap.TBL > ... Loading File < license.TBL > ...

The following error message generally indicates that this problem has occurred:

HST013: Host < host-name > Socket Write Error $A23:Operation Would Block

The VCO has closed the VCO-Host socket connection because the VCO's TCP/IP transmit (tx) queue has completely filled up. There is no more free space in the VCO's TCP/IP transmit (tx) queue for the application residing on the VCO to write data to.

Here is an example VCO log file (A-MMMDD.LOG in C:/log/) foot print:

<snip> #DF Fri Jan 21, 2000 14:48:46 A-ACT HST013: Host < HOST2 > Socket Write Error $A23:Operation Would Block #DF Fri Jan 21, 2000 14:48:46 A-ACT HST002: Host < HOST2 > Has Failed Due To: Socket Write Error #DF Fri Jan 21, 2000 14:48:46 A-ACT FRM504: Major Alarm Set For - ALM011: No Hosts Available (SA) #DF Fri Jan 21, 2000 14:49:38 A-ACT HST012: Host < HOST1 > Login Attempt From-10.10.25.156 33322 #DF Fri Jan 21, 2000 14:49:38 A-ACT HST003: Host < HOST1 > Restored #DF Fri Jan 21, 2000 14:49:38 A-ACT FRM511: Major Alarm Clear For - ALM011: No Hosts Available <snip>

There are two causes of HST013.

Cause 1:

The Host's TCP/IP receive (rx) queue is filling up faster than the Host can empty it (read it). You can easily determine this by monitoring the VCO-Host socket connection with a LAN protocol analyzer (sniffer).

You will see in the protocol analyzer's trace that the Host is sending to the VCO a "window size = zero (0)" message. When the VCO reads this message, it then instructs itself to "store" the messages in its transmit (tx) buffer and not send them to the Host. If the application residing on the VCO is still continuing to write to its tx queue while the tx queue is in the "store" state, the tx queue will eventually completely fill up. When the tx queue completely fills up, the VCO closes the VCO-Host socket and issues "HST013: Host < host-name > Socket Write Error $A23:Operation Would Block" to the VCO log file.

Example LAN protocol analyzer trace:

Packet Number 3974 ETH: ====( 60 bytes transmitted on interface en3 )==== 12:15:30.991421683 ETH: [ 00:06:29:ac:e0:96 -> 08:00:3e:2b:49:73 ] type 800 (IP) IP: < SRC = 10.10.25.156 > IP: < DST = 10.10.25.19 > (sfBA4K) IP: ip_v=4, ip_hl=20, ip_tos=0, ip_len=40, ip_id=41667, ip_off=0 IP: ip_ttl=60, ip_sum=954a, ip_p = 6 (TCP) TCP: <source port=33502, destination port=2001(dc) > TCP: th_seq=de0adf53, th_ack=4fffce1e TCP: th_off=5, flags<ACK> TCP: th_win=0, th_sum=2e6, th_urp=0

In packet 3974 above, the Host, "10.10.25.156" (SRC = source), is sending to the VCO, "10.10.25.19" (DST = destination), a "window size = 0 bytes" message (th_win=0) implying that the Host is not reading the data out of its rx queue fast enough.

Packet Number 3981 ETH: ====( 64 bytes received on interface en3 )==== 12:15:31.863284297 ETH: [ 08:00:3e:2b:49:73 -> 00:06:29:ac:e0:96 ] type 800 (IP) IP: < SRC = 10.10.25.19 > (sfBA4K) IP: < DST = 10.10.25.156 > IP: ip_v=4, ip_hl=20, ip_tos=0, ip_len=40, ip_id=1963, ip_off=0 IP: ip_ttl=30, ip_sum=4e63, ip_p = 6 (TCP) TCP: <source port=2001(dc), destination port=33502 > TCP: th_seq=4fffce1e, th_ack=de0adf63 TCP: th_off=5, flags<ACK> TCP: th_win=4096, th_sum=f2d5, th_urp=0

In packet 3981 above, just 4 packets later, the VCO, "10.10.25.19" (SRC = source), is sending to the Host, "10.10.25.156" (DST = destination), a "window size = 4096 bytes" message (th_win=4096) implying that the VCO's tx queue is not full.

Further Isolation and Possible Resolution

To improve operation:

• Ensure that the LAN network segment is not being disrupted by excessive amounts of other traffic which could keep the Host from accessing the VCO. Remove the other traffic.
  
  
• Increase the Host's rx queue size.
  
  
• Look at the task priorities that you run on your Host. Verify that there are no tasks, used to communicate with the VCO, that are running at a priority lower than other tasks that provide similar functionality. Your Host runs at "0% CPU idle", the effected task would not get enough CPU time and would start to fall behind (when your Host CPU runs at less than 0% idle, this task would get enough time to run and you would see no problem). To address this issue, try raising the priority of the task in question.
  
  

Cause 2:

The application residing on the VCO is filling up its tx queue faster than it can empty it (transmit it). You will see nothing abnormal in the LAN protocol analyzer trace, and the HST013 message will be generated in the log file.

The application residing on the VCO (generic) writes its output directly to the tx queue, which is the low-level network device driver that resides on the "vrtx ethernet controller" on the CPU, and, in this case, it is filling up its tx queue faster than it can send the data out to the Host. The "vrtx ethernet controller" will eventually close the socket and not report an error on the wire.

Further Isolation and Possible Resolution

The VCO is trying to send report messages to the host as fast as it can in response to the rate at which the host is sending command messages to the VCO.

To improve operation:

• Try reducing the rate at which the host sends call processing commands (e.g., $49 commands) to the VCO.
  
  
• If you are using the $83, $82, or $90 commands, see if you can reduce their frequency of use.
  
  

---

Note   One $83, $82, or $90 command (each a "non-callprocessing" command) has the potential to generate up to 30 VCO acknowledgement reports.

---

• Evaluate the bit settings in each "non-callprocessing" command and determine (for example) if your host application actually needs to be informed of the status of every port in the system at this particular time.
  
  

The system allows you to move (using the Rename Files command) a log file that is "open" (i.e., named as today's date) to the root directory (C:/) or to another C-drive directory. But, the system will not automatically create a new today's log file in C:/log until midnight.

To restart today's log file without having to wait until midnight:

Step 1   After moving and renaming the log file, make a note of the time on the system and on your watch.

Step 2   Change the time on the system to 23:59:55, from the Clock/Calendar Configuration screen, and wait one minute. A new log file, "tomorrow's" log file, will be created in C:/log and "today's" log file will close.

Step 3   Change the time on the system back to the time you need. This will create a new "today's" log file in C:/log. Today's log file operation has now been restored.

Step 4   Delete "tomorrow's" log file from C:/log.

Step 1   Disable Host Message Trace and NBC Message Trace from the System Trace Configuration screen.

Step 2   Copy the file if you need it (we recommend using FTP).

Either delete the file, or move it (using the Rename Files command) to the root directory (C:/) or to another C-drive directory.

Step 3   Turn tracing on again (creating a new trace file) and let it run for 3 minutes; and then turn it off (you can do this during a slow traffic period).

---

Note   The system writes 1000 time stamps to the new trace file and then it samples the size of the trace file. Three minutes (even during a slow traffic period) should be sufficient time to generate 1000-plus writes. When the Generic detects that 1000 writes have been made, it will then sample the trace file. When it samples the trace file, it will see approximately 40 kBytes (which is less than 1 MB) and clear ALM096. Every 1000 writes generates approximately 40 kBytes of tracing.

---

Step 4   Verify that ALM096 clears in your Host, in the Log File, and in the System Alarms Display/Major screen.

Step 5   Delete this new trace file.

Step 1   Identify and then eliminate the root cause of the streaming log file messages through troubleshooting.

---

Note   Streaming log file messages are typically the result of a Service Circuit card or Network Interface card that is failing. In this case the corrective action would be to remove the defective card from the system.

---

Step 2   Copy the file if you need it (we recommend using FTP).

Step 3   After midnight, either delete the file, or, move it (using the Rename Files command) to the root directory (C:/) or to another C-drive directory.

---

Note   The system does not permit you to delete a log file that is open (i.e., named as today's date). The message "Error during Disk Operation" is returned. The system does allow you to move (rename) a log file that is open but the system will still not create a new log file until midnight (see CSCdm85746 on CCO). See the "VCO allows you to move to the root directory a log file with today's date, but will not automatically create a new log file until midnight" section. The system will automatically clear ALM097 when the following conditions are met: The system needs to write 1000 date/time stamps to the subsequent log file(s) before it samples the size of the new open log file. Approximately 107 kBytes of logging (29 printed pages of log file messages) will generate 1000-plus writes. When the Generic detects that 1000 writes have been made, it will then sample the log file and see less than 1 MB and clear ALM097.

---

---

Note   If the system then runs without error after the alarm incident, it will take the system approximately 50 days before generating 1000-plus log file writes (necessary to automatically clear ALM097).

---

You can also clear ALM097 within 2 hours.

You can clear ALM097 in approximately 1 hour and 45 minutes by running Test Service Circuits as specified below. This procedure will manually introduce log file messages. (we recommend performing this procedure during a slow traffic period.)

The Test Port Card procedure setup:

T E S T P O R T C A R D Port Card R,L,S 1 1 15-1-2 Start Channel 1_ End Channel 24 MRC/DRC R,L,S 1 1 11-1-3 Start Port 1_ End Port 32 Port Card Type ____ Receiver Type ____ (*)Print Error Messages Only (Y/N) N (*)Continually Loop Through Test (Y/N) Y

---

Note   The network span and the service circuit engine that you select for the Test Port Card procedure must first be placed in Diagnostic Mode using the Card Maintenance screen. Spans in the diagnostic state cannot be used for call processing. Note also the settings indicated by (*).

---

The Test Test Service Circuits procedure setup:

S E R V I C E C I R C U I T T E S T U T I L I T Y Receiver R,L,S 1 1 11-1-3__ Card Type DTMF Monitor R,L,S,P _ _ ______ ___ Start Test at Port 1 End Test at Port 32 (*)Print Error Messages Only (Y/N) N (*)Continually Loop Through Test (Y/N) Y

---

Note   Port numbers start at 1. The start port must be specified. If the end port is not specified, or if the end port is less than or equal to the start port, only the start port will be tested. The service circuit engine that you select for the Service Circuit Test Utility procedure must first be placed in Diagnostic Mode from the Card Maintenance screen. Spans in the diagnostic state cannot be used for call processing. Note also the settings indicated by (*).

---

Verify that ALM097 clears in your Host, in the Log File, and in the System Alarms Display/Major screen.

FRM340: CODE Error - ethernet_rx.c,393: SOCK == EOF

For details, see issue CSCdk80666 on CCO.

If you see "FRM 340 CODE Error - ethernet_rx.c,393: SOCK == EOF" in the log file, along with "HST013: Host <host-link-name-Z> ... $A30:Addr already in use", then the Ethernet socket connection between the Host and the VCO, called <host-link-name-Z>, has become permanently unusable.

The only way to make this socket reusable is to reboot that side of the VCO.

Note   This problem is intermittent. Your system may never experience this problem.

The best defensive strategy is to define, in the VCO Host Configuration screen, all eight sockets. If you use only two socket connections between the host and a given side of the VCO, still define all eight sockets.

When a socket becomes permanently unusable, reassign the host connection to one of the other sockets currently not in use.

When all sockets become permanently unusable on one side of a VCO, that side of the VCO must be rebooted.

During the sys-config, when the Name Service screen is reached, there are four options to select from. If the customer is not using any Name Service, then the obvious selection is None. But when that is selected and F2 is pressed, the following message is displayed:

switch file: config-nsswitch error -4, errno 2, No such file or directory.

Resolution:

When you run sys-config, the system expects a name service to be defined. When the Name Service screen option is set to NONE, the system still tries to look for a service type and becomes stuck in a loop. If the system is in this situation, do the following:

Step 1   The terminal you are using determines how you get to the system prompt.

• If you are using a UNIX/Solaris terminal, press the Stop key followed by the 'a' key.
  
  

This places you at an OK prompt.

• If you are using a WYSE terminal / PC, press the Cntrl (control) key followed by the Pause/Break key.
  
  

(Sometimes the Alt key followed by Pause/Break works.)

This will place you at an OK prompt.

Step 2   From the OK prompt, type

boot -s.

This brings up the system in single-user mode.

Step 3   The system will prompt with a message:

Press Ctrl-D to continue with normal startup or a root password for maintenance.

Step 4   Enter the standard root password followed by the Return/Enter key

Step 5   Once logged in as root, change directory to /etc and create the file nsswitch.conf and nsswitch.files as dummy files.

Basically you are creating files that the system was missing.

Files could easily be created using the vi editor:

vi /etc/nsswitch.conf (this creates the file when exit from edit mode by option wq)

vi /etc/nsswitch.files (this creates the file when exit from edit mode by option wq)

Step 6   Verify if the files now exist under the directory /etc.

Step 7   Change directory to /usr/sbin.

Step 8   Run the command sys-unconfig to revert to an unconfigured system.

Step 9   Type reboot to bring the system back in configuration mode (sys-config).

Step 10   Continue with the regular sys-config procedure. This time when Name Service option is set to NONE, the system will allow you to configure the rest to the fields without any problem.

Step 3   Obtain a copy of the CKTINT Log with Debug Flags 2, 5, 9, 11, 12, 21, 22 and 28 enabled which shows the COT Failures and a copy of the VCO Host and NBC Trace (if possible).