[ MSDN 文章收集 ] I/O Concepts : I/O Completion Ports

前言 : (原文) 
I/O completion ports 提供了在多處理器上一個有效處理多個非同步 I/O 請求的線程模式. 當處理器提供 I/O completion port, 則系統必須建立一個 associated queue 物件來服務請求(request). 而透過一個 Thread pool 的概念, 多處理器將能有效處理併發的 I/O requests 而不是在接收 request 當下才開始建立 Thread. 

How I/O Completion Ports Work : 
在函式 CreateIoCompletionPort 建立了一個 I/O completion port 並將之與一個或多個 file handles 關聯. 當有非同步 I/O 操作發生在這些 file handles 上, 這些 I/O completion packet 都會以 FIFO (first-in-first-out) 順序出現在與之關聯的 I/O completion port 上. 而這樣的好處是可以將多個 file handles 的操作同步於同一個物件上 (即 I/O completion port). 當然好處不僅於此. 而這裡的 file handle 系統用來抽像代表一個 overlapped I/O 的終端而不是真的是一個檔案或是磁碟裝置. 舉例來說它可以是一個網路終端, TCP socket, named pipe, 或 mail slot. 任何系統物件支援 overlapped I/O 都可以適用於接下來的主題. 

當一個 file handle 與 completion port 相關聯就會進入 block 狀態, 直到有 packet 從 completion port 進出. 另外線程可以透過呼叫 GetQueuedCompletionStatus 函式來等待 completion packet 進入 I/O completion port queue 而取代了直接等待非同步的 I/O 結束. 而因此而進入 block 狀態的線程是以 LIFO (last-in-first-out) 順序進入 unblock. 因為 completion packet 從 I/O completion port queue 出來的順序是 FIFO, 意味著最新的線程可能會一直在處理較舊的 completion packet. 

當一個 completion packet 進行 I/O completion port, 系統會先確認共有多少線程於該 completion port 關聯並正在執行的狀態. 如果數目少於設定併發數目 (discussed in the next section), 則其中一個 waiting 的線程 (最近進入 wating 的那個) 將會被允許去處理該 completion packet. 當該線程處理完畢則會在呼叫 GetQueuedCompletionStatus 函式並處理 queue 下一個 completion packet. 

線程可以使用 PostQueuedCompletionStatus 函式來置放 completion packets 到 I/O completion port 的 queue 裡. 藉由這個功能, completion port 也可以用來接收同一個 process 其它線程的訊息. 因為 I/O completion port handle 與file handles 相關聯, 因此在你使用 CloseHandle 關閉 I/O completion port handle 前, 你必須將與之關聯的 file handles 釋放或關閉. 
Ps. 

An I/O completion port is associated with the process that created it and is not shareable between processes. However, a single handle is shareable between threads in the same process.

Threads and Concurrency : 
I/O completion port 一個重要的屬性必須要注意的就是 concurrency value (併發數). completion port 的 concurrency 值在你使用 CreateIoCompletionPort 函式建立 completion port 時在 NumberOfConcurrentThreads 參數設定. 這個值限定了多少執行線程可以與這個建立的 completion port 相關聯. 當執行線程的數目到達了設定的 concurrency 值, 接著未來任何線程打算與建立的 completion port 相關聯都會被 block, 直到執行的線程低於設定的 concurrency 直為止. 

而最有效率的情況就是當在 queue 裡的 completion packets 不會造成線程被 block (執行的線程到達 concurrency 的值) 並且completion packets 沒有等待的情形. 考慮在多處理器時而 concurrency 的值為一, 而當你設定的線程數目大於一時, 則除了正在處理 completion packets 的線程在執行外且 completion port 總是有 packets 在等待 (因為 thread pool is LIFO), 其他的線程將都因為呼叫 GetQueuedCompletionStatus 函數處於block 的狀態. 而上述情形將導致被 block 的線程們無法執行造成執行上效能的考量. 因此最好的情形就是將 concurrency 值設為與該台電腦上的 CPU 數目一樣, 這樣就不會有任何的線程會因為被 block 而增加了需要線程 context switch 的成本. 但事實上情形往往會更複雜, 因為你的線程可能因為程式的設計會被 Suspend 而使得執行的線程數小於設定的 concurrency 值而無法發會最好的效能, 因此往往你會建立起比 concurrency 值還要多的線程來讓任何時刻你都有 concurrency 數目的執行線程來處理 queue 中的 packets. 而底下是原文針對上述情況作的說明 : 

The system also allows a thread waiting in GetQueuedCompletionStatus to process a completion packet if another running thread associated with the same I/O completion port enters a wait state for other reasons, for example the SuspendThread function. When the thread in the wait state begins running again, there may be a brief period when the number of active threads exceeds the concurrency value. However, the system quickly reduces this number by not allowing any new active threads until the number of active threads falls below the concurrency value. This is one reason to have your application create more threads in its thread pool than the concurrency value. Thread pool management is beyond the scope of this topic, but a good rule of thumb is to have a minimum of twice as many threads in the thread pool as there are processors on the system. For additional information about thread pooling, see Thread Pools.

範例代碼 : 
底下分別是使用 overlapped I/O completion port 的 Server/Client 範例代碼 : 

- Server side :

view plaincopy to clipboardprint?

1. #pragma comment(lib, "WS2_32")  // Link to mswsock.lib  
2. #pragma comment(lib, "Mswsock")  
3. #include   
4. #include   
5. #include   
6.   
7. #define PORT 23232  
8. #define DATA_BUFSIZE 8192  
9. enum SA{Send,Recv};  
10.   
11. typedef struct _SOCKET_INFORMATION {  
12.    CHAR Buffer[DATA_BUFSIZE];  
13.    WSABUF DataBuf;  
14.    SOCKET Socket;  
15.    WSAOVERLAPPED Overlapped;  
16.    DWORD BytesSEND;  
17.    DWORD BytesRECV;  
18.    SA           Action;  
19. } SOCKET_INFORMATION, * LPSOCKET_INFORMATION;  
20.   
21. DWORD WINAPI ProcessRecvCP(LPVOID completionPortID); // Receiving IO Process in Completion Port mode  
22.   
23. DWORD EventTotal = 0;  
24. WSAEVENT EventArray[WSA_MAXIMUM_WAIT_EVENTS];  
25. LPSOCKET_INFORMATION SocketArray[WSA_MAXIMUM_WAIT_EVENTS];  
26. CRITICAL_SECTION CriticalSection;  
27.   
28. int main()  // main_server_cp  
29. {  
30.     HANDLE CompletionPort;   
31.     WSADATA wsaData;  
32.     SOCKET ServerSocket, AcceptSocket;  
33.     SOCKADDR_IN InternetAddr;  
34.     DWORD Flags;  
35.     DWORD ThreadId;  
36.     DWORD RecvBytes;  
37.     LPSOCKET_INFORMATION SI;  
38.   
39.     InitializeCriticalSection(&CriticalSection);  
40.   
41.     CompletionPort = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 0);    
42.     if(CompletionPort == NULL)  
43.     {  
44.         printf("\t[ServerCP] Failed to Create CompletionPort with error %d!\n", GetLastError());  
45.         return 1;  
46.     }  
47.     if (WSAStartup((2,2),&wsaData) != 0)  
48.     {  
49.         printf("\t[ServerCP] WSAStartup() failed with error %d\n", WSAGetLastError());  
50.         WSACleanup();  
51.         return 1;  
52.     }  
53.     else  
54.         printf("\t[ServerCP] WSAStartup() looks nice!\n");  
55.   
56.     if ((ServerSocket = WSASocket(AF_INET, SOCK_STREAM, 0, NULL, 0, WSA_FLAG_OVERLAPPED)) == INVALID_SOCKET)  
57.     {  
58.         printf("\t[ServerCP] Failed to get a socket %d\n", WSAGetLastError());  
59.         return 1;  
60.     }  
61.     else  
62.         printf("\t[ServerCP] WSASocket() is OK lol!\n");  
63.   
64.     InternetAddr.sin_family = AF_INET;  
65.     InternetAddr.sin_addr.s_addr = htonl(INADDR_ANY);  
66.     InternetAddr.sin_port = htons(PORT);  
67.   
68.     if (bind(ServerSocket, (PSOCKADDR) &InternetAddr, sizeof(InternetAddr)) == SOCKET_ERROR)  
69.     {  
70.         printf("\t[ServerCP] bind() failed with error %d\n", WSAGetLastError());  
71.         return 1;  
72.     }  
73.     else  
74.         printf("\t[ServerCP] You see, bind() is working!\n");  
75.   
76.     if (listen(ServerSocket, 5))  
77.     {  
78.         printf("\t[ServerCP] listen() failed with error %d\n", WSAGetLastError());  
79.         return 1;  
80.     }  
81.     else  
82.         printf("\t[ServerCP] listen() is OK maa...\n");  
83.   
84.     // Setup the listening socket for connections  
85.    if ((AcceptSocket = WSASocket(AF_INET, SOCK_STREAM, 0, NULL, 0, WSA_FLAG_OVERLAPPED)) == INVALID_SOCKET)  
86.    {  
87.       printf("\t[ServerCP] Failed to get a socket %d\n", WSAGetLastError());  
88.       return 1;  
89.    }  
90.    else  
91.       printf("\t[ServerCP] WSASocket() looks OK!\n");     
92.   
93.    // Create a thread to service overlapped requests  
94.     if (CreateThread(NULL, 0, ProcessRecvCP, CompletionPort, 0, &ThreadId) == NULL)  
95.     {  
96.         printf("\t[ServerCP] CreateThread() failed with error %d\n", GetLastError());  
97.         return 1;  
98.     }  
99.     else  
100.         printf("\t[ServerCP] Nothing to say, CreateThread() is OK!\n");  
101.   
102.     EventTotal = 1;  
103.    while(TRUE)  
104.    {  
105.        // Accept inbound connections  
106.       if ((AcceptSocket = accept(ServerSocket, NULL, NULL)) == INVALID_SOCKET)  
107.       {  
108.           printf("\t[ServerCP] accept() failed with error %d\n", WSAGetLastError());  
109.           return 1;  
110.       }  
111.       else  
112.           printf("\t[ServerCP] accept() is OK!\n");  
113.   
114.       EnterCriticalSection(&CriticalSection);  
115.       // Create a socket information structure to associate with the accepted socket  
116.       if ((SocketArray[EventTotal] = (LPSOCKET_INFORMATION) GlobalAlloc(GPTR, sizeof(SOCKET_INFORMATION))) == NULL)  
117.       {  
118.          printf("\t[ServerCP] GlobalAlloc() failed with error %d\n", GetLastError());  
119.          return 1;  
120.       }  
121.       else  
122.          printf("\t[ServerCP] GlobalAlloc() for LPSOCKET_INFORMATION is pretty fine!\n");  
123.   
124.       // Fill in the details of our accepted socket  
125.       SI = SocketArray[EventTotal];  
126.       SI->Socket = AcceptSocket;  
127.       ZeroMemory(&(SI->Overlapped), sizeof(OVERLAPPED));  
128.       SI->BytesSEND = 0;  
129.       SI->BytesRECV = 0;  
130.       SI->DataBuf.len = DATA_BUFSIZE;  
131.       SI->DataBuf.buf = SocketArray[EventTotal]->Buffer;  
132.       SI->Action = Recv;  
133.   
134.      CreateIoCompletionPort(    (HANDLE) AcceptSocket,    
135.                                                         CompletionPort,   
136.                                                         (DWORD) SI,   
137.                                                         0);  
138.   
139.       // Post a WSARecv() request to begin receiving data on the socket  
140.       Flags = 0;  
141.       if (WSARecv(  SI->Socket,  
142.                                     &(SI->DataBuf),   
143.                                     1,   
144.                                     &RecvBytes,   
145.                                     &Flags,   
146.                                     &(SI->Overlapped), NULL) == SOCKET_ERROR)  
147.       {  
148.          if (WSAGetLastError() != ERROR_IO_PENDING)  
149.          {  
150.             printf("\t[ServerCP] WSARecv() failed with error %d\n", WSAGetLastError());  
151.             return 1;  
152.          }  
153.       }  
154.       else  
155.             printf("\t[ServerCP] WSARecv() should be working!\n");  
156.       LeaveCriticalSection(&CriticalSection);       
157.    }  
158. }  
159. DWORD WINAPI ProcessRecvCP(LPVOID completionPortID)  
160. {  
161.     HANDLE CompletionPort = (HANDLE) completionPortID;  
162.     DWORD BytesTransferred;   
163.     LPOVERLAPPED Overlapped;    
164.     LPSOCKET_INFORMATION PerHandleData;    
165.     DWORD SendBytes=0, RecvBytes=0;    
166.     DWORD Flags;   
167.     while(true)  
168.     {  
169.         BOOL ret = GetQueuedCompletionStatus(   CompletionPort,    
170.                                                                                             &BytesTransferred,  
171.                                                                                             (LPDWORD)&PerHandleData,    
172.                                                                                             (LPOVERLAPPED *) &Overlapped,   
173.                                                                                             INFINITE);  
174.         if (BytesTransferred == 0 || ret==FALSE || PerHandleData->Action != Recv)  
175.         {  
176.             SendBytes = RecvBytes = 0;  
177.             printf("\t[ProcessRecvCP] Closing socket %d\n", PerHandleData->Socket);  
178.             shutdown(PerHandleData->Socket, SD_SEND);  
179.             if (closesocket(PerHandleData->Socket) == SOCKET_ERROR)  
180.             {  
181.                 printf("\t[ProcessRecvCP] closesocket() failed with error %d\n", WSAGetLastError());  
182.             }  
183.             else  
184.                 printf("\t[ProcessRecvCP] closesocket() is OK!\n");  
185.   
186.             GlobalFree(PerHandleData);        
187.             continue;  
188.         }  
189.   
190.         if(PerHandleData->Action = Recv)  
191.         {  
192.             printf("====================================\n");  
193.             for(int i=0; i
194.             {  
195.                 printf("%c", PerHandleData->Buffer[RecvBytes+i]);  
196.             }  
197.             printf("\n====================================\n");  
198.             RecvBytes+=BytesTransferred;  
199.             printf("\t[ProcessRecvCP] Receiving data in transferred bytes=%d, total=%d...\n", BytesTransferred, RecvBytes);  
200.             ZeroMemory(&(PerHandleData->Overlapped), sizeof(WSAOVERLAPPED));  
201.             if (WSARecv(    PerHandleData->Socket,   
202.                                         &(PerHandleData->DataBuf),   
203.                                         1,   
204.                                         &RecvBytes,   
205.                                         &Flags,   
206.                                         &(PerHandleData->Overlapped),   
207.                                         NULL) == SOCKET_ERROR)  
208.             {  
209.                 if (WSAGetLastError() == WSAEOPNOTSUPP)  
210.                 {  
211.                     SendBytes = RecvBytes = 0;  
212.                     printf("\t[ProcessRecvCP] Client shutdown SD_SEND.\n");  
213.                     printf("\t[ProcessRecvCP] Closing socket %d\n", PerHandleData->Socket);  
214.                     shutdown(PerHandleData->Socket, SD_SEND);  
215.                     if (closesocket(PerHandleData->Socket) == SOCKET_ERROR)  
216.                     {  
217.                         printf("\t[ProcessRecvCP] closesocket() failed with error %d\n", WSAGetLastError());  
218.                     }  
219.                     else  
220.                         printf("\t[ProcessRecvCP] closesocket() is OK!\n");  
221.   
222.                     GlobalFree(PerHandleData);  
223.                     continue;  
224.                 }  
225.                 else if(WSAGetLastError() != ERROR_IO_PENDING)  
226.                 {  
227.                     SendBytes = RecvBytes = 0;  
228.                     printf("\t[ProcessRecvCP] WSARecv() failed with error %d\n", WSAGetLastError());  
229.                     printf("\t[ProcessRecvCP] Closing socket %d\n", PerHandleData->Socket);  
230.                     shutdown(PerHandleData->Socket, SD_SEND);  
231.                     if (closesocket(PerHandleData->Socket) == SOCKET_ERROR)  
232.                     {  
233.                         printf("\t[ProcessRecvCP] closesocket() failed with error %d\n", WSAGetLastError());  
234.                     }  
235.                     else  
236.                         printf("\t[ProcessRecvCP] closesocket() is OK!\n");  
237.   
238.                     GlobalFree(PerHandleData);  
239.                     return 1;  
240.                 }  
241.             }  
242.             else  
243.                 printf("\t[ProcessRecvCP] WSARecv() is OK!\n");  
244.         }         
245.     }  
246.     return 0;  
247. }  

- Client Side :

view plaincopy to clipboardprint?

1. #pragma comment(lib, "WS2_32")  // Link to mswsock.lib  
2. #pragma comment(lib, "Mswsock")  
3. #include   
4. #include   
5. #include   
6.   
7. #define PORT 23232  
8. #define DATA_BUFSIZE 8192  
9. enum SA{Send,Recv};  
10.   
11. typedef struct _SOCKET_INFORMATION {  
12.    CHAR Buffer[DATA_BUFSIZE];  
13.    WSABUF DataBuf;  
14.    SOCKET Socket;  
15.    WSAOVERLAPPED Overlapped;  
16.    DWORD BytesSEND;  
17.    DWORD BytesRECV;  
18.    SA           Action;  
19. } SOCKET_INFORMATION, * LPSOCKET_INFORMATION;  
20.   
21. DWORD WINAPI ProcessSendCP(LPVOID lpParameter);  // Sending IO Process in Event mode  
22.   
23. DWORD EventTotal = 0;  
24. WSAEVENT EventArray[WSA_MAXIMUM_WAIT_EVENTS];  
25. LPSOCKET_INFORMATION SocketArray[WSA_MAXIMUM_WAIT_EVENTS];  
26. CRITICAL_SECTION CriticalSection;  
27.   
28. int main()  // Client in completion port mode  
29. {  
30.     HANDLE CompletionPort;   
31.     WSADATA wsaData;  
32.     SOCKET ClientSocket, AcceptSocket;  
33.     SOCKADDR_IN InternetAddr;  
34.     DWORD Flags;  
35.     DWORD ThreadId;  
36.     DWORD RecvBytes;  
37.     LPSOCKET_INFORMATION SI;  
38.   
39.     InitializeCriticalSection(&CriticalSection);  
40.   
41.     CompletionPort = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 0);    
42.     if(CompletionPort == NULL)  
43.     {  
44.         printf("\t[ClientCP] Failed to Create CompletionPort with error %d!\n", GetLastError());  
45.         return 1;  
46.     }  
47.     if (WSAStartup((2,2),&wsaData) != 0)  
48.     {  
49.         printf("\t[ClientCP] WSAStartup() failed with error %d\n", WSAGetLastError());  
50.         WSACleanup();  
51.         return 1;  
52.     }  
53.     else  
54.         printf("\t[ClientCP] WSAStartup() looks nice!\n");  
55.   
56.     if ((ClientSocket = WSASocket(AF_INET, SOCK_STREAM, 0, NULL, 0, WSA_FLAG_OVERLAPPED)) == INVALID_SOCKET)  
57.     {  
58.         printf("\t[ClientCP] Failed to get a socket %d\n", WSAGetLastError());  
59.         return 1;  
60.     }  
61.     else  
62.         printf("\t[ClientCP] WSASocket() is OK lol!\n");  
63.   
64.     InternetAddr.sin_family = AF_INET;  
65.     InternetAddr.sin_addr.s_addr = inet_addr("127.0.0.1");  
66.     InternetAddr.sin_port = htons(PORT);  
67.   
68.     if(connect(ClientSocket, (PSOCKADDR)&InternetAddr, sizeof(InternetAddr)) == SOCKET_ERROR)  
69.     {  
70.         printf("\t[ClientCP] Error : Connect failed (%d)!\n", WSAGetLastError());  
71.         return 1;  
72.     }  
73.     else printf("\t[ClientCP] Connect ok!\n");  
74.     // Associate the accepted socket with the completion port  
75.     EventTotal = 1;  
76.     // Create a socket information structure to associate with the send socket  
77.     if ((SocketArray[EventTotal] = (LPSOCKET_INFORMATION) GlobalAlloc(GPTR, sizeof(SOCKET_INFORMATION))) == NULL)  
78.     {  
79.         printf("\t[ClientCP] GlobalAlloc() failed with error %d\n", GetLastError());  
80.         return 1;  
81.     }  
82.     else  
83.         printf("\t[ClientCP] GlobalAlloc() for LPSOCKET_INFORMATION is pretty fine!\n");  
84.   
85.     SI = SocketArray[EventTotal];  
86.     SI->Socket = ClientSocket;  
87.     ZeroMemory(&(SI->Overlapped), sizeof(OVERLAPPED));  
88.     SI->BytesSEND = 0;  
89.     SI->BytesRECV = 0;  
90.     SI->DataBuf.len = DATA_BUFSIZE;  
91.     SI->DataBuf.buf = SocketArray[EventTotal]->Buffer;  
92.     SI->Action = Send;  
93.     char* sendingData = "this is a long test string\nabc\n";  
94.     strcpy( SI->DataBuf.buf, sendingData);  
95.     SI->DataBuf.len = strlen(sendingData);  
96.     SI->BytesSEND = strlen(sendingData);  
97.     CreateIoCompletionPort( (HANDLE) ClientSocket,    
98.                                                        CompletionPort,   
99.                                                        (DWORD) SI,   
100.                                                        0);  
101.   
102.     // Create a thread to service overlapped requests  
103.     if (CreateThread(NULL, 0, ProcessSendCP, CompletionPort, 0, &ThreadId) == NULL)  
104.     {  
105.         printf("\t[ClientCP] CreateThread() failed with error %d\n", GetLastError());  
106.         return 1;  
107.     }  
108.     else  
109.         printf("\t[ClientCP] Nothing to say, CreateThread() is OK!\n");  
110.   
111.     // WSASend  
112.     DWORD SendBytes;      
113.     if (WSASend(SI->Socket, &(SI->DataBuf), 1, &SendBytes, 0, &(SI->Overlapped), NULL) == SOCKET_ERROR)  
114.     {  
115.         if (WSAGetLastError() != ERROR_IO_PENDING)  
116.         {  
117.             printf("\t[ClientCP] WSASend() failed with error %d\n", WSAGetLastError());  
118.             return 0;  
119.         }  
120.     }  
121.     else   
122.     {  
123.         printf("\t[ClientCP] WSASend() is OK!\n");  
124.         if(SendBytes==SI->BytesSEND)   
125.         {  
126.             printf("\t[ClientCP] WSASend() is completed!\n");  
127.             shutdown(SI->Socket, SD_SEND);  
128.             closesocket(SI->Socket);  
129.             GlobalFree(SI);  
130.             WSACloseEvent(EventArray[EventTotal]);  
131.         }  
132.         else  
133.         {  
134.             // Wait for event notification  
135.         }  
136.     }  
137.     system("pause");  
138. }  
139. DWORD WINAPI ProcessSendCP(LPVOID completionPortID)  
140. {  
141.     HANDLE CompletionPort = (HANDLE) completionPortID;  
142.     DWORD BytesTransferred;   
143.     LPOVERLAPPED Overlapped;    
144.     LPSOCKET_INFORMATION PerHandleData;    
145.     DWORD SendBytes=0, RecvBytes=0;    
146.     DWORD Flags;    
147.   
148.     while(true)  
149.     {  
150.         BOOL ret = GetQueuedCompletionStatus(   CompletionPort,    
151.             &BytesTransferred,  
152.             (LPDWORD)&PerHandleData,    
153.             (LPOVERLAPPED *) &Overlapped,   
154.             INFINITE);  
155.         if (BytesTransferred == 0 || ret==FALSE)  
156.         {  
157.             SendBytes = RecvBytes = 0;  
158.             printf("\t[ProcessSendCP] Closing socket %d\n", PerHandleData->Socket);  
159.             shutdown(PerHandleData->Socket, SD_SEND);  
160.             if (closesocket(PerHandleData->Socket) == SOCKET_ERROR)  
161.             {  
162.                 printf("\t[ProcessSendCP] closesocket() failed with error %d\n", WSAGetLastError());  
163.             }  
164.             else  
165.                 printf("\t[ProcessSendCP] closesocket() is OK!\n");  
166.   
167.             GlobalFree(PerHandleData);        
168.             continue;  
169.         }  
170.   
171.         if(PerHandleData->Action == Send)  
172.         {  
173.             SendBytes += BytesTransferred;  
174.             printf("\t[ProcessSend] Sending data in transferred bytes=%d/%d...\n", BytesTransferred, SendBytes);  
175.             if(PerHandleData->BytesSEND>SendBytes)   
176.             {  
177.                 PerHandleData->DataBuf.buf = PerHandleData->Buffer + SendBytes;  
178.                 PerHandleData->DataBuf.len = PerHandleData->BytesSEND - SendBytes;  
179.                 ZeroMemory(&(PerHandleData->Overlapped), sizeof(WSAOVERLAPPED));  
180.                 if (WSASend(PerHandleData->Socket, &(PerHandleData->DataBuf), 1, &SendBytes, 0, &(PerHandleData->Overlapped), NULL) == SOCKET_ERROR)  
181.                 {  
182.                     if (WSAGetLastError() != ERROR_IO_PENDING)  
183.                     {  
184.                         printf("\t[ProcessSend] WSASend() failed with error %d\n", WSAGetLastError());  
185.                         shutdown(PerHandleData->Socket, SD_SEND);  
186.                         if (closesocket(PerHandleData->Socket) == SOCKET_ERROR)  
187.                         {  
188.                             printf("\t[ProcessSendCP] closesocket() failed with error %d\n", WSAGetLastError());  
189.                         }  
190.                         else  
191.                             printf("\t[ProcessSendCP] closesocket() is OK!\n");  
192.   
193.                         GlobalFree(PerHandleData);  
194.                         return 0;  
195.                     }  
196.                 }  
197.                 else  
198.                     printf("\t[ProcessSend] WSASend() is OK!\n");  
199.             }  
200.             else  
201.             {  
202.                 printf("\t[ProcessSend] Sending Action is completed!\n");  
203.                 return 0;  
204.             }  
205.         }  
206.         else  
207.         {  
208.             printf("\t[ProcessSend] Unknow Action!\n");  
209.             return 0;  
210.         }  
211.     }  
212. }  

補充說明 : 
* Winsock I/O Methods (Socket I/O Models 之三 - Overlapped Model)