[ NP in MS ] Winsock I/O Methods (Part2 : Socket I/O Models 之三 - Overlapped Model)

Preface : 
The overlapped I/O model in Winsock offers applications better system performance than any of the I/O models explained so far. The overlapped model's basic design allows your application to post one or more asynchronous I/O requests at a time using an overlapped data structure. At a later point, the application can service the submitted requests after they have completed. This model is available on all Windows platforms except Windows CE. The model's overall design is based on the Windows overlapped I/O mechanisms available for performing I/O operations on devices using the ReadFile and WriteFilefunctions. 
Originally, the Winsock overlapped I/O model was available only to Winsock 1.1 applications running on Windows NT. Applications could take advantage of the model by calling ReadFile and WriteFile on a socket handle and specifying an overlapped structure. Since the release of Winsock 2, overlapped I/O has been incorporated into new Winsock functions, such as WSASend and WSARecv. As a result, the overlapped I/O model is now available on all Windows platforms that feature Winsock 2. 
Ps. 

With the release of Winsock 2, overlapped I/O can still be used with the functions ReadFile and WriteFile under Windows NT and Windows 2000. However, this functionality was not added to Windows 95, Windows 98, and Windows Me. For compatibility across platforms, you should always consider using the WSARecv and WSASend functions instead of the Windows ReadFile and WriteFile functions. This section will only describe how to use overlapped I/O through the Winsock 2 functions.

To use the overlapped I/O model on a socket, you must first create a socket that has the overlapped flag set. See Chapter 2 for more information on creating overlapped enabled sockets. After you successfully create a socket and bind it to a local interface, overlapped I/O operations can commence by calling the Winsock functions listed below and specifying an optional WSAOVERLAPPED structure : 

* WSASend
* WSASendTo
* WSARecv
* WSARecvFrom
* WSAIoctl
* WSARecvMsg
* AcceptEx
* ConnectEx
* TransmitFile
* TransmitPackets
* DisconnectEx
* WSANSPIoctl

To use overlapped I/O, each function takes a WSAOVERLAPPED structure as a parameter. When these functions are called with a WSAOVERLAPPED structure, they complete immediately—regardless of the socket's mode (described at the beginning of this chapter). They rely on the WSAOVERLAPPED structure to manage the completion of an I/O request. There are essentially two methods for managing the completion of an overlapped I/O request : your application can wait for event object notification or it can process completed requests through completion routines. The first six functions in the list have another parameter in common : a WSAOVERLAPPED_COMPLETION_ROUTINE. This parameter is an optional pointer to a completion routine function that gets called when an overlapped request completes. We will explore the event notification method next. Later in this chapter, you will learn how to use optional completion routines instead of events to process completed overlapped requests. 
- Event Notification 
The event notification method of overlapped I/O requires associating Windows event objects with WSAOVERLAPPED structures. When I/O calls such asWSASend and WSARecv are made using a WSAOVERLAPPED structure, they return immediately. Typically, you will find that these I/O calls fail with the return value SOCKET_ERROR and that WSAGetLastError reports a WSA_IO_PENDING error status. This error status simply means that the I/O operation is in progress. At a later time, your application will need to determine when an overlapped I/O request completes by waiting on the event object associated with the WSAOVERLAPPED structure. The WSAOVERLAPPED structure provides the communication medium between the initiation of an overlapped I/O request and its subsequent completion, and is defined as : 

view plaincopy to clipboardprint?

1. typedef struct WSAOVERLAPPED  
2. {   
3.     DWORD    Internal;  
4.     DWORD    InternalHigh;  
5.     DWORD    Offset;  
6.     DWORD    OffsetHigh;  
7.     WSAEVENT hEvent;  
8. } WSAOVERLAPPED, FAR * LPWSAOVERLAPPED;  

The Internal, InternalHigh, Offset, and OffsetHigh fields are all used internally by the system and an application should not manipulate or directly use them. The hEvent field, on the other hand, allows an application to associate an event object handle with this operation. 
When an overlapped I/O request finally completes, your application is responsible for retrieving the overlapped results. In the event notification method, Winsock will change the event-signaling state of an event object that is associated with a WSAOVERLAPPED structure from non-signaled to signaled when an overlapped request finally completes. Because an event object is assigned to the WSAOVERLAPPED structure, you can easily determine when an overlapped I/O call completes by calling the WSAWaitForMultipleEvents function, which we also described in the WSAEventSelect I/O model. 
WSAWaitForMultipleEvents waits a specified amount of time for one or more event objects to become signaled. We can't stress this point enough: remember that WSAWaitForMultipleEvents is capable of waiting on only 64 event objects at a time. Once you determine which overlapped request has completed, you need to determine the success or failure of the overlapped call by calling WSAGetOverlappedResult, which is defined as : 

view plaincopy to clipboardprint?

1. BOOL WSAGetOverlappedResult(   
2.     SOCKET s,  
3.     LPWSAOVERLAPPED lpOverlapped,   
4.     LPDWORD lpcbTransfer,   
5.     BOOL fWait,   
6.     LPDWORD lpdwFlags  
7. );  

The s parameter identifies the socket that was specified when the overlapped operation was started. The lpOverlapped parameter is a pointer to theWSAOVERLAPPED structure that was specified when the overlapped operation was started. The lpcbTransfer parameter is a pointer to a DWORD variable that receives the number of bytes that were actually transferred by an overlapped send or receive operation. The fWait parameter determines whether the function should wait for a pending overlapped operation to complete. If fWait is TRUE, the function does not return until the operation has been completed. IffWait is FALSE and the operation is still pending, WSAGetOverlappedResult returns FALSE with the error WSA_IO_INCOMPLETE. Because in our case we waited on a signaled event for overlapped completion, this parameter has no effect. The final parameter, lpdwFlags, is a pointer to a DWORD that will receive resulting flags if the originating overlapped call was made with the WSARecv or the WSARecvFrom function. 
If the WSAGetOverlappedResult function succeeds, the return value is TRUE. This means that your overlapped operation has completed successfully and that the value pointed to by lpcbTransfer has been updated. If the return value is FALSE, one of the following statements is true : 

* The overlapped I/O operation is still pending (as we previously described).
* The overlapped operation completed, but with errors.
* The overlapped operation's completion status could not be determined because of errors in one or more of the parameters supplied toWSAGetOverlappedResult.

Upon failure, the value pointed to by lpcbTransfer will not be updated, and your application should call the WSAGetLastError function to determine the cause of the failure. The following sample of code demonstrates how to structure a simple server application that is capable of managing overlapped I/O on one socket using the event notification described above : 

view plaincopy to clipboardprint?

1. #define DATA_BUFSIZE                        4096  
2. void main(void)  
3. {  
4.     WSABUF DataBuf;  
5.     char buffer[DATA_BUFSIZE];  
6.     DWORD EventTotal = 0,  
7.           RecvBytes=0,  
8.           Flags=0;  
9.     WSAEVENT EventArray[WSA_MAXIMUM_WAIT_EVENTS];  
10.     WSAOVERLAPPED AcceptOverlapped;  
11.     SOCKET ListenSocket, AcceptSocket;  
12.   
13.     // Step 1:  
14.     //  Start Winsock and set up a listening socket  
15.     ...  
16.   
17.     // Step 2:  
18.     //  Accept an inbound connection  
19.     AcceptSocket = accept(ListenSocket, NULL, NULL);  
20.   
21.     // Step 3:  
22.     //  Set up an overlapped structure  
23.   
24.     EventArray[EventTotal] = WSACreateEvent();  
25.   
26.     ZeroMemory(&AcceptOverlapped,  
27.         sizeof(WSAOVERLAPPED));  
28.      AcceptOverlapped.hEvent = EventArray[EventTotal];  
29.   
30.     DataBuf.len = DATA_BUFSIZE;  
31.     DataBuf.buf = buffer;  
32.   
33.     EventTotal++;  
34.   
35.     // Step 4:  
36.     //  Post a WSARecv request to begin receiving data  
37.     //  on the socket  
38.   
39.     if (WSARecv(AcceptSocket, &DataBuf, 1, &RecvBytes,  
40.         &Flags, &AcceptOverlapped, NULL) == SOCKET_ERROR)  
41.     {  
42.         if (WSAGetLastError() != WSA_IO_PENDING)  
43.         {  
44.              // Error occurred  
45.         }  
46.     }  
47.   
48.     // Process overlapped receives on the socket  
49.   
50.     while(TRUE)  
51.     {  
52.      DWORD    Index;  
53.         // Step 5:  
54.         //  Wait for the overlapped I/O call to complete  
55.         Index = WSAWaitForMultipleEvents(EventTotal,  
56.             EventArray, FALSE, WSA_INFINITE, FALSE);  
57.   
58.         // Index should be 0 because we   
59.         // have only one event handle in EventArray  
60.   
61.         // Step 6:  
62.         //  Reset the signaled event  
63.         WSAResetEvent(  
64.             EventArray[Index - WSA_WAIT_EVENT_0]);  
65.   
66.         // Step 7:  
67.         //  Determine the status of the overlapped  
68.         //  request  
69.         WSAGetOverlappedResult(AcceptSocket,  
70.             &AcceptOverlapped, &BytesTransferred,  
71.             FALSE, &Flags);  
72.       
73.         // First check to see whether the peer has closed  
74.         // the connection, and if so, close the  
75.         // socket  
76.   
77.         if (BytesTransferred == 0)  
78.         {  
79.             printf("Closing socket %d\n", AcceptSocket);  
80.   
81.             closesocket(AcceptSocket);  
82.             WSACloseEvent(  
83.                 EventArray[Index - WSA_WAIT_EVENT_0]);  
84.             return;  
85.         }  
86.   
87.         // Do something with the received data   
88.         // DataBuf contains the received data  
89.         ...  
90.   
91.         // Step 8:  
92.         //  Post another WSARecv() request on the socket  
93.   
94.         Flags = 0;  
95.         ZeroMemory(&AcceptOverlapped,  
96.             sizeof(WSAOVERLAPPED));  
97.   
98.         AcceptOverlapped.hEvent = EventArray[Index -   
99.             WSA_WAIT_EVENT_0];  
100.   
101.         DataBuf.len = DATA_BUFSIZE;  
102.         DataBuf.buf = buffer;  
103.   
104.         if (WSARecv(AcceptSocket, &DataBuf, 1,  
105.             &RecvBytes, &Flags, &AcceptOverlapped,  
106.             NULL) == SOCKET_ERROR)  
107.         {  
108.             if (WSAGetLastError() != WSA_IO_PENDING)  
109.             {  
110.                 // Unexpected error  
111.             }  
112.         }  
113.     }  
114. }  

The application outlines the following programming steps : 

1. Create a socket and begin listening for a connection on a specified port.

2. Accept an inbound connection.

3. Create a WSAOVERLAPPED structure for the accepted socket and assign an event object handle to the structure. Also assign the event object handle to an event array to be used later by the WSAWaitForMultipleEvents function.

4. Post an asynchronous WSARecv request on the socket by specifying the WSAOVERLAPPED structure as a parameter.

5. Call WSAWaitForMultipleEvents using the event array and wait for the event associated with the overlapped call to become signaled.

6. Determine the return status of the overlapped call by using WSAGetOverlappedResult.

7. Reset the event object by using WSAResetEvent with the event array and process the completed overlapped request.

8. Post another overlapped WSARecv request on the socket.

9. Repeat steps 5–8.

This example can easily be expanded to handle more than one socket by moving the overlapped I/O processing portion of the code to a separate thread and allowing the main application thread to service additional connection requests. 
Ps. 

If a Winsock function is called in an overlapped fashion (either by specifying an event within the WSAOVERLAPPED structure or with a completion routine), the operation might complete immediately. For example, calling WSARecv when data has already been received and buffered causes WSARecvto return NO_ERROR. If any overlapped function fails with WSA_IO_PENDING or immediately succeeds, the completion event will always be signaled and the completion routine will be scheduled to run (if specified). For overlapped I/O with a completion port, this means that completion notification will be posted to the completion port for servicing.

- Completion Routines : 
For newcomers, the completion port model seems overwhelmingly complicated because extra work is required to add sockets to a completion port when compared to the initialization steps for the other I/O models. However, as you will see, these steps are not that complicated once you understand them. Also, the completion port model offers the best system performance possible when an application has to manage many sockets at once. Unfortunately, it's available only on Windows NT, Windows 2000, and Windows XP; however, the completion port model offers the best scalability of all the models discussed so far. This model is well suited to handling hundreds or thousands of sockets. 
Essentially, the completion port model requires you to create a Windows completion port object that will manage overlapped I/O requests using a specified number of threads to service the completed overlapped I/O requests. Note that a completion port is actually a Windows I/O construct that is capable of accepting more than just socket handles. However, this section will describe only how to take advantage of the completion port model by using socket handles. To begin using this model, you are required to create an I/O completion port object that will be used to manage multiple I/O requests for any number of socket handles. This is accomplished by calling the CreateIoCompletionPort function, which is defined as : 

view plaincopy to clipboardprint?

1. HANDLE CreateIoCompletionPort(  
2.     HANDLE FileHandle,  
3.     HANDLE ExistingCompletionPort,  
4.     DWORD CompletionKey,  
5.     DWORD NumberOfConcurrentThreads  
6. );  

Before examining the parameters in detail, be aware that this function is actually used for two distinct purposes : 

* To create a completion port object
* To associate a handle with a completion port

When you initially create a completion port object, the only parameter of interest is NumberOfConcurrentThreads; the first three parameters are not significant. The NumberOfConcurrentThreads parameter is special because it defines the number of threads that are allowed to execute concurrently on a completion port. Ideally, you want only one thread per processor to service the completion port to avoid thread context switching. The value 0 for this parameter tells the system to allow as many threads as there are processors in the system. The following code creates an I/O completion port : 

view plaincopy to clipboardprint?

1. CompletionPort = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 0);  

This will return a handle that is used to identify the completion port when a socket handle is assigned to it. 

Worker Threads and Completion Ports : 
After a completion port is successfully created, you can begin to associate socket handles with the object. Before associating sockets, though, you have to create one or more worker threads to service the completion port when socket I/O requests are posted to the completion port object. At this point, you might wonder how many threads should be created to service the completion port. This is actually one of the more complicated aspects of the completion port model because the number needed to service I/O requests depends on the overall design of your application. It's important to note the distinction between number of concurrent threads to specify when calling CreateIoCompletionPort versus the number of worker threads to create; they do not represent the same thing. We recommended previously that you should have the CreateIoCompletionPort function specify one thread per processor to avoid thread context switching. The NumberOfConcurrentThreads parameter of CreateIoCompletionPort explicitly tells the system to allow only n threads to operate at a time on the completion port. If you create more than n worker threads on the completion port, only n threads will be allowed to operate at a time. (Actually, the system might exceed this value for a short amount of time, but the system will quickly bring it down to the value you specify in CreateIoCompletionPort.) You might be wondering why you would create more worker threads than the number specified by the CreateIoCompletionPort call. As we mentioned previously, this depends on the overall design of your application. If one of your worker threads calls a function—such as Sleep or WaitForSingleObject—and becomes suspended, another thread will be allowed to operate in its place. In other words, you always want to have as many threads available for execution as the number of threads you allow to execute in the CreateIoCompletionPort call. Thus, if you expect your worker thread to ever become blocked, it is reasonable to create more worker threads than the value specified in CreateIoCompletionPort's NumberOfConcurrentThreads parameter. 
Once you have enough worker threads to service I/O requests on the completion port, you can begin to associate socket handles with the completion port. This requires calling the CreateIoCompletionPort function on an existing completion port and supplying the first three parameters—FileHandle,ExistingCompletionPort, and CompletionKey—with socket information. The FileHandle parameter represents a socket handle to associate with the completion port. The ExistingCompletionPort parameter identifies the completion port to which the socket handle is to be associated with. The CompletionKey parameter identifies per-handle data that you can associate with a particular socket handle. Applications are free to store any type of information associated with a socket by using this key. We call it per-handle data because it represents data associated with a socket handle. It is useful to store the socket handle using the key as a pointer to a data structure containing the socket handle and other socket-specific information. As we will see later in this chapter, the thread routines that service the completion port can retrieve socket-handle–specific information using this key. 
Let's begin to construct a basic application framework from what we've described so far. The following example demonstrates how to start developing an echo server application using the completion port model. In this code, we take the following preparation steps : 

1. Create a completion port. The fourth parameter is left as 0, specifying that only one worker thread per processor will be allowed to execute at a time on the completion port.

2. Determine how many processors exist on the system.

3. Create worker threads to service completed I/O requests on the completion port using processor information in step 2. In the case of this simple example, we create one worker thread per processor because we do not expect our threads to ever get in a suspended condition in which there would not be enough threads to execute for each processor. When the CreateThread function is called, you must supply a worker routine that the thread executes upon creation. We will discuss the worker thread's responsibilities later in this section.

4. Prepare a listening socket to listen for connections on port 5150.

5. Accept inbound connections using the accept function.

6. Create a data structure to represent per-handle data and save the accepted socket handle in the structure.

7. Associate the new socket handle returned from accept with the completion port by calling CreateIoCompletionPort. Pass the per-handle data structure to CreateIoCompletionPort via the completion key parameter.

8. Start processing I/O on the accepted connection. Essentially, you want to post one or more asynchronous WSARecv or WSASend requests on the new socket using the overlapped I/O mechanism. When these I/O requests complete, a worker thread services the I/O requests and continues processing future I/O requests, as we will see later in the worker routine specified in step 3.

9. Repeat steps 5–8 until server terminates.

Below is the sample code for above steps : 

view plaincopy to clipboardprint?

1. HANDLE CompletionPort;  
2. WSADATA wsd;  
3. SYSTEM_INFO SystemInfo;  
4. SOCKADDR_IN InternetAddr;  
5. SOCKET Listen;  
6. int i;  
7.   
8. typedef struct _PER_HANDLE_DATA   
9. {  
10.     SOCKET          Socket;  
11.         SOCKADDR_STORAGE  ClientAddr;  
12.     // Other information useful to be associated with the handle  
13. } PER_HANDLE_DATA, * LPPER_HANDLE_DATA;  
14.   
15. // Load Winsock  
16. StartWinsock(MAKEWORD(2,2), &wsd);  
17.   
18. // Step 1:  
19. // Create an I/O completion port  
20.   
21. CompletionPort = CreateIoCompletionPort(  
22.     INVALID_HANDLE_VALUE, NULL, 0, 0);  
23.   
24. // Step 2:  
25. // Determine how many processors are on the system  
26.   
27. GetSystemInfo(&SystemInfo);  
28.   
29. // Step 3:  
30. // Create worker threads based on the number of  
31. // processors available on the system. For this  
32. // simple case, we create one worker thread for each  
33. // processor.  
34.   
35. for(i = 0; i < SystemInfo.dwNumberOfProcessors; i++)  
36. {  
37.     HANDLE ThreadHandle;  
38.   
39.     // Create a server worker thread, and pass the  
40.     // completion port to the thread. NOTE: the  
41.     // ServerWorkerThread procedure is not defined  
42.     // in this listing.  
43.   
44.     ThreadHandle = CreateThread(NULL, 0,  
45.         ServerWorkerThread, CompletionPort,  
46.         0, NULL);  
47.   
48.     // Close the thread handle  
49.     CloseHandle(ThreadHandle);  
50. }  
51.   
52. // Step 4:  
53. // Create a listening socket  
54.   
55. Listen = WSASocket(AF_INET, SOCK_STREAM, 0, NULL, 0,  
56.     WSA_FLAG_OVERLAPPED);  
57.   
58. InternetAddr.sin_family = AF_INET;  
59. InternetAddr.sin_addr.s_addr = htonl(INADDR_ANY);  
60. InternetAddr.sin_port = htons(5150);  
61. bind(Listen, (PSOCKADDR) &InternetAddr,  
62.     sizeof(InternetAddr));  
63.   
64. // Prepare socket for listening  
65.   
66. listen(Listen, 5);  
67.   
68. while(TRUE)  
69. {  
70.     PER_HANDLE_DATA *PerHandleData=NULL;  
71.     SOCKADDR_IN saRemote;  
72.     SOCKET Accept;  
73.     int RemoteLen;  
74.     // Step 5:  
75.     // Accept connections and assign to the completion  
76.     // port  
77.   
78.     RemoteLen = sizeof(saRemote);  
79.     Accept = WSAAccept(Listen, (SOCKADDR *)&saRemote,   
80.     &RemoteLen);  
81.   
82.     // Step 6:  
83.     // Create per-handle data information structure to   
84.     // associate with the socket  
85.     PerHandleData = (LPPER_HANDLE_DATA)   
86.         GlobalAlloc(GPTR, sizeof(PER_HANDLE_DATA));  
87.   
88.     printf("Socket number %d connected\n", Accept);  
89.     PerHandleData->Socket = Accept;  
90.     memcpy(&PerHandleData->ClientAddr, &saRemote, RemoteLen);  
91.   
92.     // Step 7:  
93.     // Associate the accepted socket with the  
94.     // completion port  
95.   
96.     CreateIoCompletionPort((HANDLE) Accept,  
97.         CompletionPort, (DWORD) PerHandleData, 0);  
98.   
99.     // Step 8:  
100.     //  Start processing I/O on the accepted socket.  
101.     //  Post one or more WSASend() or WSARecv() calls  
102.     //  on the socket using overlapped I/O.  
103.     WSARecv(...);  
104. }  
105.   
106.     DWORD WINAPI ServerWorkerThread(LPVOID lpParam)  
107.     {  
108.     // The requirements for the worker thread will be   
109.     // discussed later.  
110.     return 0;  
111. }  

- Completion Ports and Overlapped I/O 
After associating a socket handle with a completion port, you can begin processing I/O requests by posting overlapped send and receive requests on the socket handle. You can now start to rely on the completion port for I/O completion notification. Basically, the completion port model takes advantage of the Windows overlapped I/O mechanism in which Winsock API calls such as WSASend and WSARecv return immediately when called. It is up to your application to retrieve the results of the calls at a later time through an OVERLAPPED structure. In the completion port model, this is accomplished by having one or more worker threads wait on the completion port using the GetQueuedCompletionStatus function, which is defined as : 

view plaincopy to clipboardprint?

1. BOOL GetQueuedCompletionStatus(  
2.     HANDLE CompletionPort,   
3.     LPDWORD lpNumberOfBytesTransferred,   
4.     PULONG_PTR lpCompletionKey,   
5.     LPOVERLAPPED * lpOverlapped,   
6.     DWORD dwMilliseconds  
7. );  

The CompletionPort parameter represents the completion port to wait on. The lpNumberOfBytesTransferred parameter receives the number of bytes transferred after a completed I/O operation, such as WSASend or WSARecv. The lpCompletionKey parameter returns per-handle data for the socket that was originally passed into the CreateIoCompletionPort function. As we already mentioned, we recommend saving the socket handle in this key. The lpOverlappedparameter receives the WSAOVERLAPPED structure of the completed I/O operation. This is actually an important parameter because it can be used to retrieveper I/O–operation data, which we will describe shortly. The final parameter, dwMilliseconds, specifies the number of milliseconds that the caller is willing to wait for a completion packet to appear on the completion port. If you specify INFINITE, the call waits forever. 

- Per-handle Data and Per-I/O Operation Data 
When a worker thread receives I/O completion notification from the GetQueuedCompletionStatus API call, the lpCompletionKey and lpOverlapped parameters contain socket information that can be used to continue processing I/O on a socket through the completion port. Two types of important socket data are available through these parameters: per-handle data and per-I/O operation data. 
The lpCompletionKey parameter contains what we call per-handle data because the data is related to a socket handle when a socket is first associated with the completion port. This is the data that is passed as the CompletionKey parameter of the CreateIoCompletionPort API call. As we noted earlier, your application can pass any type of socket information through this parameter. Typically, applications will store the socket handle related to the I/O request here. 
The lpOverlapped parameter contains an OVERLAPPED structure followed by what we call per-I/O operation data, which is anything that your worker thread will need to know when processing a completion packet (echo the data back, accept the connection, post another read, and so on). Per-I/O operation data is any number of bytes contained in a structure also containing an OVERLAPPED structure that you pass into a function that expects an OVERLAPPED structure. A simple way to make this work is to define a structure and place an OVERLAPPED structure as a field of the new structure. For example, we declare the following data structure to manage per-I/O operation data : 

view plaincopy to clipboardprint?

1. typedef struct  
2. {  
3.     OVERLAPPED Overlapped;  
4.     char       Buffer[DATA_BUFSIZE];  
5.     int            BufferLen;  
6.     int        OperationType;  
7. } PER_IO_DATA;  

This structure demonstrates some important data elements you might want to relate to an I/O operation, such as the type of I/O operation (a send or receive request) that just completed. In this structure, we consider the data buffer for the completed I/O operation to be useful. To call a Winsock API function that expects an OVERLAPPED structure, you dereference the OVERLAPPED element of your structure. For example : 

view plaincopy to clipboardprint?

1. PER_IO_OPERATION_DATA PerIoData;  
2. WSABUF wbuf;  
3. DWORD Bytes, Flags;  
4. // Initialize wbuf ...  
5. WSARecv(socket, &wbuf, 1, &Bytes, &Flags, &(PerIoData.Overlapped),   
6. NULL);  

Later in the worker thread, GetQueuedCompletionStatus returns with an overlapped structure and completion key. To retrieve the per-I/O data the macro CONTAINING_RECORD should be used. For example : 

view plaincopy to clipboardprint?

1. PER_IO_DATA  *PerIoData=NULL;  
2. OVERLAPPED   *lpOverlapped=NULL;  
3.   
4. ret = GetQueuedCompletionStatus(  
5.          CompPortHandle,  
6.          &Transferred,  
7.          (PULONG_PTR)&CompletionKey,  
8.          &lpOverlapped,  
9.             INFINITE);  
10. // Check for successful return  
11. PerIoData = CONTAINING_RECORD(lpOverlapped, PER_IO_DATA, Overlapped);  

This macro should be used; otherwise, the OVERLAPPED member of the PER_IO_DATA structure would always have to appear first, which can be a dangerous assumption to make (especially with multiple developers working on the same code). 
You can determine which operation was posted on this handle by using a field of the per-I/O structure to indicate the type of operation posted. In our example, the OperationType member would be set to indicate a read, write, etc., operation. One of the biggest benefits of per-I/O operation data is that it allows you to manage multiple I/O operations (such as read/write, multiple reads, and multiple writes) on the same handle. You might ask why you would want to post more than one I/O operation at a time on a socket. The answer is scalability. For example, if you have a multiple-processor machine with a worker thread using each processor, you could potentially have several processors sending and receiving data on a socket at the same time. 
Before continuing, there is one other important aspect about Windows completion ports that needs to be stressed. All overlapped operations are guaranteed to be executed in the order that the application issued them. However, the completion notifications returned from a completion port are not guaranteed to be in that same order. That is, if an application posts two overlapped WSARecv operations, one with a 10 KB buffer and the next with a 12 KB buffer, the 10 KB buffer is filled first, followed by the 12 KB buffer. The application's worker thread may receive notification from GetQueuedCompletionStatusfor the 12 KB WSARecv before the completion event for the 10 KB operation. Of course, this is only an issue when multiple operations are posted on a socket.
To complete this simple echo server sample, we need to supply a ServerWorkerThread function. The following code outlines how to develop a worker thread routine that uses per-handle data and per-I/O operation data to service I/O requests : 

view plaincopy to clipboardprint?

1. DWORD WINAPI ServerWorkerThread(  
2.     LPVOID CompletionPortID)  
3. {  
4.     HANDLE CompletionPort = (HANDLE) CompletionPortID;  
5.     DWORD BytesTransferred;  
6.     LPOVERLAPPED Overlapped;  
7.     LPPER_HANDLE_DATA PerHandleData;  
8.     LPPER_IO_DATA PerIoData;  
9.     DWORD SendBytes, RecvBytes;  
10.     DWORD Flags;  
11.       
12.     while(TRUE)  
13.     {  
14.         // Wait for I/O to complete on any socket  
15.         // associated with the completion port  
16.       
17.         ret = GetQueuedCompletionStatus(CompletionPort,  
18.             &BytesTransferred,(LPDWORD)&PerHandleData,  
19.             (LPOVERLAPPED *) &PerIoData, INFINITE);  
20.   
21.         // First check to see if an error has occurred  
22.         // on the socket; if so, close the   
23.         // socket and clean up the per-handle data  
24.         // and per-I/O operation data associated with  
25.         // the socket  
26.   
27.         if (BytesTransferred == 0 &&  
28.             (PerIoData->OperationType == RECV_POSTED ││  
29.              PerIoData->OperationType == SEND_POSTED))  
30.         {  
31.             // A zero BytesTransferred indicates that the  
32.             // socket has been closed by the peer, so  
33.             // you should close the socket. Note:   
34.             // Per-handle data was used to reference the  
35.             // socket associated with the I/O operation.  
36.   
37.             closesocket(PerHandleData->Socket);  
38.   
39.             GlobalFree(PerHandleData);  
40.             GlobalFree(PerIoData);  
41.             continue;  
42.         }  
43.   
44.         // Service the completed I/O request. You can  
45.         // determine which I/O request has just  
46.         // completed by looking at the OperationType  
47.         // field contained in the per-I/O operation data.  
48.          if (PerIoData->OperationType == RECV_POSTED)  
49.         {  
50.             // Do something with the received data  
51.             // in PerIoData->Buffer  
52.         }  
53.   
54.         // Post another WSASend or WSARecv operation.  
55.         // As an example, we will post another WSARecv()  
56.         // I/O operation.  
57.   
58.         Flags = 0;  
59.   
60.         // Set up the per-I/O operation data for the next  
61.         // overlapped call  
62.         ZeroMemory(&(PerIoData->Overlapped),  
63.             sizeof(OVERLAPPED));  
64.   
65.         PerIoData->DataBuf.len = DATA_BUFSIZE;  
66.         PerIoData->DataBuf.buf = PerIoData->Buffer;  
67.         PerIoData->OperationType = RECV_POSTED;  
68.   
69.         WSARecv(PerHandleData->Socket,   
70.             &(PerIoData->DataBuf), 1, &RecvBytes,  
71.             &Flags, &(PerIoData->Overlapped), NULL);  
72.     }  
73. }  

If an error has occurred for a given overlapped operation, GetQueuedCompletionStatus will return FALSE. Because completion ports are a Windows I/O construct, if you call GetLastError or WSAGetLastError, the error code is likely to be a Windows error code and not a Winsock error code. To retrieve the equivalent Winsock error code, WSAGetOverlappedResult can be called specifying the socket handle and WSAOVERLAPPED structure for the completed operation, after which WSAGetLastError will return the translated Winsock error code. 
One final detail not outlined in the last two examples we have presented is how to properly close an I/O completion port—especially if you have one or more threads in progress performing I/O on several sockets. The main thing to avoid is freeing an OVERLAPPED structure when an overlapped I/O operation is in progress. The best way to prevent this is to call closesocket on every socket handle—any overlapped I/O operations pending will complete. Once all socket handles are closed, you need to terminate all worker threads on the completion port. This can be accomplished by sending a special completion packet to each worker thread using the PostQueuedCompletionStatus function, which informs each thread to exit immediately. PostQueuedCompletionStatus is defined as : 

view plaincopy to clipboardprint?

1. BOOL PostQueuedCompletionStatus(  
2.     HANDLE CompletionPort,  
3.     DWORD dwNumberOfBytesTransferred,   
4.     ULONG_PTR dwCompletionKey,   
5.     LPOVERLAPPED lpOverlapped  
6. );  

The CompletionPort parameter represents the completion port object to which you want to send a completion packet. The dwNumberOfBytesTransferred,dwCompletionKey, and lpOverlapped parameters each will allow you to specify a value that will be sent directly to the corresponding parameter of theGetQueuedCompletionStatus function. Thus, when a worker thread receives the three passed parameters of GetQueuedCompletionStatus, it can determine when it should exit based on a special value set in one of the three parameters. For example, you could pass the value 0 in the dwCompletionKey parameter, which a worker thread could interpret as an instruction to terminate. Once all the worker threads are closed, you can close the completion port using theCloseHandle function and finally exit your program safely. 
The completion port I/O model is by far the best in terms of performance and scalability. There are no limitations to the number of sockets that may be associated with a completion port and only a small number of threads are required to service the completed I/O. For more information on using completion ports to develop scalable, high-performance servers, see Chapter 6. 

Supplement : 
* [ MSDN 文章收集 ] I/O Concepts : I/O Completion Ports