Make shared memory work for you, not against you
Making the most of shared memory isn’t always easy. In this article, IBM’s Sachin Agrawal shares his expertise in C++, showing how the object-oriented among us can take key advantage of a uniquely useful interprocess communications channel.
In terms of time and space, shared memory is probably the most efficient inter-process communication channel provided by all modern operating systems. Shared memory is simultaneously mapped to the address space of more than one process: a process simply attaches to the shared memory and starts communicating with other processes by using it as it would use ordinary memory.
However, in the object-oriented programming world, processes prefer to share objects rather than raw information. With objects, there is no need to serialize, transport, and de-serialize the information contained within the object. Shared objects also reside in shared memory, and although such objects “belong” to the process that created them, all processes on the system can access them. Hence, all of the information within a shared object should be strictly process-neutral.
This is a direct contradiction to the C++ object model currently adopted by all popular compilers: C++ objects invariably contain pointers to various Vee-Tables and sub-objects that are process-specific. For such objects to be sharable, you need to make sure that the target of these pointers resides at the same address in all the processes.
With the help of a small sample program, this article illustrates cases where the C++ model succeeds, where it fails to work with the shared memory model, and where possible workarounds exist. The discussion and the sample program are limited to non-static data members and virtual functions. Other situations are not as relevant to the C++ object model as these are: static and non-static non-virtual member functions do not have any issues in shared environment. Per-process static members do not reside in shared memory (and thus have no issues), while shared static members have issues similar to those discussed here.
Environmental assumptions
This text is highly specific to the Red Hat Linux 7.1 distribution for 32-bit x86 Intel architectures, as GNU C++ compiler version 2.95 and associated tools were used to build and test the sample program. However, you can apply the overall concepts equally well to any machine architecture, operating system, and compiler combinations.
Sample program
The sample program consists of two clients, shm_client1 and shm_client2, and uses the shared objects services provided by the shared library shm_server. Object definitions reside in common.h:
Listing 1. Definitions in common.h
#ifndef __COMMON_H__
   #define __COMMON_H__
   class A {
   public:
     int m_nA;
     virtual void WhoAmI();
     static void * m_sArena;
     void * operator new (unsigned int);
   };
   class B : public A {
   public:
     int m_nB;
     virtual void WhoAmI();
   };
   class C : virtual public A {
   public:
     int m_nC;
     virtual void WhoAmI();
   };
   void GetObjects(A ** pA, B ** pB, C ** pC);
   #endif //__COMMON_H__
Listing 1 defines three classes (A, B and C) with a common virtual function WhoAmI(). The base class, A, has a member named m_nA. The static member m_sArena and overloaded operator new() are there to enable the construction of objects in shared memory. Class B is simply derived from A, and class C is virtually derived from A. Both B::m_nB and C::m_nC are provided to ensure that the sizes of A, B, and C are distinct. This simplifies the implementation of A::operator new(). The interface GetObjects() returns the shared objects pointers.
The shared library implementation is in shm_server.cpp:
Listing 2. Library – shm_server.cpp
#include
   #include
   #include
   #include
   #include
   #include
   #include "common.h"
   void * A::m_sArena = NULL;
   void * A::operator new (unsigned int size)
   {
     switch (size)
     {
     case sizeof(A):
      return m_sArena;
    
     case sizeof(B):
      return (void *)((int)m_sArena + 1024);
     case sizeof(C):
     return (void *)((int)m_sArena + 2048);
     default:
      cerr m_nA = 1;
     pA = new B;
     pA->m_nA = 2;
     pA = new C;
     pA->m_nA = 3;
    }
    return;
   }
Let’s look at Listing 2 in more detail:
Lines 9-25: operator new ()
The same overloaded operator allows you to construct objects of class A, B, and C in shared memory. Object A starts right at the beginning of shared memory. Object B starts at offset 1024, and C at offset 2048.
Lines 26-34: Virtual functions
The virtual functions simply write a line of text to standard output.
Lines 35-39: GetObjects
GetObjects() returns the pointers to shared objects.
Lines 40-46: Initializer
This class stores the shared memory identifier. Its constructor creates the shared memory and objects in it. If the shared memory already exists, it simply attaches to it. The static member m_sInitializer ensures that the constrictor is invoked prior to the main() function of the client module that is using the shared library.
Lines 48-82: Initializer::Initializer()
Shared memory is created if it does not exist, and shared objects are created within it. The object construction is skipped if the shared memory already exists. Initializer::m_shmid records the identifier and A::m_sArena records the shared memory address.
The shared memory is not destroyed even after all processes detach from it. This allows you to explicitly destroy it using the ipcrm command or to make some quick observations with the ipcs command.
The client process implementation is in shm_client.cpp:
Listing 3. Client – shm_client.cpp
#include "common.h"
   #include
   #include
   int main (int argc, char * argv[])
   {
    int jumpTo = 0;
     if (1 jumpTo) || (6 m_nA WhoAmI();
    case 3:
     cout m_nA WhoAmI();
    case 5:
     cout m_nA WhoAmI();
    }
    return 0;
   }
   #include
   void DoNothingCode() {
    pthread_create(NULL, NULL, NULL, NULL);
   }
Lines 6-35
The client process gets pointers to three shared objects, makes three references to their data members, and — depending on the command-line input — invokes three virtual functions.
Lines 36-39
The uninvolved pthread_create() function is used to force linking with another shared library. Any function from any shared library will serve this purpose.
The shared library and two instances of client executable are built as follows:
gcc shared g shm_server.cpp o libshm_server.so lstdc++
gcc -g shm_client.cpp -o shm_client1 -lpthread -lshm_server -L .
gcc -g shm_client.cpp -o shm_client2 -lshm_server -L . lpthread
Note that the link sequence of shm_server and pthread for shm_client1 and shm_client2 is swapped to ensure that the base address of the shm_server shared library in both executables is different. This can further be verified using the ldd command. Sample output is usually something like the following:
Listing 4. Library map for shm_client1
ldd shm_client1
libpthread.so.0 => (0x4002d000)
libshm_server.so => (0x40042000)
libc.so.6 => (0x4005b000)
ld-linux.so.2 => (0x40000000)
Listing 5. Library map for shm_client2
ldd shm_client2
libshm_server.so => (0x40018000)
libpthread.so.0 => (0x40046000)
libc.so.6 => (0x4005b000)
ld-linux.so.2 => (0x40000000)
The main aim here is to build two client binaries that have distinct base addresses for the server library. In the context of this sample program, using the uninvoked pthread_create() function and the different link sequence for shared libraries works to accomplish this goal. However, there is no concrete rule or uniform procedure available that works across all linkers; different methods will need to be employed on a case-by-case basis.
Case 1: shm_client1 vs. shm_client1
In the following output, the shm_client1 is invoked first from the shell. Because no shared objects are present, it creates them, references their data members, invokes their virtual functions, and quits — leaving the objects behind in memory. The second time, the process simply references the data members and virtual functions.
Listing 6. Output log for shm_client1 vs. shm_client1
   Created the shared memory
   1073844224 1073845248 1073846272
   1
   Object type: A
   2
   Object type: B
   3
   Object type: C
   $ ipcs
   —— Shared Memory Segments ——–
   key shmid owner perms bytes nattch status
   0x000004d2 2260997 sachin 666 3072 0
    1073840128 1073841152 1073842176
    1
    Object type: A
    2
    Object type: B
-> 0
-> Segmentation fault (core dumped)
When the second process tries to refer to data member A::m_nA via the pointer of type C * (you will remember that C is virtually derived from A), the base sub-object pointer from within the shared object is read. The shared object was constructed in the context of a now non-existing process. Hence, garbage is read for both A::m_nA and C::WhoAmI().
This is because the Vee-Table and virtual functions lie within the shm_server shared library, which happens to be reloaded at the same virtual address. Hence, no problem is observed while de-referring pointers of type A * and B *.
Hence, the C++ object model adopted by GNU fails with virtual inheritance.
Case2: shm_client1 vs. shm_client2
In the next sample output, first the shm_client1 and then the shm_client2 are executed from the command line:
Listing 7. Output log for shm_client1 vs. shm_client2
   Created the shared memory
   1073844224 1073845248 1073846272
   1
   Object type: A
   2
   Object type: B
   3
   Object type: C
   $ ipcs
   —— Shared Memory Segments ——–
   key shmid owner perms bytes nattch status
   0x000004d2 2359301 sachin 666 3072 0
   1073942528 1073943552 1073944576
   1
-> Segmentation fault (core dumped)
   1073942528 1073943552 1073944576
   2
-> Segmentation fault (core dumped)
   1073942528 1073943552 1073944576
-> 1048594
-> Segmentation fault (core dumped)
However, the Vee-Table lies within the shm_server shared library: it is loaded to different virtual addresses within shm_client1 and shm_client2. Hence, garbage is read for both A::WhoAmI() and B::WhoAmI().
Making shared memory work
You should consider two major issues when instantiating C++ objects within shared memory. First, the Vee-Table pointer is used to access virtual functions, and data members are accessed directly using compile time offsets. Hence, for all such shared objects, Vee-Table and virtual functions should have the same virtual addresses in all processes. There is no hard-and-fast rule for doing this, but adopting a proper link sequence for dependent shared libraries should work most of the time.
In addition, keep in mind that virtually inherited objects have base pointers to address base objects. Base pointers refer to data sections of the process, and are always process specific. It is difficult to ensure the same numerical values for these across all client processes. Hence, construction of virtually inherited objects in shared memory should be avoided for the assumed C++ object model. But keep in mind, too, that different compilers adopt different models. For instance, Microsoft Compiler uses process-neutral offsets to address base objects for virtually inherited classes: thus, this issue does not arise. The important thing is to ensure the same addresses for shared libraries in all client-processes.
Resources
ld, ldd, ipcs, ipcrm and so on. Purchase Linux books at discounted prices in the Linux section of the Developer Bookstore.
*originally published at IBM DeveloperWorks
Sachin has been working extensively in C++ for five years, including three years of research into the C++ object models of various compilers. He currently works for IBM Global Services India. You can contact him at sachin_agrawal@in.ibm.com.
