Some commercial Web services software provides sophisticated Web services accounting features, recording details of Web services transactions recognized on the wire. But sometimes developers need accounting that is more modular, much more basic, and available on a shoestring. This article explains how to use advanced function composition tasks to add basic Web services monitoring capabilities.
Technologies that predate mainstream Web services offer a dual nature in the way they fit into frameworks. The likes of ONC RPC, CORBA, and XML-RPC can be comprised either through the language-specific stubs that drive their common usage or through their wire formats. Web services consist of more than just RPC style, of course, but this style does dominate its mainstream use. What this means in practice is that developers create stubs for remote objects by writing compatible interfaces by hand, processing WSDL files, or even dynamic methods and introspection. But the services invocation then comes as a simple method call.
As these method calls get marshaled to wire formats and stream across the network, developers and administrators often want to keep track of things by maintaining detailed transaction logs in order to debug service behavior, profile performance, test security controls, and even model traffic shape for service-level improvement. A variety of vendors have lined up to provide tools to help with such monitoring, almost all of which focus on actual wire transactions. You can even get a box from some vendors that you pop into the rack in your data center, and plug into the network to gain such monitoring capabilities
Wire monitoring tools are generally expensive and require a lot of set-up and care. Considering that the idea behind Web services is that interchange can come from a diversity of end-points in a diversity of languages and platforms, wire monitoring is probably the most comprehensive approach in an enterprise environment. Then again, in reality most Web services in deployment are intra-departmental and even point-to-point. When Web services are used as an applications integration tool, such comprehensive monitoring is rarely needed. Even when such comprehensiveness is justified, it often makes sense to defer deployment of full-blown monitoring systems until production testing, which means that developers in earlier phases still need a means of keeping track of Web services for debugging and planning purposes.
It is also useful to have the ability to support Web services monitoring through the operation of the programming language stubs at the end points. The most general way of approaching this is by using techniques of function composition according to available language features. Most programmers think of such techniques as building wrappers and proxies.
Compose your methods
In functional and dynamic languages, function composition is usually straightforward at its root and flexible enough to come in numerous implementations. To demonstrate the basic technique, Listing 1 shows Python code that uses dynamic features to log all invocations of a remote method to standard error on the console.
Listing 1. Demonstration of function composition in Python for mixing logging capabilities into SOAP
import sys #Use standard system object of the interpreter
import SOAPpy #Use the third-party SOAP/WSDL toolkit SOAPpy
class logging_proxy:
  """
  Requires Python 2.2 or more recent for proper operation
  """
  def __init__(self, subject, stream):
  """
   This is a documentation string, similar to Javadoc
  __init__ is the Class initializer (like a constructor)
  subject - the object to which we delegate
  stream - the file-like object for log output
  Special parameter:
  self - the instance upon which a method is invoked
  """
  self.subject = subject
  self.stream = stream
  return
  def __getattr__(self, attr):
  """
  Special method name that intercepts all requests for an
  attribute or method on an instance.
  attr - the name of the requested attribute/method
  """
  #getattr is a special global function which gets the named
  #attribute of an object.
  #Result is generally a data member or function object (method)
  obj = getattr(self.subject, attr)
  if callable(obj):
   #Then it's a method, not a plain attribute
   def log_wrapper(*args, **kwargs):
    #Write pre-call log
    self.stream.write("Invoking remote method %sn"%attr)
    self.stream.write("tpositional params: %sn"%repr(args))
    self.stream.write("tnamed params: %sn"%repr(kwargs))
    #Now invoke the service, passing on the parameters
    result = obj(*args, **kwargs)
    #Write post-call log
    self.stream.write("Method %s completedn"%attr)
    self.stream.write("tReturn value: %sn"%result)
    return result
   return log_wrapper
   else:
    return obj
WSDL_URI = "http://www.xmethods.net/sd/2001/TemperatureService.wsdl"
#Create the remote service proxy
#All the needed data comes from the WSDL
service = SOAPpy.WSDL.Proxy(WSDL_URI)
#wrap the service to the add logging feature (log to stderr)
wrapped_service = logging_proxy(service, sys.stderr)
#From here on all invocations of methods on the Web service
#Automatically go through the logging code first
zipcode = '80027'
temp = wrapped_service.getTemp(zipcode)
print 'Temperature in US postal code ', zipcode, 'is', temp, 'F'
The definition of the class logging_proxy need only occur once. You can then wrap just about any class with it to meld in the logging capability. If you’re not familiar with Python, this function might be a bit hard to follow. The heart of the technique is the definition of log_wrapper, which forms what is called a closure. For Java programmers this is reminiscent of inner classes. The closure captures the information about the actual Web service method stub to be invoked (the function object called obj) as well as the stream for the log output. The caller gets this closure when it invokes a SOAP method by name, which triggers the special method __getattr__. In usual cases of method invocation the return result is a function object, which is then executed. In this case the returned function object is a closure which encapsulates some logging code and the data needed to call the original function. The overall effect is a composition of the log_wrapper code and the actual remote method invocation. The rest of the code in the module puts the technique to use. Only one additional line is required besides the usual SOAP client code:
service = logging_proxy(service, sys.stderr)
This one-time set-up is required for each remote service. As you can see, the output stream is parameterized: You can send some logs to a file and others to a network socket. You can parameterize other details easily enough. Overall, this technique provides a huge amount of flexibility. The following snippet shows the output from Listing 1:
$ python listing1.py
Invoking remote method getTemp
    positional params: ('80027',)
    named params: {}
Method getTemp completed
    Return value: 21.0
Temperature in US postal code 80027 is 21.0 F
The last line represents the normal processing of the method result. The five previous lines are output from the logging code.
A proxy on your network
Another approach to monitoring is to listen at the network level. Simply set up a proxy server of your own that performs some monitoring action while forwarding traffic to and from each remote service end-point. The proxy server listens on the same protocol as the remote server, but when it gets a request it simply performs the necessary action, such as logging the request. It then forwards the request to the remote service, and passes any response back to the original client. Listing 2 is a program that operates as such a proxy for SOAP over HTTP. It’s also in Python, but the nature of this proxying approach is such that this code can be used with any Web service regardless of the language used to implement either SOAP end point.
Listing 2. Script that logs then forwards SOAP/HTTP messages
import sys
import BaseHTTPServer
import httplib
class ProxyHandler(BaseHTTPServer.BaseHTTPRequestHandler):
  def do_POST(self):
   clen = int(self.headers.getheader('content-length'))
  #dest is the true SOAP end point address. It comes in as
  #a standard HTTP Host header, that is the host then the port
  #separated by a colon. Break this into the components
  dest = self.headers.getheader('Host')
  dest_host, dest_port = dest.split(':')
  #Read the HTTP payload
  request_body = self.rfile.read(clen)
  #Log the HTTP details to STDERR
  sys.stderr.write('Path:' + self.path + 'n')
  sys.stderr.write('Headers:n')
  for k, v in self.headers.items():
   sys.stderr.write('t' + str(k) + ': ' + str(v) + 'n')
  sys.stderr.write('Payload:n')
  #input_body is the request SOAP envelope and contents
  sys.stderr.write(request_body)
  #Create a matching request to the actual destination
  requestor = httplib.HTTP(dest_host, dest_port)
  requestor.putrequest('POST', self.path)
  for k, v in self.headers.items():
   requestor.putheader(k, v)
  requestor.endheaders()
  requestor.send(request_body)
  #Wait for the response from the remote host and forward it
  #as a response from this host
  (status_code, message, response_headers) = requestor.getreply()
  response_body = requestor.getfile().read()
  self.send_response(200, 'OK')
  for k, v in response_headers.items():
   self.send_header(k, v)
  self.end_headers()
  self.wfile.write(response_body)
  return
LISTENING_PORT= 8888
#Set up to run on local machine
SERVER_ADDRESS = ('127.0.0.1', LISTENING_PORT)
httpd = BaseHTTPServer.HTTPServer(SERVER_ADDRESS, ProxyHandler)
print "Listening on port", LISTENING_PORT
#Go into a the main event loop
httpd.serve_forever()
The method do_POST on ProxyHandler is executed once the server receives SOAP by way of an HTTP POST invocation. It logs the HTTP headers and payload (that is, the SOAP message) and then makes an equivalent call to the actual remote end point. The HTTP response from the remote end point is forwarded back to the original client. This proxy code is very flexible because, if used correctly, it can determine the actual destination from the HTTP headers. In the code, this is read into the dest variable. This means that you can use this one SOAP/HTTP proxy to handle any number of requests for all variety of actual destination end points. The original clients see the same network traffic as if they were communicating to the remote server directly. Listing 3 is an example of client code that uses the SOAP/HTTP proxy.
Listing 3. SOAP client that proxies its messages for logging
import sys    #Use standard system object of the interpreter
import SOAPpy    #Use the third-party SOAP/WSDL toolkit SOAPpy
WSDL_URI = "http://www.xmethods.net/sd/2001/TemperatureService.wsdl"
#WSDL_URI = "http://localhost:8888/sd/2001/TemperatureService.wsdl"
#Create the remote service proxy
#All the needed data comes from the WSDL
service = SOAPpy.WSDL.Proxy(WSDL_URI)
service.soapproxy.http_proxy = 'localhost:8888'
zipcode = '80027'
temp = service.getTemp(zipcode)
print 'Temperature in US postal code ', zipcode, 'is', temp, 'F'
This is pretty much boilerplate remote procedure calling code except for the line:
service.soapproxy.http_proxy = 'localhost:8888'
Most SOAP libraries give you a mechanism for directing a SOAP request through a proxy. This line sets an HTTP proxy for the SOAP client, pointing it to port 8888 on the local machine. Because setting the HTTP Host header is a standard step in using an HTTP proxy, very little additional work is needed to get the proxy server to forward the traffic properly.
The main limitation with this HTTP proxy trick is that you have to add more customization to handle usage patterns that are more complex than the common request/response interaction. Luckily, such customization isn’t terribly difficult.
Wrap up
I have just scratched the surface of function composition and the capabilities of network service proxying. These themes have many variations and most languages support similar capabilities. If you’re using a functional language (such as Python), you can follow up on these compositional tricks by experimenting with currying and other such techniques. If you’re using a static object-oriented language such as Java, look into aspect-oriented programming (AOP). Programmers have lots of options for adding infrastructural capabilities to Web services without having to buy expensive wire monitoring tools.
Resources
Find out how you can become an IBM Certified Developer in XML and related technologies.
First published by IBM developerWorks at http://www.ibm.com/developerWorks/.
Uche Ogbuji is a consultant and co-founder of Fourthought Inc., a software vendor and consultancy specializing in XML solutions for enterprise knowledge management. Fourthought develops 4Suite, an open source platform for XML, RDF, and knowledge-management applications. Mr. Ogbuji is also a lead developer of the Versa RDF query language. He is a computer engineer and writer born in Nigeria, living and working in Boulder, Colorado, USA. You can contact Mr. Ogbuji at uche.ogbuji@fourthought.com.
