Geolocation Viewtool

Published on April 11, 2013 by in Development, Plugins, Viewtools

2

Geolocation – The Options and Issues

At Aquent, we have relied on being able to determine where our visitors are coming from for some time now. This allows us to target specific information to visitors based on where they are coming from. Getting that information is becoming easier and easier as there is no shortage of GeoIP Lookup Services available; from free services to paid, client-side and server-side, and in any programming language you could ask for. A couple of these we have used to varying degrees of success. The guys over at dotCMS have actually came up with a pretty nice example implementation on the 2.2 Demo Site:

  #set($clickstream  = $session.getAttribute("clickstream"))
  
  #set($geolocationFromSession = $session.getAttribute("geolocation"))
 
  #if(!$UtilMethods.isSet($geolocationFromSession))
    #set($locationURL = "http://www.geoplugin.net/json.gp?ip=$!clickstream.remoteAddress")
    #set($geolocation = $json.fetch("$!locationURL"))
    $session.setAttribute("geolocation", $geolocation)
  #else
    #set($geolocation = $session.getAttribute("geolocation"))
  #end

This code is using the JSON tool to access geoPlugin’s free service. It even goes as far as storing the data for the user’s session. At Aquent, we have used similar approaches. However, there is one slightly annoying issue with this approach that we ran into. What happens when geoPlugin’s service is not available? Unfortunately the JSON tool is not very resilient here. It holds on to the connection and will wait for it to timeout for a very long time. No amount of configuration seems to get it to fail-over quickly. On a busy site this quickly eats up HTTP connections and even has the potential to take your site down. Anyone who has worked with the JSON or XML tools to pull in 3rd-party resources into their dotCMS implementation knows exactly what I am talking about.

MaxMind Solution

What we wanted to do at Aquent was come up with a solution that does not depend on connections to an external site, has fast and accurate data that is easy to keep up to date, and is simple for our front-end developers to use. This is where MaxMind comes in. MaxMind provides several databases that you can download that are kept up to date weekly. Their Open Source Java API is lightning fast at looking up the data and very easy to use. They also supply a way to download the database files automatically. They offer both paid versions of the databases as well as free versions for public use (which aren’t updated quite as often, and will be a little less accurate as a result).

We decided to go with the GeoLite City Database for now, but plan on upgrading to the full version of the City Lookup Database in the near future. We decided to go the City Database so that we would be able to access a wide array of information about the visitor including:

  • Country Name/Code
  • Region
  • City Name
  • Postal Code (US only)
  • Latitude and Longitude
  • DMA, Area and Metro Codes (US only)

The only problem left to solve was to provide a way for our front-end developers to access the database in Velocity. The easiest way to accomplish this is a viewtool. You can pick up the viewtool over at our GitHub repo.

Writing the plugin

Writing the viewtool plugin was not very difficult. Here is quick overview of what we needed to do:

First, we need to compile the MaxMind Java API into a JAR that we could include in the plugin. There are some instructions in their readme file on how to do this and the resulting jar file goes into the lib folder of the plugin. Every plugin needs a MANIFEST.MF and build.xml which can be copied from the example configuration plugin.

Click Here if you want to skip the java and jump into using the viewtool. Now for the core code:

public class GeoIP implements ViewTool {
	private LookupService cityLookup;
	private boolean inited = false;
	PluginAPI pluginAPI = APILocator.getPluginAPI(); 

Every Viewtool must implement the ViewTool Interface. Then we start off by creating a LookupService.

	
	@Override
	public void init(Object initData) {

The init method of a viewtool gets executed when the viewtool is created. This can be different depending on what scope you set your viewtool as in your toolbox.xml. For this project, I want the tool to init once when dotCMS starts up. This is why I made the tool application scoped.

		String dbFileName = "";
		try {
			dbFileName = pluginAPI.loadProperty("com.aquent.plugins.geolocation", "maxmind.dbFileName");
		} catch (Exception e) {
			Logger.error(this,"Unable to load plugin property - maxmind.dbFileName", e);
			return;
		}
	
		String dbPath = "";
		if (UtilMethods.isSet(Config.getStringProperty("ASSET_REAL_PATH"))) {
			dbPath = Config.getStringProperty("ASSET_REAL_PATH") + File.separator + dbFileName;
		} else {
			dbPath = Config.CONTEXT.getRealPath(File.separator + Config.getStringProperty("ASSET_PATH") + File.separator + dbFileName);
		}	

We decided it would be best to store the database file in the assets directory. This means if you are running in a clustered environment you only need to maintain the file in one place. We then use a property in the plugin.properties file to determine the filename of the database to load.

		try {
			cityLookup = new LookupService(dbPath, LookupService.GEOIP_MEMORY_CACHE | LookupService.GEOIP_CHECK_CACHE);
		} catch (Exception e) {
			Logger.error(this,"Unable to get a LookupService",e);
			return;
		}

Now we create the LookupService and point it at our db file. We decided to use their Memory Cache option for caching the DB to make the lookups faster. You could also use the Standard or index cache options. I am not sure at this point how large the full database is in memory so we may have to move to a different cache method in the future. You can read more about the options in MaxMind’s Java API readme.

		// A flag to let the viewtool know we are good to go
		inited = true;
	}

Finally if we got to this point and there were no errors then we know the viewtool is inited. I set a boolean flag so in my methods I know if the viewtool is inited or not.

The last part threw me off a bit. I thought since the Location object contained public properties I could just return the Location Object and those properties could be accessed in Velocity. So I tried this first:

	public Location getLocation(String ip) {
		Logger.debug(this, "MaxMind GeoIP Viewtool - getLocation called with ip "+ip);

		// Get the Location from the LookupService
		Location loc = null;
		if(inited) {
			loc = cityLookup.getLocation(ip);
		} else {
			Logger.info(this,"Attempt to Call getLocation and not inited");
		}

		return loc;	
	}

Unfortunately it seems that velocity cannot just access properties on an object without getters and setters. So I needed to provide a method that returned a map with all of the properties:

	public Map<String, Object> getLocationMap(String ip) {
		Logger.debug(this, "MaxMind GeoIP Viewtool - getLocationMap called with ip "+ip);

		// Get the Location from the LookupService
		Location loc = null;
		Map<String, Object> locMap = new HashMap<String, Object>();
		if(inited) {
			loc = cityLookup.getLocation(ip);
			locMap.put("countryCode", loc.countryCode);
			locMap.put("countryName", loc.countryName);
			locMap.put("region", loc.region);
			locMap.put("city", loc.city);
			locMap.put("postalCode", loc.postalCode);
			locMap.put("latitude", loc.latitude);
			locMap.put("longitude", loc.longitude);
			locMap.put("dma_code", loc.dma_code);
			locMap.put("area_code", loc.area_code);
			locMap.put("metro_code", loc.metro_code);
		} else {
			Logger.info(this,"Attempt to Call getLocationMap and not inited");
		}

		return locMap;
	}

We decided to keep both methods because the Location Object has a nice distance method to compute the distance between 2 locations. We even decided to include a few helper methods so that you can pass in a latitude and longitude and get the distance as well. I know our front-end developers will enjoy this as we have lat/long in some of our structures already.

The last thing our plugin needed was the toolbox mapping. Pretty simple, give it a key that will be used in velocity to map to the viewtool, a scope, and point it at your viewtool’s class. As I stated before, for this project it made sense to make this an application scoped viewtool so that we only need to do the heavy operation of loading the database into memory once.

Using the plugin

So you are probably wondering at this point how to use the viewtool. The first thing you will need to do is grab the source from GitHub. You can either clone the repository or download it in a tar or zip form. You are going to want to then extract or clone that thing into your dotCMS’s plugin directory.

Next you will need to obtain the database file from MaxMind. Remember that you will need the City Database for this as the Country database will not work. You can either purchase the full version or download the Lite Version for free. Once you have the zip file you will want to extract this file into your shared assets directory. Finally, don’t forget to change the filename in conf/plugin.properties to match what you are save the db file as.

Now you are ready to stop dotCMS, run bin/deploy-plugins.sh (or bat), and startup dotCMS. On startup if you are tailing the log file you should see a message like: “MaxMind GeoIP Viewtool Started”. Now let’s dig into some velocity code:

Getting the User’s IP address

You have a couple options here. Depending if you have a load balancer and how it sends the IP in the header one or more of the following options might work for you:

#set($ip = $request.getRemoteAddr())
#set($ip = $session.getAttribute("clickstream").remoteAddress)
#set($ip = $request.getHeader('x-forwarded-for'))
#set($ip = $request.getHeader('x-cluster-client-ip'))

## A trick to finding the right header
#foreach($key in $request.getHeaderNames) 
  <b>$key</b> = $request.getHeader($key) <br />
#end

Getting the User’s Location

Now that you have the user’s IP address you can get the LocationMap and access the various properties:

## We might as well take a hint from the dotCMS Folks and take advantage of caching the user's location in the session for later use
#set($locFromSession = $session.getAttribute("loc"))
 
#if(!$UtilMethods.isSet($locFromSession))
  #set($loc = $geoip.getLocationMap($ip))
  $session.setAttribute("loc", $loc)
#else
  #set($loc = $session.getAttribute("loc"))
#end

## Now you have access to the following data:
<dl> 
  <dt>IP:</dt>
  <dd>${ip}</dd>
  <dt>${loc}</dt>
  <dd></dd>
  <dt>Country Code:</dt>
  <dd>${loc.countryCode}</dd>
  <dt>Country Name:</dt>
  <dd>${loc.countryName}</dd>
  <dt>Region:</dt>
  <dd>${loc.region}</dd>
  <dt>City:</dt>
  <dd>${loc.city}</dd>
  <dt>Postal Code:</dt>
  <dd>${loc.postalCode}</dd>
  <dt>Latitutde:</dt>
  <dd>${loc.latitude}</dd>
  <dt>Longitude:</dt>
  <dd>${loc.longitude}</dd>
  <dt>DMA Code:</dt>
  <dd>${loc.dma_code}</dd>
  <dt>Area Code:</dt>
  <dd>${loc.area_code}</dd>
  <dt>Metro Code:</dt>
  <dd>${loc.metro_code}</dd>
</dl>

Computing Distance

Lastly, if you want to determine the distance between 2 locations you have a couple of options:

## First get a couple Location Objects or pull some latitude and longitude data from one of your structures:
#set($locObj  = $geoip.getLocation($ip))
#set($loc2    = $geoip.getLocationMap('213.52.50.8'))
#set($loc2Obj = $geoip.getLocation('213.52.50.8'))

## If you have two Location Objects you can get the distance using the Location Objects:
<p>Distance between ${esc.d}loc and ${esc.d}loc2 = ${locObj.distance($loc2Obj)} miles</p>

## If you have one Location Object and a set of lat,long for a second location you can get the distance using the viewtool helper method:
<p>Distance between ${esc.d}loc and ${esc.d}loc2 = ${geoip.distance($locObj, $loc2.latitude, $loc2.longitude)} miles</p>

## If you have two sets of lat,long you can use the viewtool helper method as well:
<p>Distance between ${esc.d}loc and ${esc.d}loc2 = ${geoip.distance($loc.latitude, $loc.longitude, $loc2.latitude, $loc2.longitude)} miles</p>

Keeping your database up-to-date

MaxMind also provides a C Library that you can use to keep your database up-to-date. Setting this up is fairly simple. They have a full set of instructions over on their site. Just don’t forget to pass the -d option to tell it to save your database file in your assets folder:

Usage: geoipupdate [-hv] [-f license_file] [-d custom directory]

Compatibility

Currently I have tested the code on 1.9.5.x and 2.2.1. In the future we plan on converting the plugin to OSGI, but we wanted to maintain compatibility with the 1.9.5 folks. Let us know either on GitHub or in a comment here if it useful for you or if you have feature requests or find any bugs.

2 Responses to “Geolocation Viewtool”

  1. Tim says:

    Do you have the support for free IP2Location LITE?

    It provides ZIP code information in the free package. I wish dotCMS is supporting it.

    • Michael Fienen says:

      Tim, we can look into it, but I’m pretty sure IP2Location uses a fairly different method of data lookup that wouldn’t be immediately compatible with the way our plugin works. Maintenance would likely be harder as well, as they recommend loading their CSV file into a database table, whereas MaxMind’s data is its own flatfile scheme. MaxMind also provides Java support out of the box, but IP2Location’s tool costs $99, unless we were to write our own class.

      At any rate, I’ll at least make a note to look into it some more. It might be possible to do it more easily than I think, but at least for now, it may be a long shot.

Leave a Reply