Thursday, June 2, 2016

Getting a name for someone to connect back to your server.

When doing test automation it is often the case you need to know the name of the current machine in order to prompt another machine to connect to it, particularly if you are running your tests in parallel. This week I was trying to get the server under test to connect back to a WireMock server running on the slave test machine.

The standard response on stack overflow is to use the following pattern to get a network address. In my version here if we can't resolve the name then we are assuming we are running on a developers laptop on VPN so all the tests are run on the same machine. (Hence localhost)

String hostName = "localhost";
try {
    InetAddress addr = InetAddress.getLocalHost();
    String suggestedName = addr.getCanonicalHostName();
    // Rough test for IP address, if IP address assume a local lookup
    // on VPN
    if (!suggestedName.matches("(\\d{1,3}\\.?){4}") && !suggestedName.contains(":")) {
        hostName = suggestedName;
    }
} catch (UnknownHostException ex) {
}

System.out.println(hostName);

The problem comes is that we have to trust the local machine settings, for example /etc/hostname, which can result in a network name that is not accessible from another machine. To counter this I wrote the following code to work over the available network interfaces to find a remotely addressable network address name that can be used to talk back to this machine. (I could use an IP address but they are harder to remember, particularly as we are moving towards IPv6)

String hostName = stream(wrap(NetworkInterface::getNetworkInterfaces).get())
        // Only alllow interfaces that are functioning
        .filter(wrap(NetworkInterface::isUp))
        // Flat map to any bound addresses
        .flatMap(n -> stream(n.getInetAddresses()))
        // Fiter out any local addresses
        .filter(ia -> !ia.isAnyLocalAddress() && !ia.isLinkLocalAddress() && !ia.isLoopbackAddress())
        // Map to a name
        .map(InetAddress::getCanonicalHostName)
        // Ignore if we just got an IP back
        .filter(suggestedName -> !suggestedName.matches("(\\d{1,3}\\.?){4}")
                                 && !suggestedName.contains(":"))
        .findFirst()
        // In my case default to localhost
        .orElse("localhost");

System.out.println(hostName);

You might notice there a are a few support methods being used in there to tidy up the code, here are the required support methods if you are interested.

@FunctionalInterface
public interface ThrowingPredicate<T, E extends Exception>{

    boolean test(T t) throws E;
}

@FunctionalInterface
public interface ThrowingSupplier<T, E extends Exception>{

    T get() throws E;
}

public static <T, E extends Exception> Predicate<T> wrap(ThrowingPredicate<T, E> th) {
    return t -> {
        try {
            return th.test(t);
        } catch (Exception ex) {
            throw new RuntimeException(ex);
        }
    };
}

public static <T, E extends Exception> Supplier<T> wrap(ThrowingSupplier<T, E> th) {
    return () -> {
        try {
            return th.get();
        } catch (Exception ex) {
            throw new RuntimeException(ex);
        }
    };
}

// http://stackoverflow.com/a/23276455
public static <T> Stream<T> stream(Enumeration<T> e) {
    return StreamSupport.stream(
            Spliterators.spliteratorUnknownSize(
                    new Iterator<T>() {
                public T next() {
                    return e.nextElement();
                }

                public boolean hasNext() {
                    return e.hasMoreElements();
                }
            },
                    Spliterator.ORDERED), false);
}

Thursday, April 7, 2016

Converting string configuration properties to other types, with a bit of Optional.

Somedays you come across some code and think that's pretty, why didn't I think of that? So my long time colleague Mark Warner has a nice twist on the standard name/value store pattern using method references to deal with converting from a String.

int size = store.getProperty("cache.limit", 500, Integer::parseInt);
    boolean enabled = store.getProperty("cache.enabled", true, Boolean::getBoolean);

I took his example and refactored it slightly to return Optional, and I ended up with the following:

public Optional<String> getProperty(
        String propertyName) {
    return Optional.ofNullable(map.get(propertyName));
}
   
public <T> Optional<T> getProperty(
        String propertyName, 
        ThrowingFunction<String,? extends T,? extends Exception> func ) {

    return getProperty(propertyName).map(val -> {
        try {
            return func.apply( val );
        } catch ( Exception e ) {
            LOGGER.severe( () -> "Invalid property transform, will default " + e.getMessage() );
            return null;
        }
    });
}

This means that the default value ends up being provided by the Optional which is a nice application of OAOO.

int size = store.getProperty("cache.limit", Integer::parseInt).orElse(500);
    boolean enabled = store.getProperty("cache.enabled", Boolean::getBoolean).orElse(true);

I think this is even tidier; but it does depend on who you feel about using Optionals.

Thursday, March 10, 2016

Lambda of Lambda, if/else from an Optional

So I got frustrated with two limitations of the Optional interface in JDK 8. The first problem is that there is no obvious way to perform an else operation in a block as there is only a isPresent method unless you are using an old school if statement. The second problem is of course the old chestnut that even if you could do that the methods would not be able to throw a checked exception. (Yes you can wrap with a RuntimeException but it is not the prettiest.)

The workaround I found was to use the map function as the success case and the orElseGet to return the failure case. In both branches the code returns an instance of ThrowingRunnable by having a lambda return a lambda. The run() is then called at the end and it can throw any exception it wants to.

@FunctionalInterface
public interface ThrowingRunnable<E extends Throwable>  {
    
    public void run() throws E;
}



Optional<Credential> credential = ....

credential.<ThrowingRunnable<IOException>>map(auth -> () -> {
                PasswordWrapper pw = auth.getToken();
                ... // something that might throw an IOException
            }).orElseGet(() -> () -> {
                        response.setStatus(401);
                        LOGGER.log(Level.INFO, "credential is not found");
                    }
            ).run();


This is possibly excessive for this particular use case; but I can see this technique being useful elsewhere and it is worth knowing what it looks like so it is not a surprise in others code.

Update 14th March 2016: Thanks to Michael Rasmussen on the DZone version of this article he noted I could use orElse rather than orElseGet to remove the second double lambda.

credential.<ThrowingRunnable<IOException>>map(auth -> () -> {
                PasswordWrapper pw = auth.getToken();
                ... // something that might throw an IOException
            }).orElse(() -> {
                        response.setStatus(401);
                        LOGGER.log(Level.INFO, "credential is not found");
                    }
            ).run();



Wednesday, March 2, 2016

Getting hold of an un-used port when writing a test

A really quick post today, I was writing a unit test that needed to put up a temporary JAX-RS resource in order to test something very specific with out the need of a large testing framework. Unfortunately the code I was using didn't know about replacing zero with a random allocated port so I had to do something different. The nice tidy code that does this looks like the following:

try(ServerSocket ss = new ServerSocket(0)) {
   freePort = ss.getLocalPort();
}

server = createServerWithPort(freePort);

But of course there is a race condition here as there could be another process on the machine running the exact same code that gets lucky with its time on the CPU - note the API I was using couldn't use the ServerSocket directly so there is a gap where the port is not tied up. So to rule out the one in a million failure chance, which always comes on a Friday at 5pm, we need to wrap this code in a loop.

while(server==null) {
    try(ServerSocket ss = new ServerSocket(0)) {
       freePort = ss.getLocalPort();
    }

    server = createServerWithPort(freePort);
}

This may seem extreme; but if you are running some kind of CI system like Hudson or Jenkins it is not unusual to be running multiple executors on the same machine. You might have two developers running preflights at the same time causing an annoying intermittent failure as described above.

Finally a version using a JDK 8 stream with a helper method to convert the IOException from ServerSocket into a RuntimeException. Some would argue this is less readable; but I find it more explicit.

server = Stream.generate(convertException(() -> {
    try(ServerSocket ss = new ServerSocket(0)) {
        return ss.getLocalPort();
    }
})).map(TestClass::createServerWithPort)
.filter(Objects::nonNull)
.findFirst().get();

Wednesday, January 20, 2016

Removing multiple items from a JavaScript array, no for loops

So I was reviewing some code today that was using two for loops to remove elements from a JavaScript array, I won't relate the code here but I figured there as got to be a nicer way. So the first guess is to make use of the filter method; but that creates a new array which might be problem if the array is large but not if it is fairly small or not called very often.

providers = providers.filter(function(provider) {
     return provider.age > 35
}

For performance or API reasons you might want to perform an in place removal using splice, so you can simple map the values to indexes you want to remove, reverse the order then perform a sequence of splice operations:

providers.map(function(provider, index) {
    return provider.age > 35 ? index : -1;
}).filter(function(index) {
    return index >= 0;
}).reverse().forEach(function(index) {
    providers.splice(index,1);
});

You can play with this code in this jsfiddle.

Update 21 Jan 20125: The hazard of doing full stack development is that it is easy to accidentally duck type between languages whilst forgetting the impact. In Java the equivalent streaming code using map just returns another lazy step in the stream whereas in JavaScript you end up creating a new array of the same size as the original. This removes the performance improvement over the first example if you have lots of data. This is simple too fix though as you can use the reduce function to just create an array and populate it using the reduce operation:

providers.reduce(function(list, provider, index) {
    if (provider.age> 35) list.push(index);
    return list;
}, []).reverse().forEach(function(index) {
    providers.splice(index,1);
});

Here is the updated fiddle. I guess at some point I am going to have to benchmark these variants to see which is best; but that is for a less busy day.

Tuesday, January 12, 2016

Beware of slightly different behaviour between ConcurrentMap and ConcurrentHashMap for computeIfAbsent

So I was refactoring some code to use non-locking collection classes and I noticed the this significant different between ConcurrentMap.computeIfAbsent and ConcurrentHashMap.computeIfAbsent. They key different is that for the former default implementing the mapping function can be called many times for a particular key where as for the concrete implementation it will be called only once. This will likely affect whether the code in the function needs to be thread safe or not.

Right okay so armed with that you know that any collection that implements just ConcurrentMap will inherit this behaviour, further to that I found in particular guava would return a different implementation depending on the passed on parameters:

ConcurrentMap map1 = new MapMaker().makeMap();
System.out.println(map1.getClass());

ConcurrentMap map2 = new MapMaker().weakKeys().makeMap();
System.out.println(map2.getClass());

....

class java.util.concurrent.ConcurrentHashMap
class com.google.common.collect.MapMakerInternalMap

MapMakerInternalMap doesn't override computerIfAbsent therefore the behaviour of this function will be significantly different depending on the parameters to pass into the maker, something that might be apparent from the get go.

Wednesday, November 25, 2015

A very simple implementation of Suite to make it easy to run across a bunch of test machines at the same time.

So I was presented with a large suite that I need to distribute amongst an arbitrary number of Hudson slaves. Now my first guess would be to hand balance this into N sub suites; but that would require on-going work and would need to change depending on the number of nodes. (We have a target of 30 minutes to run all tests in parallel which is going to take some machines)

@RunWith(Suite.class)
@Suite.SuiteClasses({
    TableCRUD.class,
    TableCRUDChild.class,
    TableFilterSort.class,
    TableDefaultQuery.class,
    TableActions.class,
    ....
})
public class DevSuite {

}

So the quickest way I found to do this was to write different version of Suite that allocated the tests to different bins. This implementation doesn't ensure properly balanced bins as it is biased towards keeping the order stable and consistent which is useful for comparing tests results split into multiple jobs.

import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
import java.util.function.Function;
import java.util.function.ToIntFunction;
import java.util.stream.Collector;
import java.util.stream.Collectors;
import org.junit.runner.Runner;
import org.junit.runners.Suite;
import org.junit.runners.model.InitializationError;
import org.junit.runners.model.RunnerBuilder;

/**
 * Try to proportion a suites tests into "SPLIT" number of equals sections and
 * then just run "SECTION" rather than all of the tests. Defined by
 * "SplitSuite.split" and "SplitSuite.section" respectively
 */
public class SplitSuite
        extends Suite {

    private static int SECTIONS
            = Integer.getInteger("SplitSuite.split", -1);
    private static int SECTION
            = Integer.getInteger("SplitSuite.section", -1);

    public SplitSuite(RunnerBuilder builder, Class<?>[] classes) throws InitializationError {
        super(builder, classes);
    }

    public SplitSuite(Class<?> klass, RunnerBuilder builder) throws InitializationError {
        super(klass, builder);
    }

    public List<Runner> getChildren() {
        if (SECTIONS < 0) {
            return super.getChildren();
        } else if (SECTIONS == 0) {
            throw new IllegalArgumentException("SplitSuite.split needs to be a positive integer, it cannot be zero. A negative valid will disable this feature.");
        } else if (SECTION >= SECTIONS) {
            throw new IllegalArgumentException("SplitSuite.section parameter " + SECTION + " needs to be less than SplitSuite.split " + SECTIONS);
        }

        final Map<Integer, List<Runner>> collect = originalList.stream().collect(
                assignToGroups(SECTIONS, Runner::testCount, originalList));

        return collect.get(SECTION);
    }

    
    /**
     * Assigns object to buckets, moving to the next when filled, whilst preserving
     * the original order.
     * @param <Type> The type that has some kind of size property
     * @param sections The number of sections to split the code across
     * @param sizer A function to return the size of the given object
     * @param originalList The original list to provide a base line size
     * @return A map that contains an order list of sections
     */
    public static <Type> Collector<Type, ?, Map<Integer, List<Type>>> assignToGroups(
            int sections, 
            final ToIntFunction<Type> sizer,
            Collection<Type> originalList) {

        // Get length
        int count = originalList.stream().collect(Collectors.summingInt(sizer));
        // Workout section length
        int sectionLength = count / sections;
        
        Function<Type, Integer> grouping = new Function<Type, Integer>() {
            
            int counter = 0;
            
            @Override
            public Integer apply(Type t) {
                int section = Math.min(counter / sectionLength, sections - 1);
                counter += sizer.applyAsInt(t);                
                return section; 
            }
        };
        
        
        return Collectors.groupingBy(grouping, TreeMap::new, Collectors.toList());
    }
    
    
}



So a quick change to the suite class....

@RunWith(SplitSuite.class)
@Suite.SuiteClasses({
    TableCRUD.class,
    TableCRUDChild.class,
    TableFilterSort.class,
    TableDefaultQuery.class,
    TableActions.class,
    ....
})
public class DevSuite {

}

Now the actual test step looks like this, note that this is running under hudson but you should be able to achieve something similar on the CI server of your choice. In order to make my life easier I derive the section number from the job name for so 3 sections I have ..._0 ..._1 and ..._2. In hudson the later jobs all cascade from the the first sharing all properties which makes maintenance a lot easier.

java ... -DSplitSuite.sections=${SECTIONS} -DSplitSuite.section=`echo ${JOB_NAME} | sed -e s/.*_//` ...

And it appears to work, obviously the algorithm to split the suites could be better to produce more balanced results; but that is an effort for another day.....