Thursday, 19 November 2015

Negative diagonal elements in the covariance matrix returned by numpy.polyfit

I ran into a strange issue fitting a line to a small number of data points using numpy.polyfit that I thought was worth documenting.

I ran a command of the form:

p, cov = np.polyfit(x,y,1,w,cov=True),

where x, y and w were arrays of length 3.

The command returned the correct slope and y-intercept values, however the covariance matrix, cov, had strictly negative diagonal terms. This is apparantly because numpy scales the covariance matrix as described in here.

The scaling applied is a factor such that

factor = resids / (len(x) - order - 2.0)

If, like me, you are making a first order polynomial fit to a dataset of 3 values, the denominator has the effect of multiplying the expected matrix by -1. If I was unlucky enough to have 4 points, it would have thrown bigger errors.

In my case, looking at the results here, I could recover the correct values just by multiplying the matrix by minus one. This is a strange weighting to apply to a small dataset - I assume it makes sense if you have many points and the developers wanted to keep numpy.polyfit consistent.

Monday, 5 October 2015

Fixing AIPS after upgrading to El Capitan on OSX

I recently ran into some AIPS trouble after upgrading from Mavericks to El Capitan. My installation of AIPS complained about X11 and about being unable to find some libraries that normally come with the binary distribution I had actually installed.

The library fix should have been the easiest - AIPS was looking for:

  • libsvml.dylib
  • libirc.dylib
  • libimf.dylib
All of these come with the binary and are located in 31DEC14/MACINT/LIBR/INTELCMP of the AIPS directory (note I'm still using the frozen AIPS from December 2014).

AIPS automatically adds this location to DYLD_LIBRARY_PATH when you run LOGIN.SH, but even though the libraries are on the path, AIPS does not seem to be able to find them when it needs them. I fixed this temporarily by making a simlink for each of the three libraries in 31DEC14/MACINT/LOAD, where all the AIPS executables are stored. Now when the tasks look for the libraries it appears as though they are in the same directory. Unfortunately this means having to launch AIPS from 31DEC14/MACINT/LOAD, but it works for now. I'll update this if I get around to finding out why the libraries on DYLD_LIBRARY_PATH aren't being found.

The second problem was that the El Capitan upgrade broke the X11 installation I had. I reinstalled X11 from http://xquartz.macosforge.org/landing/, but AIPS still complained. Reading the error message, it looked like AIPS was looking for X11 in /usr/X11R6, whereas I had it installed in /usr/X11. Again I fixed this with a simlink (ln -s /usr/X11 /usr/X11R6), but because of some changes in El Capitan I could not edit /usr even with superuser privileges. I restarted the computer in recovery mode (holding cmd+r) and used the terminal to enable changes to /usr with the command "csrutil disable" and rebooted. I then made the simlink, rebooted into recovery mode and re-enabled protected mode with "csrutil enable" (the steps are detailed at http://stackoverflow.com/questions/32590053/copying-file-under-root-got-failed-in-os-x-el-capitan-10-11/32590885#32590885).

AIPS is now working fine on the upgraded OS. It's likely that these issues will be taken care of automatically in a fresh install of AIPS when Eric Greisen updates the Mac binaries for El Capitan, so hopefully these steps will only be necessary if you're keeping an old version.

Sending an SMS via Intent in Android

Alchemy allows users to donate to their chose charities via text message, which it used to do automatically using Android's SmsManager ...