Using Python 3 to extract files from an encrypted archive with a password

Recently I needed to use python to extract the contents of a password-protected zip archive and I came across a few issues I thought would be good to document.

Python has a build in zipfile library that is really good at handling zip files, but unfortunately has a few limitations when it comes to encrypted zip files. This is how Python3 can be used to extract a file from an encrypted zip archive:


The first issue that I came across was some unclear documentation for the "open" method of zipfile in Python 3. The open method uses the "pwd" argument to pass the password for the file, but in Python 3 you need to convert this to bytes before calling open. Unfortunately, the library only seems to support CRC-32 based encryption  - meaning that the default linux zip encryption will work, but AES will not. I was also unable to get this to work with 7zip and WinZip.

New website design

It had been a while since I updated my website so I found a new template from http://themes.3rdwavemedia.com and redesigned the whole site. I particularly like the github and blog integration with the new theme - hopefully that will encourage me to write more here too, especially as I'm working on the new Alchemy app.

New LOFAR baseline comparison tool

I've written a new tool for comparing the lengths of different baselines to a single LOFAR station. It should be useful when looking for delay and rate solutions with FRING or a similar task. The tool is located at http://www.colmcoughlan.com/lofar.html.

Installing AIPS on Ubuntu 16.04

I've recently installed AIPS (version 31DEC16) on Ubuntu 16.04. Here are my notes on the installation experience. Due to recent compiler work on AIPS this was a much better experience than on recent new versions of Ubuntu. Note that this was a "text" (compile-from-source) install - if you just need to use AIPS you will likely get a faster experience using the pre-compiled binaries available from the binary installation (see http://www.aips.nrao.edu/dec16.shtml for full instructions)


As before, the Ubuntu dependencies are:


        libx11-dev and its dependancies
        x11proto-xext-dev
        libxext-dev
        libxpm-dev
        libncurses5-dev
        libncursesw5-dev
        libbsd-dev
        libedit-dev


I use gcc and gfortran to compile AIPS - you need to specify their location in the AIPS installer (/usr/bin/gfortran is both the fortran compiler and the linker for me, while /usr/bin/gcc compiles for C). I also have trouble with the fortran compiler flags -axWPT and -ip options, so I turn them off both for gfortran and gcc. XAS isn't built using these directives however, so you need to go to Y/SERVERS and edit XAS.SHR to remove the use of the axWPT and -ip flags when compiling the TV server.


A new problem with this combination of Ubuntu and AIPS is an errant "-c" ending up in the link command for making XAS after XAS.SHR is run. This means you will get an error that XAS failed to link, but if you open the makefile at Y/SERVERS/XAS/makefile and remove the "-c" option from the LOCALOPTS_LNX64 variable, then you can just type "make" to correctly make and set up XAS (I did this in a separate terminal while the rest of the installation was proceeding). Make sure the executable xas file has been successfully moved to LNX64/LOAD.


Make sure to add


        sssin        5000/tcp    SSSIN        # AIPS TV
        ssslock        5002/tcp    SSSLOCK        # AIPS TV Lock
        msgserv        5008/tcp    MSGSERV        # AIPS Message Server
        tekserv        5009/tcp    TEKSERV        # AIPS TekServer
        aipsmt0        5010/tcp    AIPSMT0
        aipsmt1        5011/tcp    AIPSMT1
        aipsmt2        5012/tcp    AIPSMT2
        aipsmt3        5013/tcp    AIPSMT3
        aipsmt4        5014/tcp    AIPSMT4
        aipsmt5        5015/tcp    AIPSMT5
        aipsmt6        5016/tcp    AIPSMT6
        aipsmt7        5017/tcp    AIPSMT7


to /etc/services to get AIPS to run properly. To do this you may need to open the file as root.


One last thing - add the aips path to your profile so you can run it straight from the terminal. For me the line was ". /home/colm/aips/LOGIN.SH; $CDTST". The CDTST bit at the end is needed if you intend to compile custom AIPS tasks.

New website

I've made a website, where I hope to include information about some of the work that I'm doing. You can see it at www.colmcoughlan.com

Negative diagonal elements in the covariance matrix returned by numpy.polyfit

I ran into a strange issue fitting a line to a small number of data points using numpy.polyfit that I thought was worth documenting.

I ran a command of the form:

p, cov = np.polyfit(x,y,1,w,cov=True),

where x, y and w were arrays of length 3.

The command returned the correct slope and y-intercept values, however the covariance matrix, cov, had strictly negative diagonal terms. This is apparantly because numpy scales the covariance matrix as described in here.

The scaling applied is a factor such that

factor = resids / (len(x) - order - 2.0)

If, like me, you are making a first order polynomial fit to a dataset of 3 values, the denominator has the effect of multiplying the expected matrix by -1. If I was unlucky enough to have 4 points, it would have thrown bigger errors.

In my case, looking at the results here, I could recover the correct values just by multiplying the matrix by minus one. This is a strange weighting to apply to a small dataset - I assume it makes sense if you have many points and the developers wanted to keep numpy.polyfit consistent.

Fixing AIPS after upgrading to El Capitan on OSX

I recently ran into some AIPS trouble after upgrading from Mavericks to El Capitan. My installation of AIPS complained about X11 and about being unable to find some libraries that normally come with the binary distribution I had actually installed.

The library fix should have been the easiest - AIPS was looking for:

• libsvml.dylib
• libirc.dylib
• libimf.dylib

All of these come with the binary and are located in 31DEC14/MACINT/LIBR/INTELCMP of the AIPS directory (note I'm still using the frozen AIPS from December 2014).

AIPS automatically adds this location to DYLD_LIBRARY_PATH when you run LOGIN.SH, but even though the libraries are on the path, AIPS does not seem to be able to find them when it needs them. I fixed this temporarily by making a simlink for each of the three libraries in 31DEC14/MACINT/LOAD, where all the AIPS executables are stored. Now when the tasks look for the libraries it appears as though they are in the same directory. Unfortunately this means having to launch AIPS from 31DEC14/MACINT/LOAD, but it works for now. I'll update this if I get around to finding out why the libraries on DYLD_LIBRARY_PATH aren't being found.

The second problem was that the El Capitan upgrade broke the X11 installation I had. I reinstalled X11 from http://xquartz.macosforge.org/landing/, but AIPS still complained. Reading the error message, it looked like AIPS was looking for X11 in /usr/X11R6, whereas I had it installed in /usr/X11. Again I fixed this with a simlink (ln -s /usr/X11 /usr/X11R6), but because of some changes in El Capitan I could not edit /usr even with superuser privileges. I restarted the computer in recovery mode (holding cmd+r) and used the terminal to enable changes to /usr with the command "csrutil disable" and rebooted. I then made the simlink, rebooted into recovery mode and re-enabled protected mode with "csrutil enable" (the steps are detailed at http://stackoverflow.com/questions/32590053/copying-file-under-root-got-failed-in-os-x-el-capitan-10-11/32590885#32590885).

AIPS is now working fine on the upgraded OS. It's likely that these issues will be taken care of automatically in a fresh install of AIPS when Eric Greisen updates the Mac binaries for El Capitan, so hopefully these steps will only be necessary if you're keeping an old version.