Tuesday 14 October 2014

Installing LOFAR software on a mac (Mavericks)

I've installed some of the LOFAR data reduction software on a macbook pro running OSX 10.9 (Mavericks). This is probably not the best method in the world to install the LOFAR software, but it works reasonably well and I haven't seen any other documentation for a mac installation so it might be useful to someone.

I roughly followed the same procedure as for Ubuntu, but with a few differences.

First I used macports to download as many of the dependencies I listed here as I could find. The full list of macports I installed is

  binutils @2.24_0 (active)

  boost @1.56.0_1+no_single+no_static+python27 (active)
  bzip2 @1.0.6_0 (active)
  cctools @855_1+llvm35 (active)
  cctools-headers @855_0 (active)
  cfitsio @3.340_0 (active)
  cloog @0.18.2_0 (active)
  cmake @3.0.2_0 (active)
  curl @7.38.0_0+ssl (active)
  curl-ca-bundle @7.38.0_0 (active)
  cython_select @0.1_0 (active)
  db48 @4.8.30_3 (active)
  db_select @0.1_2 (active)
  expat @2.1.0_0 (active)
  fftw-3 @3.3.4_0 (active)
  gcc49 @4.9.1_0 (active)
  gcc_select @0.1_8 (active)
  gettext @0.19.2_0 (active)
  gmp @6.0.0_1 (active)
  hdf5 @1.8.13_0+cxx (active)
  icu @53.1_1 (active)
  isl @0.13_0 (active)
  ld64 @236.3_1+llvm35 (active)
  libarchive @3.1.2_0 (active)
  libcxx @183506_1 (active)
  libedit @20121213-3.0_0 (active)
  libffi @3.1_4 (active)
  libgcc @4.9.1_0 (active)
  libiconv @1.14_0 (active)
  libidn @1.29_0 (active)
  libmpc @1.0.2_0 (active)
  libxml2 @2.9.1_0 (active)
  llvm-3.5 @3.5-r216817_0+assertions (active)
  llvm_select @1.0_0 (active)
  lzo2 @2.06_0 (active).
  mpfr @3.1.2-p10_0 (active)
  ncurses @5.9_2 (active)
  nosetests_select @0.1_0 (active)
  openssl @1.0.1i_0 (active)
  pcre @8.35_0 (active)
  py27-cython @0.21_0 (active)
  py27-nose @1.3.1_0 (active)
  py27-numpy @1.9.0_0 (active)
  py27-pyfits @3.3_0 (active)
  py27-pywcs @1.11-4.8.2_1 (active)
  py27-scipy @0.14.0_0+gcc48 (active)
  py27-setuptools @6.0.2_0 (active)
  python27 @2.7.8_2 (active)
  python_select @0.3_3 (active)
  sqlite3 @3.8.6_0 (active)
  SuiteSparse @4.2.1_3 (active)
  swig @3.0.2_0 (active)
  swig-python @3.0.2_0 (active)
  wcslib @4.23_1 (active)
  xz @5.0.7_0 (active)
  zlib @1.2.8_0 (active)

I then downloaded and compiled FFTW and boost manually. This was required for boost because macports has trouble building boost with the gnu compiler suite and the version of boost built with clang doesn't seem to work with the LOFAR software. I thought something similar was happening with FFTW, but it might have turned out to be a linking error on my behalf - so you can try to link the macports-provided installation of FFTW instead of building your own if you prefer.

The commands I used were:

Boost:

./bootstrap.sh --prefix=/Users/admin/boost --withpython=/opt/local/bin/python2.7 --with-toolset=gcc
./b2

./b2 install

FFTW

./configure --prefix=/Users/admin/fftw --enable-threads
make -j 8
make install
./configure --prefix=/Users/admin/fftwf --enable-float --enable-threads
make -j 8
make install

Issues

There were lots of failures the first time I tried to compile the LOFAR software. Refer to the Ubuntu instructions for the gist of what I was doing. I installed everything LOFAR-related into the /opt/lofar-software directory.

cmake -DBUILD_TESTING=NO -DCMAKE_INSTALL_PREFIX=/opt/lofar-software -DUSE_FFTW3=Yes -DUSE_THREADS=YES -DUSE_HDF5=YES -DUSE_OPENMP=YES -DDATA_DIR=/opt/lofar-software/data -DFFTW3_ROOT_DIR=/usr/local/lib ../..

Installed as usual

export LOFAR_ROOT="/opt/lofar-software"
export PATH="${PATH}:${LOFAR_ROOT}/bin/"
export LD_LIBRARY_PATH="${LOFAR_ROOT}/lib/"
export PYTHONPATH="{LOFAR_ROOT}/lib/python/"

Next is the installation of PYRAP. PYRAP has not been updated in a while and you need to apply the patch indicated here to make sure it installs correctly on a mac (the bug reporter links the patch, the plus lines in the diff file indicate new lines to be added, the minuses lines that need to be taken away). Basically you need to ensure it only tries to build for one architecture. I did this manually by going to the affected setup.py files (you'll spot them if you try to build), but you might be able to do this a lot faster if you know more about patching than me. Then build with

./batchbuild-trunk.py --casacore-root=/opt/lofar-software/ --prefix=/opt/lofar-software/ --python-prefix=/opt/lofar-software/lib/python --extra-root=/opt/local/ --boost-lib=boost_python-mt –universal=x86_64

(You might notice the boost_python-mt library here. That is from the macports boost install (clang) which caused no problems in this step.)

Compiled and installed CASAREST as usual

Added data folder as usual.

I needed to patch some LOFAR files to get them to compile successfully. I now suspect some of these patches might have been unnecessary as I was using incorrect settings.

Necessary patches:

1. LCS/pytools. tconvert.cc needs to have the python.h library included at the start (#include <python.h>.

2. Baselinefitting.cc needs to have sincosf replaced with separate calls to sin and cos (apparently sincosf is not available in OSX's version of the standard math library?).  I commented out the original function and replaced it as follows:


//sincosf(phase_ij_obs - phase_ij_model, &sin_dphase, &cos_dphase);
sin_dphase = sin(phase_ij_obs - phase_ij_model);
cos_dphase = cos(phase_ij_obs - phase_ij_model);

Probably unnecessary patches:

The first three patches are probably redundant if you remember to include the DUSE_SHMEM=OFF compiler option.

1. LCS/common. dlmalloc.c. Comment out reference around Have_usr_include_malloc_h, thus forcing the code to define its own.
2. shmem_alloc.cc comment out def of "union semum".
3. shmem_alloc.h comment out inclusion of malloc.h.

I don't know why these didn't re-occur:

4. image2d.cpp needs to have #include <cstring> added.
5. tickset.h needs to have exp10 functions replaced with pow(10,x) functions.
6. imagewidget.cpp same exp10 patch as above.
7. processcommander.cpp. Add definition for HOST_NAME_MAX (a large long).

Note that the CMAKE/variants/GNU.cmake file needs to be edited to point to the macports compilers as follows:

set(GNU_C         /opt/local/bin/gcc)      # GNU C compiler
set(GNU_CXX       /opt/local/bin/g++)      # GNU C++ compiler
set(GNU_Fortran   /opt/local/bin/gfortran) # GNU Fortran compiler
set(GNU_ASM       /opt/local/bin/gcc)      # GNU assembler

After applying the necessary patches above, the following compilation command should work when called from /build/gnu_opt.


cmake -DCASACORE_ROOT_DIR=/opt/lofar-software/ -DBUILD_SHARED_LIBS=ON -DUSE_OPENMP=ON -DBUILD_PACKAGES="ParmDB Calibration DP3 Pipeline MSLofar LofarFT GSM" -DCMAKE_INSTALL_PREFIX:PATH=/opt/lofar-software/ -DCMAKE_C_COMPILER=/opt/local/bin/gcc -DCMAKE_CXX_COMPILER=/opt/local/bin/g++ -DFFTW3_LIBRARY=/Users/admin/fftw/lib/libfftw3.a -DFFTW3F_LIBRARY=/Users/admin/fftwf/lib/libfftw3f.a -DFFTW3F_THREADS_LIBRARY=/Users/admin/fftwf/lib/libfftw3f_threads.a -DPYTHON_EXECUTABLE:FILEPATH=/opt/local/bin/python2.7 -DF2PY_EXECUTABLE=/opt/local/bin/f2py-2.7 -DUSE_LOG4CPLUS=OFF -DUSE_SHMEM=OFF -DBOOST_ROOT=/Users/admin/boost -Wno-dev ../..

Then run make and make install to finish up.

One final thing to do is to add the following lines to the .profile file in your home directory.

source /opt/lofar-software/lofarinit.sh

export DYLD_LIBRARY_PATH="$LD_LIBRARY_PATH"

This should allow LOFAR commands to be run from any terminal.

Tuesday 7 October 2014

Installing LOFAR software on Ubuntu 12.04

Installing LOFAR software on Ubuntu 12.04

These are some notes on how I installed some LOFAR software on Ubuntu 12.04. They are based on some notes I found on the LOFAR wiki at http://www.lofar.org/operations/doku.php?id=engineering:user_software:ubuntu_12_4.

Install main LOFAR suite
===============================================
N.B. Notes below install to strange directory. Build in one directory, and install everything else into the CASACORE directory (binary)


Dependencies:
libgtkmm-2.4-dev python-matplotlib python-pyfits libatlas-base-dev
mpi-default-bin mpi-default-dev libfreetype6-dev python-setuptools
libxml2-dev libpng12-dev libcfitsio3 libcfitsio3-dev libboost-all-dev
autoconf autoconf-archive autogen automake binutils-dev cmake cmake-curses-gui
cvs doxygen flex gfortran git guile-1.8-dev ipython libblas-dev libblitz0-dev
libboost-all-dev libboost-dev libfftw3-dev libfftw3-doc libgfortran3
libglib2.0-dev libgsl0-dev liblapack-dev liblog4cxx10 liblog4cxx10-dev
libopenmpi-dev libpqxx3-dev libx11-dev mgdiff mpi-default-dev patch pgplot5
python-dev python-numeric python-numpy python-scipy scons subversion-tools
swig bison libbison-dev
tcl tcl-dev tk tk-dev tk8.5-dev tcl8.5-dev
libhdf5-dev (oder: libhdf5-serial-1.8.4 libhdf5-serial-dev)
### "libhdf5-serial" is needed for DAL, it doesn't work with "libhdf5-openmpi"
wcslib-dev liblog4cplus-dev liblog4cplus-1.0-4 cython
###for parmdbplot:
python-sip python-qt4




########## LOFAR software:

Download packages:
##################
  mkdir Downloads
  cd Downloads
  wget ftp://ftp.atnf.csiro.au/pub/software/asap/data/asap_data.tar.bz2

Download and Build Casacore:
############################
  tar -xjvf ../Downloads/asap_data.tar.bz2
(This creates the "data" subdirectory)
  mkdir BuildDir/casacore
  cd BuildDir/casacore
  svn co http://casacore.googlecode.com/svn/trunk source
  mkdir -p build/opt
  cd build/opt
  cmake -DBUILD_TESTING=NO -DCMAKE_INSTALL_PREFIX=/opt/lofar-stuff -DUSE_FFTW3=Yes -DUSE_THREADS=YES  -DUSE_HDF5=YES -DUSE_OPENMP=YES -DDATA_DIR=/opt/lofar-stuff/data  ../../source
make -j12
make install

Install Casacore data:
############################
  cd BuildDir/..
  tar -xjvf Download/asap_data.tar.bz2

Download and Build pyrap
############################
  mkdir /cluster/lofar/BuildDir/pyrap
  cd /cluster/lofar/BuildDir/pyrap
  svn co http://pyrap.googlecode.com/svn/trunk dev-source
  export LOFAR_STUFF_ROOT="/opt/lofar-stuff"
  export PATH="${PATH}:${LOFAR_STUFF_ROOT}/bin/"
  export LD_LIBRARY_PATH="${LOFAR_STUFF_ROOT}/lib/"
  export PYTHONPATH="${LOFAR_STUFF_ROOT}/lib/python/"
  ln -s /opt/soft/lofar-stuff/lib/ /opt/soft/lofar-stuff/lib64
  cd dev-source/
  ./batchbuild-trunk.py --casacore-root=/opt/lofar-stuff --prefix=/opt/lofar-stuff --python-prefix=/opt/lofar-stuff/lib/python

Download and Build casarest
###########################
  mkdir /cluster/lofar/BuildDir/casarest
  cd /cluster/lofar/BuildDir/casarest
  svn co https://svn.astron.nl/casarest/trunk/casarest source
  mkdir build
  cd build
  cmake -DCASACORE_ROOT_DIR=/opt/lofar-stuff -DBUILD_ALL=1 -DCMAKE_INSTALL_PREFIX:PATH=/opt/lofar-stuff ../source
  make -j12
  make install


Download and Build the LOFAR Software
#####################################
Download latest copy of LOFAR software
 mkdir -p build/gnu_opt
 cd build/gnu_opt
 cmake -DCASACORE_ROOT_DIR=/opt/lofar-stuff/ -DBUILD_SHARED_LIBS=ON -DUSE_OPENMP=ON -DBUILD_PACKAGES="ParmDB Calibration DP3 Pipeline MSLofar LofarFT GSM" -DCMAKE_INSTALL_PREFIX:PATH=/opt/lofar-stuff ../../LOFAR/
  make -j12
  make install



Note: I found that the above instructions installed the pyrap python libraries in a separate folder to where LOFAR was looking, so I just found them in lofar-software/lib/python and manually pasted them into the right directory (the one LOFAR regards as PYTHONPATH)

HTML form won't submit (Angular)

 It turns out if you mix normal HTML forms with Angular ones (i.e. using FormsModule) Angular will disable the default behaviour of forms on...