University of Miami

A Data Carpentry Workshop

University of Miami

Coral Gables, FL

Mar 30-31, 2015

9:00 am - 5:00 pm

Instructors: John Gosset, François Michonneau

Helpers: Xu Fei

Workshop Information

This Data Carpentry workshop is aimed at researchers in the life sciences at all career stages. Data Carpentry is designed to teach basic concepts, skills and tools for working more effectively with data.

The workshop will cover the following topics:

How to use spreadsheet programs (such as Excel) more effectively, and the limitations of such programs.
Getting data out of spreadsheets and into more powerful tools (python).
Using databases, including managing and querying data in SQL.
Workflows and automating repetitive tasks, in particular using the command line shell.

The participants will need to bring their own laptops, with the software already installed. Please follow the setup instructions (below) for the list of software to install.

We especially encourage to register those who may be less familiar with the above topics. There is no prerequisite as to what computing skills and knowledge is required.

Data Carpentry is a partnership of several NSF-funded BIO Centers (NESCent, iPlant, iDigBio, BEACON and SESYNC) and Software Carpentry, and is sponsored by the Data Observation Network for Earth (DataONE). The structure and objectives of the curriculum as well as the teaching style are informed by Software Carpentry.

Etherpad

Link to the etherpad for the workshop.

Data sets

Notes from the lessons

Schedule

This schedule is tentative and may change.

Monday	09:00 - 9:30	Introduction
	09:30 - 10:30	Better use of spreadsheets - part 1
	10:30 - 11:00	Break
	11:00 - 12:00	Better use of spreadsheets - part 2
	12:00 - 13:00	Lunch break
	13:00 - 14:30	Tools for manipulating life science data (details TBC)
	14:30 - 15:00	Coffee break
	15:00 - 16:45	Introduction to working with data in SQL
	16:45 - 17:00	Wrap-up

Tuesday	09:00 - 11:00	Introduction to shell - part 1
	11:00 - 11:30	Break
	11:30 - 13:00	Introduction to shell - part 2
	13:00 - 14:00	Lunch
	14:00 - 15:00	Data manipulation using python - part 1
	15:00 - 15:30	Break
	15:30 - 16:45	Data manipulation using python - part 2
	16:45 - 17:00	Wrap-up and feedback

Venue

TBD

Syllabus

Better use of spreadsheets

Formatting data tables in spreadsheets.
Common formatting mistakes by spreadsheet users.
Dates as data.
Basic quality control and data manipulation in spreadsheets.
Exporting data from spreadsheets.
Caveats of popular data and file formats.

The Unix Shell

Files and directories: pwd, cd, ls, mkdir, ...
History and tab completion
Pipes and redirection
Looping over files
Creating and running shell scripts
Finding things: grep, find, ...

Setup

To participate in a Data Carpentry workshop, you will need working copies of the software described below. Please make sure to install everything before the start of your workshop.

Overview

Editor

When you're writing scripts or text, it's nice to have a text editor that is optimized for writing code, with features like automatic color-coding of key words. The default text editor on Mac OS X and Linux is usually set to Vim, which is not famous for being intuitive. if you accidentally find yourself stuck in it, try typing the escape key, followed by ':wq!' (colon, lower-case 'q', exclamation mark), then hitting Return to return to the shell. (This will exit Vim and save the changes you made in the file.)

The Bash Shell

Bash is a commonly-used shell. Using a shell gives you more power to do more tasks more quickly with your computer.

Python

Python is a popular language for scientific computing, and great for general-purpose programming as well. Installing all of its scientific packages individually can be a bit difficult, so we recommend an all-in-one installer (e.g. Anaconda). Regardless of how you choose to install it, please make sure you install Python version 2.x and not version 3.x (e.g., 2.7 is fine but not 3.4). Python 3 introduced changes that will break some of the code we teach during the workshop.

SQL

SQL is a specialized programming language used with databases. We use a simple database manager called SQLite, either directly or through a browser plugin.

Spreadsheet

We'll be using a spreadsheet program. If you already have a spreadsheet program installed, like LibreOffice, Excel or OpenOffice, you can use whatever you already have. If you don't have a spreadsheet program, please download and install LibreOffice.

Everyone needs to install these (regardless of your platform)

Firefox SQLite Plugin

Instead of using sqlite3 from the command line, you will use this plugin for Firefox instead. If you don't already have firefox, install it first. To install the sqlite plugin:

Start Firefox.
Go to the plugin homepage.
Click the "Add Now" button.
Click "Install Now" on the dialog that appears after the download completes.
Restart Firefox when prompted.
To check that it was installed correctly, select "SQLite Manager" from the "Tools" menu. If the menu is hidden, use the "Alt" key near the spacebar to make it appear.

Open Refine

We will use Open Refine to clean up data. Go to their download page and install the stable version for your operating system.

Windows

Editor

Notepad++ is a popular free code editor for Windows. Be aware that you must add its installation directory to your system path in order to launch it from the command line (or have other tools like Git launch it for you). The instructions to modify your path are available online here. Please ask your instructor to help you do this.

Git Bash

Install Git (version control) and a Bash shell for Windows from the msysGit project's homepage. This will provide you with Bash in the Git Bash program.

Python

Download and install Anaconda. Download the default Python 2 installer (do not follow the link to version 3). Use all of the defaults for installation except make sure to check Make Anaconda the default Python.

SQLite

Install the Firefox SQLite browser plugin described above.

Software Carpentry Installer

Other tools used in Data Carpentry have been packaged up by Software Carpentry in an installer. This installer requires an active internet connection.

Download the installer.
Double click on the file to run it.
A command window will open and then close again. The installer downloads and installs a couple of command line programs that we'll need for course.
If you saw it open and then close again, you're set!

Mac OS X

Bash

The default shell in all versions of Mac OS X is bash, so no need to install anything. You access bash from the Terminal (found in /Applications/Utilities). You may want to keep Terminal in your dock for this workshop.

Editor

We recommend Text Wrangler or Sublime Text. Alternatively, you can use nano, which should be pre-installed.

Python

SQLite

sqlite3 comes pre-installed on Mac OS X. Also install the Firefox SQLite browser plugin described above.

Linux

Bash

The default shell is usually bash, but if your machine is set up differently you can run it by opening a terminal and typing bash. There is no need to install anything.

Editor

Kate is one option for Linux users. Alternatively, you can use nano, which should be pre-installed.

Python

We recommend the all-in-one scientific Python installer Anaconda. (Installation requires using the shell and if you aren't comfortable doing the installation yourself just download the installer and we'll help you at the boot camp.)

Download the installer that matches your operating system and save it in your home folder. Download the default Python 2 installer (do not follow the link to version 3).
Open a terminal window.
Type:
```
bash Anaconda-
```
and then press tab. The name of the file you just downloaded should appear.
Press enter. You will follow the text-only prompts. When there is a colon at the bottom of the screen press the down arrow to move down through the text. Type yes and press enter to approve the license. Press enter to approve the default location for the files. Type yes and press enter to prepend Anaconda to your PATH (this makes the Anaconda distribution the default Python).

SQLite

sqlite3 comes pre-installed on Linux. Also install the Firefox SQLite browser plugin described above.