# Welcome at MicroData!
This is the Handbook page of CEU MicroData. If you are here for the first time, welcome to the team! If you are revisiting something or upgrading this page it's at least as good!
The site consists of three main parts:
* **Onboarding** covers the basics to get you ready
* **Tools** includes materials that is useful in our day-to-day work
* **About us** tells you who we are and what we do
This is how this page works: We have lessons designed for you that are going to guide you through some of the most important tools, data and procedures we work with. It is designed to help you throughout your time here at the MicroData. Hopefully it contains information that you will often revisit and learn from. Nevertheless, it definitely doesn't replace learning from your peers. Whenever you run into any questions that a quick google search / Stack Overflow can't answer, make sure that you ask them on Slack. Even better: If you find out the answer you may want to add it here, so that others will find it more easily.
Enjoy both the lessons and your time at MicroData!
> ## License
>
> The content of this Handbook is provided under a Creative Commons Attribution-NonCommercial license (CC BY-NC 4.0). You are free to share and adapt this material under the restrictions described at . Full license documentation is available [here](https://creativecommons.org/licenses/by-nc/4.0/legalcode).
# Code of conduct
This page describes our policies on behavior and scientific integrity. Please read it carefully.
## Behavior
CEU MicroData is a workplace free of any sort of discrimination or harassment. Each member has to comply with the relevant University policies (see the [CEU Code of Ethics](https://documents.ceu.edu/documents/p-1009-1v2106) and the [CEU Policy on Harassment](https://documents.ceu.edu/documents/p-1402-1v2103)). To sum up, you must avoid uninvited or unwelcome physical, verbal or non-verbal conduct directed at an individual's or a group of individuals' supposed or actual nationality, race, ethnicity, language, age, sex, sexual orientation, gender identity and expression, religion, cultural identity, social origin, socioeconomic status, educational level, disability, genetic trait, mental or physical health condition, or any similar ground. We are committed to the principles of free inquiry and free expression, the pursuit of truth wherever it may lead, and to creating an environment that encourages the active exploration and exchange of ideas. We do not intend to impair this freedom. Prohibited harassment, however, is neither protected expression nor the proper exercise of academic freedom.
### Privacy
In general, you should comply with GDPR and the local privacy protection laws. You should ask for explicit consent of each affected person before taking any photos or recordings. You should ask for explicit consent again in case you intend to share data of any kind on a person with anyone else in any way (e.g. posting to social media, sending it to a colleague or a friend etc.).
## Confidentiality
As a rule of thumb consider any information, research data and material to be confidential unless it has already been made publicly available by MicroData. Any personal data or dataset containing personal information is strictly confidential and should be stored on encrypted medium. In case you intend to share any information or data with any person or organization outside MicroData make sure to obtain a written approval of a Principal Investigator first.
## Scientific Integrity
You must avoid any academic misconduct (see details in the Code of Ethics, but basically you should report your results honestly and make proper attributions to other people's research and ideas). You should make sure that any result you report can be reproduced. The best practice is to work in beads. Mistakes are a part of the research process. You will at some point likely make a mistake. The important part is to acknowledge it and to never try to hide it. The integrity of our science is our top priority. Reporting any mistake immediately helps us fixing it sooner and avoiding making similar mistakes in the future.
## Reporting
If you notice or experience any violation of the basic rules outlined above, please notify Miklos immediately. If Miklos is the cause of your concern, please notify Adam. If you have any suggestions on how we should operate or further improve our culture you are always welcome to share it on the meetings or privately with any PI or Project Manager.
# Checklist
To-dos before starting the onboarding process
> ## Prerequisites
>
> Make sure that you have a signed contract. Even though some lessons are useful in general, it is best if you have full access to the microdata resources (e.g. server, slack, github, etc.) first. To gain access to every resource, you need a CEU-issued e-mail address, which you should receive shortly after signing your contract.
A project manager should go through the checklist below to make sure you have access to every resource.
* Add new member to the ceumicrodata Slack. You can follow [these steps.](https://slack.com/intl/en-hu/help/articles/201330256-Invite-new-members-to-your-workspace)
* Send invitation to GitHub team. This has to be done by a GitHub team owner. You can find the owners of the team [here.](https://github.com/orgs/ceumicrodata/people)
* Invite member to Project Management Trello board. Any member can do this. Follow [this guide.](https://help.trello.com/article/717-adding-people-to-a-board)
* Include them in the weekly meeting calendar event (if needed). Any member with access to the event can do this. If unsure, follow [these steps.](https://support.google.com/calendar/answer/37161)
* Add them to the Political Connections dropbox folder (if needed). Any member can do this. If you need help, [follow this guide.](https://help.dropbox.com/files-folders/share/share-with-others)
* Request a CEU-issued e-mail address. Ask a project administrator for help with this.
* Request CEU VPN access. Ask a project administrator for help with this.
* Once the VPN access is ready, request server access to haflinger. Write to the admin Slack channel with the VPN username and an e-mail address that the new member uses.
* Invite them to the microdata stackoverflow (requires CEU e-mail address). To invite, direct them to [this page.](https://stackoverflow.com/c/ceu-microdata) They can auto-join after registering with a CEU-issued e-mail address.
# Setup
This is a comprehensive list of what you are going to need throughout these lessons. To participate in the MicroData workflow, you will need access to the software described below. In addition, you will need an up-to-date web browser. The Carpemtries maintain a list of common issues that occur during installation as a reference for instructors that may be useful on the [Configuration Problems and Solutions wiki page](https://github.com/swcarpentry/workshop-template/wiki/Configuration-Problems-and-Solutions).We will go into details after a quick overview:
> ### Key requirements
>
> * Python 3 installation with pip3 and virtualenv
> * BEAD executable
> * Git installation, Git Bash on Windows
A useful script to install required programs on Linux can be found here: . Let's go through them one by one.
## The Bash Shell
Bash is a commonly-used shell that gives you the power to do simple tasks more quickly.
{% tabs %}
{% tab title="Windows" %}
[Video Tutorial](https://www.youtube.com/watch?v=339AEqk9c-8)
1. Download the Git for Windows [installer](https://git-for-windows.github.io/).
2. Run the installer and follow the steps below:
1. Click on "Next" four times (two times if you've previously installed Git). You don't need to change anything in the Information, location, components, and start menu screens.
2. **Select "Use the nano editor by default" and click on "Next".**
3. Keep "Git from the command line and also from 3rd-party software" selected and click on "Next". If you forgot to do this programs that you need for the workshop will not work properly. If this happens rerun the installer and select the appropriate option.
4. Click on "Next".
5. Select "Use the native Windows Secure Channel library", and click "Next".
6. Keep "Checkout Windows-style, commit Unix-style line endings" selected and click on "Next".
7. **Select "Use Windows' default console window" and click on "Next".**
8. Leave all three items selected, and click on "Next".
9. Do not select the experimental option. Click "Install".
10. Click on "Finish".
3. If your "HOME" environment variable is not set (or you don't know what this is): 1. Open command prompt (Open Start Menu then type `cmd` and press \[Enter]) 2. Type the following line into the command prompt window exactly as shown:
`setx HOME "%USERPROFILE%"`
1. Press \[Enter], you should see `SUCCESS: Specified value was saved.`
2. Quit command prompt by typing `exit` then pressing \[Enter]
This will provide you with both Git and Bash in the Git Bash program.
{% endtab %}
{% tab title="MacOS" %}
The default shell in all versions of macOS is Bash, so no need to install anything. You access Bash from the Terminal (found in `/Applications/Utilities`). See the Git installation [video tutorial](https://www.youtube.com/watch?v=9LQhwETCdwY) for an example on how to open the Terminal. You may want to keep Terminal in your dock.
{% endtab %}
{% tab title="Linux" %}
The default shell is usually Bash, but if your machine is set up differently you can run it by opening a terminal and typing `bash`. There is no need to install anything.
{% endtab %}
{% endtabs %}
## Git
Git is a version control system that lets you track who made changes to what when and has options for easily updating a shared or public version of your code on [github.com](https://github.com/). You will need a [supported web browser](https://help.github.com/articles/supported-browsers/).
You will need an account at [github.com](https://github.com/) for parts of the Git lesson. Basic GitHub accounts are free. We encourage you to create a GitHub account if you don't have one already. Please consider what personal information you'd like to reveal. For example, you may want to review these [instructions for keeping your email address private](https://help.github.com/articles/keeping-your-email-address-private/) provided at GitHub.
{% tabs %}
{% tab title="Windows" %}
Git should be installed on your computer as part of your Bash install (described above).
{% endtab %}
{% tab title="MacOS" %}
[Video Tutorial](https://www.youtube.com/watch?v=9LQhwETCdwY)
**For OS X 10.9 and higher**, install Git for Mac by downloading and running the most recent "mavericks" installer from [this list](http://sourceforge.net/projects/git-osx-installer/files/). Because this installer is not signed by the developer, you may have to right click (control click) on the .pkg file, click Open, and click Open on the pop up window. After installing Git, there will not be anything in your `/Applications` folder, as Git is a command line program. **For older versions of OS X (10.5-10.8)** use the most recent available installer labelled "snow-leopard" [available here](http://sourceforge.net/projects/git-osx-installer/files/).
{% endtab %}
{% tab title="Linux" %}
If Git is not already available on your machine you can try to install it via your distro's package manager. For Debian/Ubuntu run `sudo apt-get install git` and for Fedora run `sudo dnf install git`.
{% endtab %}
{% endtabs %}
After installation, the further steps are recommended:
* Configure installation
* `git config --global user.name "your-full-name"`
* `git config --global user.email "your-email-address"`
* SSH key: for faster usage (no password will be needed afterwards)
* Check if you already have: Is it anything in .ssh? `ls .ssh`
* if no, create a new one and [add to ssh agent](https://help.github.com/en/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent)
* if yes, go to next step
* [Add new SSH key](https://help.github.com/en/articles/adding-a-new-ssh-key-to-your-github-account) to your github account
* Download [Sublime Merge](https://www.sublimemerge.com/download) (recommended git client)
## Text Editor
When you're writing code, it's nice to have a text editor that is optimized for writing code, with features like automatic color-coding of key words. The default text editor on macOS and Linux is usually set to Vim, which is not famous for being intuitive. If you accidentally find yourself stuck in it, hit the `Esc` key, followed by `:`+`Q`+`!` (colon, lower-case 'q', exclamation mark), then hitting `Return` to return to the shell.
{% tabs %}
{% tab title="Windows" %}
nano is a basic editor and the default that we use, it is installed along with Git.
Other editors that you can use are [Notepad++](https://notepad-plus-plus.org/) or [Sublime Text](https://www.sublimetext.com/). **Be aware that you must add its installation directory to your system path.** Please ask your instructor to help you do this.
{% endtab %}
{% tab title="MacOS" %}
nano is a basic editor and the default that we use. See the Git installation [video tutorial](https://www.youtube.com/watch?v=9LQhwETCdwY) for an example on how to open nano. It should be pre-installed.
Other editors that you can use are [BBEdit](https://www.barebones.com/products/bbedit/) or [Sublime Text](https://www.sublimetext.com/).
{% endtab %}
{% tab title="Linux" %}
nano is a basic editor and the default that instructors use in the workshop. It should be pre-installed.
Other editors that you can use are [Gedit](https://wiki.gnome.org/Apps/Gedit), [Kate](https://kate-editor.org/) or [Sublime Text](https://www.sublimetext.com/).
{% endtab %}
{% endtabs %}
## Stata
In MicroData we use the statistical package [Stata](https://www.stata.com/products/)[™](https://korenmiklos.github.io/2020-08-23-eea/license.html), version Stata 17 MP. If you do not have a single user Stata license, please let your instructor know so that they can request one for you. We prefer to use the server version of the Stata. You can start Stata from VNC bash command line by the help of `xstata-mp` command.
If you are newly installing single user Stata on your computer, follow these instructions.
{% tabs %}
{% tab title="Windows" %}
1. Go to
2. Log in using your username and password.
3. Download and launch the installer: SetupStata17.exe version 64.bit.
4. Once the installation is done, start Stata from the Start Menu. The first time you do this, you will have to activate your licence.
5. Choose version StataMP.
6. Enter the serial number provided and press enter.
7. Enter the code and press enter.
8. Enter the authorization and press enter.
9. It should return “Good. The serial number, code, and authorization make sense. Shall we continue?” Type Y and press enter.
10. When it asks for the first line, it should say “CEU”.
11. When it asks for the second line, it should say “CEU”.
12. It will ask for confirmation. Type “Y” and press enter.
{% endtab %}
{% tab title="MacOS" %}
1. Go to
2. Download and launch the installer: Stata17.dmg
3. Once the installation is done, start Stata from the Start Menu. The first time you do this, you will have to activate your licence.
4. Choose version StataMP.
5. Enter the serial number provided and press enter.
6. Enter the code and press enter.
7. Enter the authorization and press enter.
8. It should return “Good. The serial number, code, and authorization make sense. Shall we continue?” Type Y and press enter.
9. When it asks for the first line, it should say “CEU”.
10. When it asks for the second line, it should say “CEU”.
11. It will ask for confirmation. Type “Y” and press enter.
{% endtab %}
{% tab title="Linux" %}
1. Go to
2. Download Stata17Linux64.tar.gz.
3. Open a terminal and navigate to the directory where your downloaded file is located (e.g. cd \~/Downloads/)
4. Get superuser rights (sudo su)
5. Create a new directory (e.g. mkdir stata\_install)
6. Move the downloaded file to this new directory (mv Stata17Linux64.tar.gz. stata\_install/)
7. Enter the directory (cd stata\_install)
8. Extract the installation files using tar xzf Stata17Linux64.tar.gz
9. Create a directory for your stata installation (mkdir /usr/local/stata17)
10. Navigate to the stata directory (cd /usr/local/stata17)
11. Start the installation by executing the extracted install file (e.g. /home/username/Downloads/stata\_install/install)
12. Choose version StataMP.
13. Whenever the installer asks if you want to proceed type “y” and press enter
14. Once the installation is done, type ./stinit to activate your licence
15. Whenever it asks you if you want to continue, type “Y” and press enter
16. Enter the serial number provided and press enter
17. Enter the code and press enter
18. Enter the authorization and press enter
19. It should return “Good. The serial number, code, and authorization make sense. Shall we continue?” Type Y and press enter.
20. When it asks for the first line, it should say “CEU”
21. When it asks for the second line, it should say “CEU”
22. It will ask for confirmation. Type “Y” and press enter.
23. Try to start stata by ./xstata. If it gives you the following error message (./stata: error while loading shared libraries: libpng12.so.0: cannot open shared object file: No such file or directory), continue with the steps below:
`Issue the following commands one by one in your terminal window: apt-get install zlib1g-dev wget http://mirrors.kernel.org/ubuntu/pool/main/libp/libpng/libpng12-0_1.2.54-1ubuntu1_amd64.deb dpkg -i libpng12-0_1.2.54-1ubuntu1_amd64.deb`
{% endtab %}
{% endtabs %}
## Python 3
If you use Ubuntu 20.04 LTS or newer, you are good to go. Otherwise you should visit [Python's official page](https://www.python.org/downloads/) and download the latest python version for your operating system. Always follow the installation guide on Python's website.
## Bead
1. (windows only:) Install Git Bash
2. Install python if not already installed.
Latest release depends on Python 3.8.5., Python 3 only (requires at least 3.7). You can check the version of your Python from the terminal with `python -V`.
1. Download latest version from
you will need only the platform specific binary: `bead`
2. Put the downloaded file in a location, that is on the PATH
* `$HOME/bin` (single-user, laptop, desktop, traditional location)
* `$HOME/.local/bin` (single-user, laptop, desktop, new XDG standard?)
* `/usr/local/bin` (system, servers, multi-user)
3. (linux and mac only): make the file executable
E.g. the following commands would install version v0.8.1 (latest release at the time of writing) on linux and mac:
```
# ensure user bin directory existst (for user specific scripts)
mkdir -p ~/.local/bin
# download bead
cd ~/.local/bin
curl -sLo bead https://github.com/e3krisztian/bead/releases/download/v0.8.1/bead
# make executable
chmod +x bead
```
At the end, you can check whether your installation was successful by typing `bead version` to the terminal, then it should show v0.8.1.
(source: )
## Make
Linux and Mac users don't have to install it. Windows users should follow the next steps:
1. Install chocolatey
We are going to install make through the package manager chocolatey, therefore first we have to install chocolatey. Follow the instructions on: [chocolatey website](https://chocolatey.org/install)
* search for powershell in the windows start menu and run it as administrator
* run the command in the field under the second point (install with powershell.exe)
2. Install gnu-make
* type in the powershell `choco install make`.
# Using the terminal
> ### Overview
>
> Questions
>
> * How to navigate in the terminal?
> * How to execute commands and start applications?
> * What are some of the most important command line tools?
>
> Objectives
>
> * Understanding how the command line interface works.
> * Learn the most important command line operations.
## The Command Line Interface
You probably mostly use your computer via something called a GUI, or Graphical User Interface. Practically any modern operating system comes with a GUI. It is the GUI that, for example, allows you to instruct your computer to run applications by double clicking on an application's icon instead of issuing your commands in writing.
However, it is not the only way to use your computer. Before the quick rise in computing capacity and memory, graphical interfaces were impossible to develop. Originally computers are instructed via written commands in something called a CLI, or Command Line Interface. Practically anything that is feasible using a GUI can be done using the CLI, even though it might be much more difficult. Then why bother with using the CLI at all? - you might ask. There are two main reasons for familiarizing yourself with the CLI at MicroData:
1. The servers that we use have very limited graphical support, and you will mostly use an Ubuntu terminal when you work on the servers.
2. Most freshly developed tools for data analysis simply do not have a GUI. It takes a lot of resources to develop GUIs, and most open source developers focus their attention to the core performance of the tools instead of a GUI. If you want to use these, you have to familiarize yourself with the CLI of your computer.
### How are commands interpreted?
Every operating system comes with a command line interface. The CLI, just like the GUI is, however, different on each operating system. Most modern Windows operating systems use a CLI called Windows PowerShell, while Unix-like systems like MacOS, Ubuntu and other Linux distributions usually use a CLI called Terminal.
It is not only the name of the CLI that differs. The commands that you issue in the CLI have to be interpreted by the computer (translated to a long list of ones and zeros) in order to execute your commands. The way that commands are translated also differs across operating systems. Without going into too much details, it practically depends on something called the command line interpreter, which is a software that does this translation for you. On Windows, this is called cmd.exe, while on most Unix-like systems it's called Bash. The main problem with having mulitple CLIs and command line interpreters is that they understand different commands and translate these commands differently. Something that works on Ubuntu will very likely not work on Windows and vice-versa.
However, as mentioned before, the command line interpreter is just a software. Some of these interpreters have versions available for many different operating systems. The most widely used interpreter is probably Bash. If you are using MacOS or Ubuntu, you are good to go, you already have Bash installed on your computer. If you work with Windows, then you have already installed Git-bash, which, as it's name suggests, is a Bash interpreter. You will have to use this instead of the default Windows CLI throughout this episode.
### The Terminal
The CLI that we are going to use is called the Terminal (on Unix-like systems) or Git-bash (on Windows). From here on, they both will be referenced as the Terminal. If you open a Terminal window, you are supposed to see an almost-blank window with something like this written on it:
> `johndoe@haflinger:~$`
It should be followed by a blinking cursor. This is called the command prompt. It tells you some important information about where you are currently working. It is structured the following way:
> `username@machine:current_directory$`
These are possibly the most important pieces of information that you need to be aware of when using the CLI. What do they tell you?
1. username: This is your username on your computer. In most cases it will not change. In many systems you can switch to a user called root, that can do anything on the computer that a standard user cannot. You should only see your username there, if it changes you should close the Terminal and start a new session unless you are really sure about what you are doing.
2. machine: This is separated from the username by an `@` sign in most cases. It will change for example if you log in to the server. You will be able to control the MicroData servers using the Terminal as well. It is important to know whether you are controlling your own computer or the server in the Terminal window, always make sure that you are working on the proper machine.
3. current\_directory: This is usually separated from the name of the machine by a colon. In many operating systems you have something called your home folder or user folder. It is usually referenced in the Terminal by the `~` sign. If you change the working directory (which will be discussed in a second), it will change accordingly. For example on an other machine and in a different folder it might be something like this:
> `johndoe@johndoe-DELL:~/Documents/onboarding/_episodes`
In order to use your computer via the Terminal, you will have to type commands and press enter. They will be executed one-by-one. The rest of this episode will be about the most important commands in the Terminal.
## The most important Terminal commands
Terminal commands can be executed by pressing enter after typing them. The general structure of a command is the following: `command `. Some commands work by themselves, while others require arguments (for example if you want to change the working directory, you have to specify the new working directory). Positional arguments always have to be specified, while optional arguments are, as their name suggests, optional. You can almost always get a detailed explanation on the positional and optional arguments by opening up the manual of the command by executing `man ` or by calling the command with its help optional argument by ` --help` or `help `
You can find the most commonly used commands with a short description below by categories. If you prefer, you can check out [this Carpentries page](https://datacarpentry.org/shell-economics/), which contains a more detailed walkthrough for each command.
### Navigation
* `pwd` returns the path to the current working directory. In most cases this is part of the command prompt, however, if you are deep down in the folder structure, the command prompt will only display a few parent directories.
{% tabs %}
{% tab title="Bash" %}
```bash
$ pwd
```
{% endtab %}
{% tab title="Output" %}
```bash
~/Documents/GitRepos/CEU_MD_Onboard/onboarding/
```
{% endtab %}
{% endtabs %}
Now, we are at the *\~/Documents/GitRepos/CEU\_MD\_Onboard/onboarding/* after typing the `pwd`. (Don't worry about the tilde ("\~") sign. You are going to learn about it in a minute.
* `cd` changes the working directory. It has a positional argument, which is the target directory. The target directory can be either given as an absolute path or a relative path.
```
#absolute path
$ cd /srv/sandbox/
#relative path
$ cd ../../srv/sandbox/
```
In relative paths you can reference the parent directory by two dots `..`, thus if you want to go to the parent directory, you should issue `cd ..`. Some more commonly used `cd` commands:
```
#Going to your home directory
$ cd
#going to the root folder
$ cd /
#going up to the parent directory
$ cd ..
#going to the Desktop directory in the home directory
$ cd ~/Desktop
```
The tilde ("\~") character in the last command is a shortcut for indicating the home directory.
* `start` will open a file in the default application associated with it on **Windows** (aka your default application on Windows).
* `open` will open a file in the default application associated with it on **MacOS** (aka your default application on MacOS).
* `xdg-open` will open a file in the default application associated with it on Ubuntu and many other **Linux** distributions (aka your default application on any Linux distros).
The example is opening a picture in the default picture viewer.
```
#On Windows
$ start my-picture.png
#On MacOS
$ open my-picture.png
#On (most) Linux distros
$ xdg-open my-picture.png
```
### File System Exploration
* `ls` lists the content of the current working directory. It has a wide set of optional arguments that you can combine to get a listing you prefer. A few of these are:
* `-l` will give you the list of files with one file in one line, and it will include additional information on file permissions, file owners, file size and modification date.
* `-a` will list all files and folders including hidden ones.
* `-S` will sort the files by size in descending order before listing them.
* `-t` will sort the files by modification time (newest first) before listing them.
* `-r` will revert the order of files before listing them.
* `-h` will display the file sizes in a human-readable format.
* You can combine these options, so for example `ls -ltrha` will list all files, including hidden ones, in a list where one file will be one line, and the oldest file will be the first (notice the revert option, that is why it's not the newest) and file sizes will be human-readable.
```
$ ls -ltrha
```
* `less` will show you the content of a text file. It has one positional argument, the text file. It's worth noting that it can be any text file, for example `.py` python codes can be viewed as well as `.csv` data files.
```
$ less trial.py
```
You can scroll up and down using the arrows on your keyboard and exit by pressing `q`.
FIXME: Adding a table of useful `less` associated keyboard combos
* `file` will determine file type. In fact, one of the common ideas in Unix-like operating systems such as Linux is that “everything is a file.”
{% tabs %}
{% tab title="Bash" %}
```bash
$ file picture.jpg
```
{% endtab %}
{% tab title="Output" %}
```
picture.jpg: JPEG image data, JFIF standard 1.01
```
{% endtab %}
{% endtabs %}
### Files and Directories Manipulation
* `cp` copies a file. It has two positional arguments, the source file and the target file.
```
#Copy a file to the parent folder
$ cp my-file.txt ../my-file.txt
#Copy a folder to the parent folder by using -R recursive option
$ cp -R my-folder/ ../
```
* `mv` moves a file or folder. It has two positional arguments, the source and the target path.
```
#Move a file to the parent directory
$ mv my-file.txt ../
#Move the directory "my-folder" and its content to the parent directory
$ mv my-folder ../
```
It is also the way to rename files.
```
#"old-filename.txt" will be renamed to "new-filename.txt"
$ mv old-filename.txt new-filename.txt
```
* `mkdir` creates a new folder in the current working directory. It has a single positional argument, which is the name of the new folder.
```
$ mkdir my-new-folder
```
It can also have multiple optional arguments if you wish to create multiple folders.
```
$ mkdir dir1 dir2 dir3
```
It is even possible to create nested folder structures by providing the `-p` argument.
```
$ mkdir -p dir4/dir5/dir6
```
* `rmdir` removes an empty directory. It's only positional argument is the folder to be removed. It only removes empty folders, you need to delete it's content first.
* `rm` removes a file. It has a positional argument, which is a list of files to be removed. You can give multiple files separated by spaces to remove.
```
#Remove a single file
$ rm my-text.txt
#Remove multiple files
$ rm my-text.txt my-data.dta
```
**IMPORTANT**: If you remove a file by `rm` it will be **permanently** deleted, be careful with it!
* Recursive `rm` removes a directory with all of its subdirectories and files. It can be accessed with the `-r` optional argument.
```
$ rm my_folder/ -r
```
**IMPORTANT**: Recursive `rm` will remove the directory with all of it's content **permanently**.
* Recursive forced `rm` removes a directory with all of its subfolders and files even if the files are read-only. It can be executed using the `rf` optional argument.
```
$ rm my_folder/ -rf
```
**IMPORTANT**: deletion is **permanent**. You should not use it in general, it is only a last resort. The presence of read-only files strongly suggests that they should not be deleted using `rm`, but in some other ways (e.g. `bead nuke` in case of beads). There are some cases when this is useful, but use it with care and only if it is unavoidable.
* Some useful shortcuts which could be used with the some of the above-mentioned commands:
* `?` , a question mark can be used to indicate "any single character".
* `*`, an asterisk can be used to indicate "zero or more characters".
```
#Instead of using
$ cat test_1.txt test_2.txt test_3.txt
#Better usage is
$ cat test_?.txt
#An even shorter solution is
$ cat test_*
```
You are going to learn about `cat` command in the next section.
### Redirection and some other useful commands
* `cat` will print the content of files on your terminal screen. It's positional argument is a file list separated by spaces.
```
$ cat my-code.py
```
If you specify multiple files, it will con`cat`enate (its name comes from here), meaning that linking together and output each of them, one after the other, as a single block of text as it could be seen at the end of the **Files and Directories Manipulation** section.
* `head` and `tail` shows you the first and last few lines of a text file, respectively. It has one positional argument, the text file.
```
$ head my-data.csv
$ tail my-data.csv
```
* `echo` will print the value of its argument on your terminal screen.
{% tabs %}
{% tab title="Bash" %}
```bash
$ echo hello world!
```
{% endtab %}
{% tab title="Output" %}
```
hello world!
```
{% endtab %}
{% endtabs %}
* Some, other useful commands and shortcuts:
* `>` redirection operator will redirect standard output to another file instead of the screen. However, using this will *ALWAYS* overwrite the content of that file
* `>>` redirection operator will do the same as `>` does, **BUT** it will **not** overwrite the output file. Instead, it will append the content of that file.
* `|` pipeline operator connects the ouput of one command with the input of a second command.
* `sort` will sort lines of text.
* `uniq` will report or omit repeated lines.
* `wc` will print newline, word, and byte counts for each file. `wc -l` only counts the number of the lines.
* `cut` will print selected parts of lines from each file to standard output.
* `history` will display previous commands typed in the command line.
* `nano` will open the Nano text editor.
* `grep` will print lines matching a pattern.
For more information, please consult the **Chapter 6 - Redirection** in "The Linux Command Line" book.
### File Permissions
For concise description, please visit the *File Permissions* section of the following link:
## Useful resources for learning Terminal:
* Introdutction to the Command Line for Economics:
* The Unix Shell:
* Official Ubuntu tutorial:
* The Linux Command Line by William Shotts: (freely available)
> ### Key Points
>
> * The computer can be controlled using a GUI or a CLI.
> * Anything that you can do using the GUI can be done using a CLI.
> * In the CLI, you have to issue commands with possibly some positional and optional arguments.
# Git and GitHub
A quick introduction
> ### Overview
>
> Questions
>
> * What are the basic terms used by version control systems?
> * Which files are contained within the .git directory?
> * How to install git?
> * How does the basic collaborative workflow look like?
> * What are some of the most important git commands?
>
> Objectives
>
> * Understand the basic terminology of Git and Github
> * Install and setup git
> * Understand the collaborative workflow and commit message etiquette
> * List some of the most useful commands that can be easily accessed in your everyday work
## Version Control Basics
There are various Version Control Systems such as:
* [Git](https://git-scm.com/)
* [Subversion](https://subversion.apache.org/)
* [Mercurial](https://www.mercurial-scm.org/)
A version control system can be either:
* centralized - all users connect to a central, master repository
* distributed - each user has the entire repository on their computer
### Terminology
#### Version Control System / Source Code Manager
A version control system (or source code manager) is a tool that manages different versions of source code. It helps to create snapshots ("commits") of project files, thereby, supporting the tractability of a project.
#### Repository / repo
A repository is a directory which the version control system tracks and should contain all the files of your project. Besides your project files, a repository contains (hidden) files that git uses for configuration purposes. Git, by default, tracks all your files in a repository. If there are files you do not wish to track, you can include them in the manually created .gitignore file.
Repositories can be located either on a local computer or on the servers of an online version control platform (such as Github).
#### Staging Area / Staging Index / Index
Before committing changes to your project code, the files you want to snapshot need to be added to the Staging Index. Changes to these files can be captured in a commit.
#### Commit
A commit is a snapshot of the files that are added to the Staging Index. Creating a commit can help you to save a particular version of your project. When committing changes, you should also include a commit message that explains the changes of the project files since the previous commit. Therefore, commits track the evolution of your project and allows you to see the changes from one commit to another. It is also useful when experimenting with new code, as git makes it possible to jump back to a previous commit in case your code changes do not work out as planned.
#### SHA
A SHA("Secure Hash Algorithm") is an identification number for each commit. It is a 40-character string composed of characters (0–9 and a–f) such as `e2adf8ae3e2e4ed40add75cc44cf9d0a869afeb6`.
#### Branch
A branch is a line of development that diverges from the main line of development. It further allows you to experiment with the code without modifying the line of development in the master branch. When the project development in a branch turns out successful, it can be merged back to the master branch.
#### Checkout
Checkout allows you to point your working directory to a different commit. Therefore, you can jump to a particular SHA or to a different branch.
#### .Git Directory Contents
The .git directory contains:
* config file - it stores the configuration settings
* description file - file used by the GitWeb program
* hooks directory - client-side or server-side scripts can be placed here to hook into Git's lifecycle events
* info directory - contains the global excludes file
* objects directory - stores all the commits
* refs directory - holds pointers to commits (e.g "branch" and "tag")
### Git workflow
#### Making changes
* `git status`, `git init` (store all changes you commit in this folder), `git add` (except `.gitignore`)
* `git commit` often, to avoid conflict
* commit message should be: If applied, this commit will “your message” (eg. `git commit -m "change label names"` or "update README").
* commit message could also refer to Issues (eg. “close #3”) (issues are like to-do-s)
* `git log`: view the history of commits you’ve made
#### Collaborate with others
In practice, it is good to be sure that you have an updated version of the repository you are collaborating on, so **you should git pull before making your changes**. The basic collaborative workflow would be:
* update your local repo with `git pull origin master`,
* make your changes and stage them with `git add`,
* commit your changes with `git commit -m`, and
* upload the changes to GitHub with `git push origin master`
It is better to **make many commits with smaller changes** rather than of one commit with massive changes: small commits are easier to read and review.
* fork: copy to origin / clone: copy remote repo to create local repo / pull: copies changes from a remote repository to a local repository.
* .gitignore (eg. large files OR on Mac hidden .DS\_Store files OR data in ss-descriptives)
#### Branching, conflicts
Note, if someone pushes a commit to GitHub before you push your changes, you’ll need to integrate those into your code (and test them!) before pushing up to GitHub.
* `git diff` displays differences between commits.
* `git checkout` recovers old versions of files / `git revert` backs out commit
Branching
* `git branch` creates a new branch
* `git checkout` switches to a different branch
Merging
* `git merge` merges other\_branch into the current branch
**Conflicts occur when two or more people change the same file(s) at the same time.** The version control system does not allow people to overwrite each other’s changes blindly, but highlights conflicts so that they can be resolved.
* **Push**: You will not be able to push to GitHub if merging your commits into GitHub’s repo would cause a merge conflict. Git will instead report an error, telling you that you need to pull changes first and make sure that your version is “up to date”. Up to date in this case means that you have downloaded and merged all the commits on your local machine, so there is no chance of divergent changes causing a merge conflict when you merge by pushing.
* **Pull**: Whenever you pull changes from GitHub, there may be a merge conflict! These are resolved in the exact same way as when merging local branches: that is, you need to edit the files to resolve the conflict, then add and commit the updated versions.
### Git LFS
GitHub recommends repositories remain small, ideally less than 1 GB, and less than 5 GB is strongly recommended. Smaller repositories are faster to clone and easier to work with and maintain. Individual files in a repository are strictly limited to a 100 MB maximum size limit, and git gives a warning for updating files larger than 50 MB. Git Large File Storage (LFS) is a useful Git extension if someone has to version large files—even those as large as a couple GB in size—with Git. It replaces large files such as audio samples, videos, datasets, and graphics with text pointers inside Git, while storing the file contents on a remote server like GitHub.com or GitHub Enterprise. You can download the extension here: . Furhter informations about installation for Mac/Windows/Linux can be found here: .
To decide whether you need Git LFS or not, you can type in the command line: `find . -type f -size +1M`, it finds files greater than 1 MB in the given folder. Large files make fetching and pulling quite slow, so it is recommended to `git lfs install` in the repo with files larger than 1-10 MB size (depending on the project) and then `git lfs track` the given large files. For power users: `find . -size +1M | xargs -d '\n' git lfs track \"{}\"` finds and then tracks all files greater than 1 MB in a given repository. LFS tracking modifies the .gitattributes file, so do not forget to commit that as well.
### Useful commands
| Code | Short description |
| ----------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| `git init` | Initialize local git repository |
| `git status` | Check the status of git repository (e.g. the branch, files to commit) |
| `git add` | Add files to staging index |
| `git add .` | Add all modified files to staging index |
| `git commit -m"Text"` | Commit changes with commit message |
| `git log` | Check git commits specifying SHA, author, date and commit message |
| `git log --oneline` | Check git commits specifying short SHA and commit message |
| `git log --stat` | Check git commits with additional information on the files changed and insertions/deletions |
| `git log -p` | Shows detailed information on lines inserted/ deleted in commits |
| `git log -p --stat` | Combines information from previous two commands |
| `git log -p -w` | Shows detailed information on commits ignoring whitespace changes |
| `git show` | Show only last commit |
| `git show