Climate Change: Part 1… Rainfall:– Is it declining??

I recently did some of Coursera’s data science modules as way to sharpen my skills in big data. The modules use R to process the data and do the statistical analysis. My goal is to use this knowledge, I will try to analyze Tobago’s precipitation and temperature for the period 1980 to 2015. The data can be found at the Trinidad & Tobago Meteorological Service. This analysis will be in two parts: Rainfall trend and Temperature trends.

For the past few (10?) years Tobago have been experiencing rather hash drought like conditions. Each year, the Water and Sewerage Authority (WASA) has implemented its Water Supply and Conservation plans in order to ensure a supply. Some of these included water rationing, a ban on watering plants or washing cars with a hose.

As a farmer, I felt the effects of the heat and lack of rain fall during the wet/rainy season. I had to use pipe borne (via a tank) water almost everyday.  Fortunately, I was able to harvest my crops though I believe the yield would have been higher if the rain fall was “normal”.

The main reason WASA has given for the short fall in water is that rain fall has been steadily declining over the years.  While the public believe WASA did not plan for increase usage by the population. So which is true?  No planning for increased usage  or decrease rainfall over time. Hey, maybe both!.  WASA has not given any “numbers” to back up their assertion. While there has been a marked increase in usage due to a higher number of visitors to the island especially during July/Aug months.

So is WASA’s reason for the short fall true?

Read more ›

Git — branching and merging

Branching and merging show the real power of git.

Takeaways:
HEAD points to the active branch
Master is the name given to the pointer of the default/main branch and points the last commit

Types of branches:
ahead — this branch is in front of the Master branch and is created with git branch branch-name master
divergent — this branch is an offshoot of the Master branch and is created with git branch branch-name (Note no master keyword)

To switch active branch: git checkout branch-name

Read more ›

Git version control system

I was given a Bertelsmann Technology Scholarship from Udacity. I am very impressed with its overall course structure and content. There are basically two parts: version control (using Git) and cloud computing (using AWS).

As someone with experience as a SysAdmin, I can appreciate what a godsend the creation of Source Control Managers (SCM)/Version Control System (VCS). Git is the most popular VCS. GitHub being the most popular application of Git. A VCS is an efficient way of managing code changes. Back in the day, anytime changes were made to a script (code), a backup copy was made before the changes. This became a rather tedious and complicated process to manage.

At a high level, Git can be broken into three parts, work area, stage area, and storage area.

Read more ›

How does your computer resolve cnn.com in the browser

I was recently asked this question as a follow-up to the interview question “what happens when you type cnn.com in the browser?”. The answer to the main question is on a high level, i.e. starting with checking the browser’s cache for the domain name. If it does exist then it uses that, if it’s not found in the cache to checks the router’s routing table… etc.

So when I was asked to drill down a bit further; how is the domain name resolved on the local computer? I responded that it checks the browser’s cache. Wrong!! It turns out that domain lookups are stored in a DNS cache which is a temporary database maintained by the operating system. So this is the first place it looks in order to resolve the domain name.

For more information click here

Bash basics example

List the number of lines in all of the csv data files.
wc -l *.csv

Next, remove the line containing the word “total”.
wc -l *.csv | grep -v total

Display the line with fewest number of lines
wc -l *.csv | grep -v total | sort -n | head -n1

Store results in a file
wc -l *.csv | grep -v total | sort -n | head -n1 > results.csv

Bash basics 2

Summary:
head and tail select rows,
cut selects columns, and
grep “phrase” selects lines that contain “phrase“.
Note: grep -v: invert the match, i.e., only show lines that don’t match

** Writing to an output file
command > output_file

** To combine commands, use the pipe (|) — The pipe symbol tells the shell to use the output of the command on the left as the input to the command on the right.
head -n5 myfile.csv | tail -n3
cut -d, -f2 filename.csv | grep -v col_name — this passes the 2nd column of the file to grep which inverts (shows only the lines that do not match) data.  so the head row is removed

Read more ›

Bash basics

Environmental variables
— are in all caps. E.g HOME, USER
— the set command displays all the environmental variable
Getting the value of an environmental variable:
1. echo $HOME — note the $ as the first character
2. set | grep HISTFILESIZ

Shell (local) variables:
— simply assign a value to a name. E.g. name=Chris Note NO SPACES around the equal sign!!
— To display variable content: echo $name
data_file= data/somefile.csv
head -n 1 $data_file — displays the first line in the file

Read more ›

Python Variable

Variable Assignment:

n = 300 — python creates an object of type integer in memory and assigns n as a pointer to it’s location.
m = n — python DOES NOT create a not integer object, but assigns m as a pointer to the same location.
n = “hello” — n no longer points to the location of 300 but now points to the location of the string object; m — continues to point to the location of 300
m = 40.3 — m now points to the location of the float type object; 300 is orphaned and there is no access to it.

Note: the id() method can show you the memory location of the object

Read more ›

Python Files

Use the open() method to access the contents of a file.
example:
FH = open(“filename”, “mode”) –> returns a file object that can be used to manipulate the file’s content.
More efficient way:
with open(“filename”, “mode”) as FH: –> returns a file object
FH.read()

NOTE:
Can loop over the file object:
for line in FH:
print(line)

Read more ›

Python files (CSV data)


Libraries:
CSV and JSON

CSV Methods:
reader() –> returns a reader object. Each row read from the csv file is returned as a list of strings. No automatic data type conversion is performed.
DictReader() –> returns a writer object responsible for converting the data into delimited strings
writer() –>

CSV Example:

Sample of the data:
datetime,host,src,proto,type,spt,dpt
2013-03-03 21:53:59,groucho-oregon,1032051418,TCP,,6000,1433

import csv

count = 0

with open(“filename”) as FH:
   datareader = csv.reader(FH, delimiter=’,’) # returns an object
      for row in datareader: # each row is a list
         if count == 0: # header row
            count += 1
         elif count > 0 and count < 5:
            data_time = row[0]
            print(data_time.split(‘,’))
            count += 1
         else:
            break

   

Kubernetes and Docker

This post is where I plan to document my learning and experiences deploying both Kubernetes and Docker for managing Jupyter notebooks in an educational setting.

Kubernettes: is defined as “a container orchestration and management tool for automating the deployment and monitoring of containerized applications. In my case Docker containers.

Read more ›

Top