Today, I finally got around to doing the R script that will daily mine social media data off Twitter about conversations around President Buhari, Acting President Osinbajo and whatever is trending in Nigeria twittosphere. Will keep it running to do fun or even commercial projects in and beyond 2019 (election year).

The interesting challenge there was to schedule this to run daily, saving the tweets in a CSV file incrementally.

I found two ways online -- the familiar Windows Task Scheduler way and using taskscheduleR

I already use Windows Task Scheduler for a couple of Python scripts and familiar with the setup, so I decided to go along with it.

It is straightforward to use. I have broken the entire process into two steps:

Step 1: Create a Batch file to run the R script
Create an empty text file and save it as .bat file. Then get the executable path of your R installation. Mine is C:\Program Files\Microsoft\MRO-3.3.1\bin\R.exe

Also get the path of your R script. 

Now set up the .bat file like I did mine below

@echo off
"C:\Program Files\Microsoft\MRO-3.3.1\bin\R.exe" CMD BATCH "C:\Users\Michael Olafusi\Documents\TwitterScrapper.R"

Step 2: Create the Scheduler Task
Launch the Task Scheduler (just search for it if you use Windows 10, 8 and 7).

In the Task Scheduler window, at the right pane, click on Create Task.

And follow my screenshot guide below.

And that is it!

And that's how I scheduled the R script to daily mine tweets about Buhari, Osinbajo, Biafra and what's trending in Nigeria.

I intend to do a trend sentimental analysis on them going into the Election days in 2019.

I am now doing more live projects with R and Python, and at the moment I am past deadline working on a big client project using R.

image: neosit.com

Due to the nature of the project and as I am not the only party involved in executing the project, I cannot go into the specifics.

The good news, though, is that I am done with the core studying of the financial application of R and Python to investment and stock trading (which is a big area of interest to me). I am done with reading the main materials I got for that learning and I have the needed concepts and algorithm toolbox to dive fully into the practical side of it all. The only problem I have is with integrating with a live trading account. The highly recommended Interactive Brokers trading platform does not allow registration for Nigerians. I am trying to see if I can get a waiver or find an alternative platform that allows programmatically connecting to their APIs for automated trading and analysis. The good news in it all? It is that I will be sharing most of my real world adventures in that sphere and other areas I find practically beneficial to deploy R and Python codes.

I am already working on a daily triggered Python script that will warehouse some interesting social media data for me that will be of use for interesting Naija-relevant analysis. And maybe even predicting some nationally important events. Especially as the 2019 campaign trains are taking off.

If you have ideas you will like me to consider incorporating in my adventures, please do share! Thanks.

R recognizes four main data types:

1. Numeric values: These are number values which can have decimal parts. Examples are 55, 27.8 and 100.255
2. Integer values: These are number values with no decimal parts. To differentiate them from Numeric values, when manually inputting them in R you append the number with the letter "L". So you'll write 5 as 5L, 10 as 10L and 50 as 50L to make R recognize them as Integer values rather than treat them as Numeric values.
3. Character values: These are text values. You surround them with quotes when manually inputting them into R. You should note that if you input number values in R but surround them with quotes R will recognize them as Character and not Numeric values. Examples are "Michael", "Data" and "200".
4. Logical values. These are TRUE and FALSE (must always be in CAPS). You enter them in R without quotes, unlike Character values. Also when you carry out comparison operations in R (often called logical operations) the results are logical values. 

Besides these four common ones that you have to be very familiar with and will extensively use in your data analysis work in R, there are two other less common data types: complex values and raw values. I won't bother discussing them because I don't see much real life practical use for them.

Above the layer of data types, we have data structures in R. These are the different standard ways you can organize your data in R. And there are six data structures in R.

1. Vectors: These are the most basic data structure in R and the first you should be familiar with. Usually the other data structures are built on top of vectors, so a proper understanding of vectors provide a fundamental advantage to using the ones built on it. A vector is a collection of values of the same data type. A very common way to create vectors in R is to use the combine function c(). For example c(2,6,7,9) creates a vector that holds the values 2, 6, 7 and 9. You can call out the elements by providing its position number from the left in a square bracket. So to call out the value 6 if I assign the previous vector to a variable called sample_vector, I can write sample_vector[2]. 
2. Factors: These are character values vectors. They hold what in statistics is called nominal data. Data that represent different categories. An example of factor is factor(c("Lagos","New York","Sydney","London")).
3. Lists: These are a more advanced data structure than vectors and factors. They allow for storage of values of different data types and allow you to give each value a name you can reference. Examples of a list are list("Michael",21,"Lagos",FALSE) and list(name="Michael", age=21,city="Lagos",married=FALSE). It is also possible to create list of lists.
4. Data Frames: These are tabular representation of data. Very similar to the way regular database (SQL tables) and spreadsheet (Excel tables) present data. They allow you to reference the values by row and column address. They are a powerful data structure often used for analysis of large records.An example is data.frame(name = c("John","Michael","Tunde"), age=c(32,21,28), city = c("Abuja","Lagos","Kaduna"), married = c(TRUE,FALSE,TRUE))
5. Matrices: Matrices are two dimensional representation of values of the same data type. The values can also be addressed by their row and column position. An example of a matrix is matrix(c(4,5,6,7,8,9), nrow=3)
6. Arrays: Arrays are multi-dimensional tables. They are not limited to two dimensions like the matrix, they can take up as many number of dimensions as desired. An example of an array is array(c(1,2,3,4,5,6,7,8), dim=c(2,2,2)). This is a three dimensional array.

Last week I wrote about doing a sentiment analysis of tweets about the President using Python's Sentiwordnet and Vader. Today, I switched to R and did a clustering and word cloud of the tweets about President Buhari.

Below are the steps I took:

1. I imported all the necessary libraries.
2. I connected to Twitter and created a search stream to gather tweets about Buhari
3. I saved the results in a csv file with append set to true so I can keep piling up the search results from different time of the day. Then I removed punctuation and stopwords.
4. I extracted the most frequently used words and created a word cloud from them. Lastly, I did a clustering of the words.

The require statements that were struck out were of libraries I didn't use but forgot take out before the screenshot

Below is a screenshot of the scrapped tweets.

On Thursday I signed for the Udemy course: Python for Finance: Investment Fundamentals & Data Analytics

It would be helping me learn how to apply Python to Financial and Investment analysis.

Below is a copy paste of a part of the course description and content.

If you are a complete beginner and you know nothing about coding, don’t worry! We will start from the very basics. The first part of the course is ideal for beginners and people who want to brush up their Python skills. And then, once we have covered the basics, we will be ready to tackle financial calculations and portfolio optimization tasks. 

Finance Fundamentals. 
And it gets even better! The Finance block of this course will teach you in-demand real-world skills employers are looking for. To be a high-paid programmer, you will have to specialize in a particular area of interest. In this course, we will focus on Finance, covering many tools and techniques used by finance professionals daily: 
Rate of return of stocks 
Risk of stocks 
Rate of return of stock portfolios 
Risk of stock portfolios 
Correlation between stocks 
Covariance 
Diversifiable and non-diversifiable risk 
Regression analysis 
Alpha and Beta coefficients 
Measuring a regression’s explanatory power with R^2 
Markowitz Efficient frontier calculation 
Capital asset pricing model 
Sharpe ratio 
Multivariate regression analysis 
Monte Carlo simulations 
Using Monte Carlo in a Corporate Finance context 
Derivatives and type of derivatives 
Applying the Black Scholes formula 
Using Monte Carlo for options pricing 
Using Monte Carlo for stock pricing 
Everything is included! All these topics are first explained in theory and then applied in practice using Python. 

Is there a better way to reinforce what you have learned in the first part of the course? 
This course is great, even if you are an experienced programmer, as we will teach you a great deal about the finance theory and mechanics you would need if you start working in a finance context.  

What makes this course different from the rest of the Programming and Finance courses out there? 
This course will teach you how to code in Python and how to apply these skills in the world of Finance. It is both a Programming and a Finance course.
High-quality production – HD video and animations (this isn’t a collection of boring lectures!)
Knowledgeable instructors. Martin is a quant geek fascinated by the world of Data Science, and Ned is a finance practitioner with several years of experience who loves explaining Finance topics, here on Udemy.
Complete training – we will cover all major topics you need to understand to start coding in Python and solving the financial topics introduced in this course (and they are many!)
Extensive Case Studies that will help you reinforce everything you’ve learned.
Course Challenge: Solve our exercises and make this course an interactive experience.
Excellent support: If you don’t understand a concept or you simply want to drop us a line, you’ll receive an answer within 1 business day.
Dynamic: We don’t want to waste your time! The instructors keep up a very good pace throughout the whole course.

I have already begun taking the course.

After a long weekend crisscrossing Lagos, Kaduna and Abuja to deliver our quarterly Business Data Analysis and In-depth Excel Training in Abuja, I rested today and decided to do an interesting data analysis task. 

Do a sentiment analysis on today's tweets about the President, Buhari.

It was a very interesting task. Made me get familiar with Python's Tweepy library. I got some help from Marco Bonzanini. You should check out his Mastering Social Media Mining with Python book.

Here is the breakdown of the steps in achieving the goal:

1. Created the needed credentials to access Twitter API (it's at https://apps.twitter.com/app/new)
2. Use Tweepy to search in realtime for tweets about Buhari
3. Save the tweets as a JSON file
4. Handle emoticons and strings peculiar to Twitter (like @, # and so on)
5. Exclude stop words (words with no significant, for sentiment analysis, meaning) like to, be, is etc
6. Apply Vader to each tweet text and calculate the entire tweet stream sentiment
7. Apply Sentiwordnet, an alternative to Vader, to do the same stream sentiment analysis

Below are the screenshots of some of the steps.

Twitter developer setup

Twitter developer setup

Vader Sentiment Analysis results

Sentiwordnet sentiment analysis result

And the results?

Well let's just say if the tweets are a mirror of what people genuinely feel about President Buhari, he should be terribly worried. Most people were very emotional in their tweets about him and overwhelmingly negative.

Vader classified the tweet stream as 54.67% negative vs 17.76% positive and 27.57% neutral.

Sentiwordnet gave a more damning result: 92.06% negative vs 7.48% positive and 0.47% neutral.

Both Vader and Sentiwordnet are rule-based supervised classification algorithms. In a context like Nigeria where we have our own Naija words and peculiar word mix, machine learning algorithm, supervised or non-supervised, might be more reliable.

R is, arguably, the world's most popular programming language for data analysis. It is an adaptation of S language developed by Bells Laboratories strictly for statistics and data analysis. S language became very popular and R grew out of it as the open-source implementation of S language in August 1993.

The popularity of R is due to its open-source nature and the huge community contributing packages to it that handle all types of common data analysis work. You can read more about R at http://www.r-project.org/ and a more detailed history of R at http://www.r-project.org/about.html

Python, on the other hand, is a general purpose programming language. It is used for all types of programming from building websites (web applications) to computer programs and data analysis. It was created by Guido van Rossum in 1991. It is also an open-source language, benefiting hugely from a large community actively contributing to it. You can read more about Python at https://www.python.org/about/ 

image: medium.com

R and Python are the two most popular languages for data analysis. And anyone serious in becoming a data scientist must be proficient in at least one of the two. I recommend that you have average knowledge in both and then become an expert in one.

Both languages have libraries. More often called packages in R. They are already built algorithms that help you achieve specific tasks. More like Excel formulas, though way more robust in nature. You load them into your R or Python work space and can access the functions they provide.

A common need you will encounter is creating graphs/charts. In R, the most commonly used package for that is ggplot2 and in Python you would use Matplotlib.

To start using R and Python, you will need to install them. 

For R, you can download R at https://cran.r-project.org/ and it is recommended to install RStudio to make using R enjoyable. RStudio is an IDE (integrated development environment) and can be installed at https://www.rstudio.com/  With those two installations, you are set to begin analysing data with R. And they work whether you have a Windows PC, or Mac or Linux.

For Python, it is recommended that you download Anaconda at https://www.continuum.io/downloads. It is regarded as the best distribution of Python for data analysis work. And as for an IDE to use, there is no obvious best as RStudio is for R. Some people are a die-hard fan of Jupyter Notebook (formerly IPython Notebook), luckily it comes pre-installed with Anaconda. Others love Spyder (again, comes pre-installed with Anaconda). And there is PyCharm, you will have to install that at https://www.jetbrains.com/pycharm/. For this training series, I will be using Rodeo downloadable at https://www.yhat.com/products/rodeo, it is an RStudio lookalike. That way you won't have to stress yourself too much in getting used to the IDEs. Once you become familiar with RStudio, you will become more comfortable with Rodeo and vice-versa.

In the next sections we will dig deeper into carrying out simple data analysis tasks in both R and Python.

There is a lot of textbook arguments on Excel vs Python vs R. In fact, a few weeks back I used to be part of that argument. I took sides with Excel. And I had my solid reasons.

But now I don't take sides anymore. I focus on a bigger picture, a picture I became aware of only after I transitioned for reading textbooks and doing video tutorials on Python and R to carrying out live interesting projects with them.

In this webinar I will be sharing my experience on using them all and what practical insight you too need to have to move from those textbook arguments to seeing the big picture. And most importantly, I will be showing you that big picture.

You shouldn't miss this webinar.

Time: 3:00pm to 4:00pm
Date: Tuesday, 21 March 2017
Venue: YouTube Live

You should set it up in your calendar with reminders so you won't miss this special edition.

See you!

Note: The webinar already took place but you can view the YouTube recorded session at https://www.youtube.com/watch?v=YLoOynh8iNQ 

There is a lot of textbook arguments on Excel vs Python vs R. In fact, a few weeks back I used to be part of that argument. I took sides with Excel. And I had my solid reasons.

But now I don't take sides anymore. I focus on a bigger picture, a picture I became aware of only after I transitioned for reading textbooks and doing video tutorials on Python and R to carrying out live interesting projects with them.

In this webinar I will be sharing my experience on using them all and what practical insight you too need to have to move from those textbook arguments to seeing the big picture. And most importantly, I will be showing you that big picture.

You shouldn't miss this webinar.

Time: 3:00pm to 4:00pm
Date: Tuesday, 21 March 2017
Venue: YouTube Live

You should set it up in your calendar with reminders so you won't miss this special edition.

See you!

Today's post will be a very short one. I am watching The Last Samurai and I have a training to prepare for tomorrow. Time is already 11:00pm. So I have to do a quick one today, more urgent than the training is the movie I have to get back to. Whenever I am under pressure, my way of coping is to walk away from it all. Today I have been under immense pressure. We have our Lagos class holding next week and not one single person has made payment. Upper week we have our Abuja training. I have booked both training room and hotel room and have to factor in that no direct flight to Abuja anymore, again not one person has paid. Also next week I have a training session for a mix of managers and operations team at Promasidor. So the pressure is really up my neck. Luckily, I love the thrill it brings. My heart pounds rather than beat and I have Excel/power BI dreams when I sleep.

So back to today's post. I hope you remember the one I made not long ago about same Nigerian Stocks analysis but with R. Well, I have replicated the major one with Python. Below is the code and the analysis chart.

I am also already working on the next phase -- building a predictive model from the time series data. I have started already with R, but I am having trouble with the time series function I am to use, it's not working well for stocks historical data because weekend dates are missing making the time portion irregular.

It's a small bridge and I hope to cross it soon. Till then, enjoy this.

We (UrBizEdge Limited) have added Python and R to the list of courses we conduct class training on. Making our course list total four: Microsoft Excel, Power BI, Python and R.

I am already working on building the curriculum from ground up just like we did the Microsoft Excel training class. There will be a comprehensive training material comprising Nigeria-centric practice data with focus on the business domain. I have started writing the training manual, ground-up, also. 

I have also gathered some of the raw data I would be using for the sample data analysis. I am prioritizing using real data over fictitious data. So I have mined data from the Nigerian stock market and the biggest listed companies. I will be carrying out exploratory data analysis on them and doing time series predictive analysis on them too. This will help participants to have a good feel of the practical real world use of those analysis concepts.

I would then create another set of analysis tutorials around classification systems and Machine Learning. Not yet sure which datasets to seek. 

The good part for you all is that I will be sharing the piece-wise build-up of the content and tutorials as blog posts. In the next few days you should see the one on time series using Python and R. In the end you might learn as much as the training participants would learn if you diligently follow-up with your own practice sessions after each tutorial posts,

image: kdnuggets.com

I have trained a class of people who paid hundreds of thousands of Naira to learn data analysis with R and I have equally paid a lot in books and online training to horn my R skills. For Python, I haven't yet trained a class on it but I have paid a lot too on books and courses to learn it.

Hands down, the best training resource is hands-on projects. After that, training classes and books rule. And today I will be sharing with you the best free training resources I have personally used. They all deliver better value that some of the resources I paid for.

Let's start with R
1. You can't beat Udacity's real-world project based training on R for data analysis free course: https://www.udacity.com/course/data-analysis-with-r--ud651 

2. There is also the amazingly rich and fully free DataCamp Introduction to R course: https://www.datacamp.com/courses/free-introduction-to-r

3. Finally, you should try out Swirl: http://swirlstats.com/students.html

Now on to Python

1. You definitely should read the official Python Tutorial: https://docs.python.org/3.5/tutorial/ . It's a short read and helps you get a very good grasp of how Python works.  Don't forget to set it to the version of Python you use.

2. Then give the everly amazing Udacity course on Python Introduction a try: https://www.udacity.com/course/programming-foundations-with-python--ud036

3. Time to deep dive into Python for Data Analysis on DataCamp: https://www.datacamp.com/courses/intro-to-python-for-data-science

4. Finally, take the aspects that delight you in the comprehensive Microsoft Professional Program Certificate in Data Science: https://www.edx.org/microsoft-professional-program-certficate-data-science

And you are well on your way to becoming a proficient data analyst. 

Best of luck!

image: xlwings.org

Two weeks ago, someone in our MVP community asked a question about Excel VBA vs Python. He is an expert Excel VBA programmer and uses it a lot at his job but now he is considering going fully into the data science field and was considering picking up Python. He wanted to know if anyone has gone a similar route, from Excel VBA to Python, and what the person's experience was.

It was my turn to shine. I am very silent in the community mostly because some of what they discuss are above my knowledge or outside my area of use of Excel. But this was a great opportunity to raise my head and show they that I also know some things. And below is the answer I provided.

"

First, I am probably in same category with you. Already great at VBA for all my automated data analysis work. And like you I have been exploring the world of Python and R. I have even managed to be competent enough in R to teach a 4-day class in it to BI folks from banks and telecoms companies here. And as for Python I am using it for the stock analysis application backend that my company is building. But all these experiences are recent – about a year and half with the usual slowness at the beginning.

Similarities

At the start, majorly due to the huge buzz around Python, I thought there were things I couldn’t do with VBA that were possible with Python. Now, I only think so very faintly. Except building a web app with Python (which, arguably, isn’t data analysis), there is nothing I have seen someone build with Python that isn’t replicable with Excel VBA. Just factor in that maybe I haven’t seen thousands; I have seen just a low hundreds. Is it from data scraping to stochastic search algorithms to time series? We may not enjoy the ready made advantage of Pandas but a good VBA programmer will achieve the same.

Differences

I find three levels of differences:

1. Market opportunities and buzz differences. I am used to clients being surprised by how much I use Excel to achieve for them – from automating the customer site survey and commissioning tasks for a telecoms company that used to pay a third party $50,000/monthly for same results using many manual labour to solving a travelling salesman algorithm question. People constantly underestimate Excel. In contrast, they don’t underestimate Python. For the consulting end, it might not be a big deal since you can always show a proof of concept or be more creative in convincing the client to go with you over the Python guy. But in the job market, they don’t even give you the chance. Every data science role wants Python and/or R. Excel? What is that?
2. Embedded applications. Or should I call them backend tasks. Service workers. Web workers. The names are many. Because you always have to lug around the full Excel application, you can’t use VBA effectively to handle backend tasks for lean applications use. I have been trying to not fall for the temptation of using Excel for our stock analysis applications backend data scrapping and storage into the cloud SQL server. It can do it but not very efficient use of resources. And when it comes to service workers that are supposed to reduce the strain on your web deployed application built on top of your data analysis/science, VBA is out the picture.
3. Pandas, Numpy and all those life easing libraries for advanced analytics. Unfortunately, we don’t have such luxuries in Excel VBA.

 Again, I am more like describing an elephant from touching one part of it. So take this as my biased perspective.

"

I spent the bulk of Saturday creating the backend python scrapper for daily FX rates for over 40 countries. Its main difference is that it will provide you the rates in Naira. So you don't have to recalculate from the usual dollar base back to Naira. When you are looking up the exchange rate for China, India, South Africa, United Arab Emirates and other countries including the popular USA, UK and Canada, you are seeing them listed in Naira. I included a lot of African countries too.

I have also created API access to them, you can checkout the documentation at https://www.nigerianelite.com/help 

Next step is to include them in my Excel add-in.

As an addition, I did similar backend scrapping for the popular stock indices in the world -- from our own NSE All Share Index to Dow Jones and FTSE. 

The best part, especially for business folks, is that you can access historical foreign exchange rate for the over 40 countries all in one place. No more jumping from one site to another. Added to the advantage is that the base is our Naira, you won't have to convert first to dollars and then to Naira. Academic folks to can use it for their research work. 

I have been slow on the Excel add-in because I am creating the newest type that requires JavaScript coding and not the usual familiar VBA. Since I am done with all the backend and API creating parts, I will soon pour myself into finishing the add-in and releasing it to everyone for their use. I will make it free in the beginning, but ultimately I plan to monetize it. It is part of my grand scheme to create mass market products.

I will also see if I can create Android, Windows and iPhone apps from the final work. Should put us in the right limelight. I just have to get this grunt work part sorted and then I can veer into the flashy parts.

I got to join this amazing community of Data Scientists in Nigeria. We are a mix of experts and beginners. Today, I created a tutorial for the beginners to see how to do a common task like frequency distribution plot in both Python and R, also decided to include my dearest Microsoft Excel as a control.

The sample data is a fictionalized data for Dominos Pizza Nigeria. One day sales data for their Lekki branch. You can download the practice along raw data file here: https://dl.dropboxusercontent.com/u/28140414/Dominos%20Pizza.csv 

So the business question we want to tackle is: Is there a pattern in the quantities each customer buys? To be more specific, we want to examine the frequency distribution of the quantities purchased per sales transaction.

In Excel, it is extremely straightforward. Just plot a histogram on the quantity field.

Now let's head to doing same with R

I use R 3.3.2 and RStudio.

First, I import the csv file into RStudio.

Though not necessary for what we want to do, but I like doing it for any data I bring into R, I run the summary command on the dataframe/table. > summary(Dominos_Pizza)

Again, not a required step. I check out the standard plot graph on the Quantity field. > plot(Dominos_Pizza$Quantity)

Finally, I do the histogram chart on the Quantity field. > hist(Dominos_Pizza$Quantity)

For now I don't bother customizing the graph elements (labels, color, title, etc.)

It is Python time.

I use Rodeo IDE and Anaconda. 

I import Pandas and use it to read in the csv file. 

And here is the plot graph, like we did in R.

Finally, I create the histogram.

I will try to follow up with more tutorials of complex tasks, and some that are best suited to R and others that are best suited to Python. As per Excel, it is in a completely different class. It is a spreadsheet application. 

Got any particular task you will like me to create a tutorial around? Ask away!