Soul Hackers Labs

The following article describes the process I followed to discover and download a new text dataset. Often researchers have to go through this process and I hope that with this article I can make someone’s life easier.

A while ago I was working on an area of Natural Language Processing named Sentiment and Emotion analysis, which in simple words aims to determine the sentiment (negative, positive, neutral), or emotion (happy, sad, etc.) expressed in a text. After a few years of not working on NLP, I find myself doing part-time research on Sarcasm Detection, which is closely related to my previous work.

The main goal of this research is to develop a sarcasm detection algorithm that can compare or surpass the state-of-the-art (SOTA), and publish a paper about it. To achieve this, the shortest path is to find a dataset widely used by many researchers in their latest work and develop algorithms using it. The best way to find such a dataset is through the Papers With Code academic website.

To find both the SOTA and a dataset for Sarcasm Detection, I visited that awesome website, searched for “Sarcasm”, and checked the date of publication of the papers returned. I focused on papers published in and after 2020. I was lucky to discover a 2020 paper introducing a completely new and “unbiased” dataset for Sarcasm Detection named iSarcasm, and later a series of 2022 papers competing on Task 6 (Sarcasm Detection) of The 16th International Workshop on Semantic Evaluation SemEval-2022. The best part was that the dataset used during SemEval-2022 Task 6 was an extended version of iSarcasm called iSarcasmEval.

The rest of this article will describe the steps taken to download and prepare the datasets, which are composed of Twitter data. Oftentimes when working with Twitter data, researchers have to follow similar steps, and therefore I hope this article is helpful to someone.

The iSarcasm Dataset

The first dataset I processed was the one introduced in 2020 by the paper titled iSarcasm: A Dataset of Intended Sarcasm. The idea was to present a dataset of tweets labeled by the authors themselves as opposed to labeled by a third party, which often introduced bias or labeling errors.

We show the limitations of previous labelling methods in capturing intended sarcasm and introduce the iSarcasm dataset of tweets labeled for sarcasm directly by their authors.

From the Papers With Code page, you can access the GitHub repository that contains the data. The repo contains two CSV files with the ID of the tweets, a sarcasm_label column with two possible values (sarcastic and not_sarcastic), and a sarcasm_type that identifies the specific type of sarcasm expressed. Why the ID of the tweets and not the texts themselves? Researchers often do this for privacy reasons and to respect some of Twitter’s guidelines. Having only an ID implies two things, first that we need to use a Twitter crawler to obtain the actual texts, and second, we may not be able to retrieve all texts as some tweets might have been deleted by their authors.

To get this dataset ready, perform the following steps.

Step 1: Clone the repo

git clone https://github.com/silviu-oprea/iSarcasm.git

Step 2: Generate query for Tweets Lookup endpoint

The Twitter API allows developers to do many things within the Twitter platform. We are particularly interested in retrieving tweets based on their IDs. For that, we can use the Tweets Lookup endpoint from the API. This tool allows us to lookup up to 100 tweets per call. We, therefore, need a simple piece of Python code that helps us group all the IDs of the dataset into query strings of 100 IDs each.

Python script to group tweet IDs into query strings of 100 IDs each

import pandas as pd
import math

# Read the CSV file with Pandas
data = pd.read_csv("iSarcasm/isarcasm_train.csv")

# Extract the column containing the ids and convert it to a list
tweet_ids = data.loc[:, "tweet_id"].tolist()

# Create a file that will contain the queries
file = open("isarcasm_queries_train.txt", "w")
# Go over all of the ids and create query strings with groups of 100 ids
query = ""
for idx, tweet_id in enumerate(tweet_ids):
    query += str(tweet_id) + ","
    
    if (idx + 1 ) % 100 == 0:
        query = query[:-1] + "\n"
        file.write(query)
        query = ""

# Do not forget the last line
query = query[:-1] + "\n"
file.write(query)

# Close the file
file.close()

Make sure to have this script next to the folder containing the dataset. Replace “iSarcasm/isarcasm_train.csv” with “iSarcasm/isarcasm_test.csv” and “isarcasm_queries_train.txt” with “isarcasm_queries_test.txt” to process the test file in the same way.

Step 3: Get the tweets with the Tweets Lookup endpoint

We can now retrieve the tweets from Twitter using the Tweets Lookup endpoint. To do so, you first need to fulfill a few requirements, like having a developer account and creating a project. These easy-to-follow steps are fully described in this quick-start document.

Prerequisites

To complete this guide, you will need to have a set of keys and tokens to authenticate your request. You can generate these keys and tokens by following these steps:

Sign up for a developer account and receive approval.

Create a Project and an associated developer App in the developer portal.

Navigate to your App’s “Keys and tokens” page to generate the required credentials. Make sure to save all credentials in a secure location.

Assuming you have met all of the requirements, we can retrieve the tweets with a simple Python script. The script will query the Twitter API and obtain 100 results at a time, some of which will contain a tweet and others an error (the tweet has been deleted or became private). In the end, all the retrieved tweets are matched with their labels in the dataset and saved in a new text file, with one tweet and label, tab-separated, per line.

Python script to retrieve tweets and save them with their corresponding labels

import os
import pandas as pd

import requests
import json

# Get this token from your developper portal
os.environ["BEARER_TOKEN"] = '<your_bearer_token>'


# To set your enviornment variables in your terminal run the following line:
# export 'BEARER_TOKEN'='<your_bearer_token>'
bearer_token = os.environ.get("BEARER_TOKEN")


'''
This function creates the url for our GET /tweets request.
Arg: query - the string with 100 comma-separated ids to retrieve
Return: a valid url for the request
'''
def create_url(query):
    tweet_fields = ""
    ids = "ids="+query
    # You can adjust ids to include a single Tweets.
    # Or you can add to up to 100 comma-separated IDs
    url = "https://api.twitter.com/2/tweets?{}&{}".format(ids, tweet_fields)
    return url


'''
This is a method required by bearer token authentication.
Arg: r - a request
Return: a valid request
'''
def bearer_oauth(r):
    """
    Method required by bearer token authentication.
    """

    r.headers["Authorization"] = f"Bearer {bearer_token}"
    r.headers["User-Agent"] = "v2TweetLookupPython"
    return r


'''
This method connects to the endpoint and tries to retrieve the tweets
Arg: url - a valid GET /tweets url
Return: a json response, hopefully with the tweets
'''
def connect_to_endpoint(url):
    response = requests.request("GET", url, auth=bearer_oauth)
    print(response.status_code)
    if response.status_code != 200:
        raise Exception(
            "Request returned an error: {} {}".format(
                response.status_code, response.text
            )
        )
    return response.json()

'''
This method connects to the endpoint multiple times to retrieve all of the tweets 
Arg: queries_path - the path of the file containing all of the queries (comma-separated tweet ids)
Return: ids - a list of the ids of the retrieved tweets 
Return: tweets - a list of texts, the actual tweets 
Return: errors - a list of errors, one per each unavailable tweet 
'''
def get_tweets(queries_path):
    
    # Open the queries file
    file = open(queries_path, "r")

    # Empty lists to accumulate the results
    tweets = []
    ids = []
    errors = []

    # Go over every query string
    for line in file:
        
        # Create the GET request with the current query
        line = line.strip()
        url = create_url(line)
        # Retrieve the tweets
        json_response = connect_to_endpoint(url)
        
        # Accumulate the tweets
        if "data" in json_response:
            for tweet in json_response["data"]:
                
                tweets.append( tweet["text"])
                ids.append(int(tweet["id"]))
                
        # Accumulate the errors     
        if "errors" in json_response:
            for error in json_response["errors"]:
                errors.append(error["title"])
                
    return ids, tweets, errors
    

ids, tweets, errors = get_tweets("isarcasm_queries_train.txt")

print("Retrieved tweets:", len(tweets))
print("Missing tweets:", len(errors))

# Load the original dataset from the CSV file and match ids to labels
data = pd.read_csv("iSarcasm/isarcasm_train.csv")
labels = data.loc[data['tweet_id'].isin(ids), "sarcasm_label"].tolist()

# Save the tweets and their matching labels into a new tab-separated file
file = open("isarcasm_train.txt", "w")
for tweet, label in zip(tweets, labels):
    tweet = " ".join(tweet.split("\n"))
    file.write(tweet + "\t" + label + "\n")
file.close()

Before running the above script, make sure to replace the <your_bearer_token> with your actual token. Also to process the test dataset, change isarcasm_queries_train.txt to isarcasm_queries_test.txt, iSarcasm/isarcasm_train.csv to iSarcasm/isarcasm_test.csv, and isarcasm_train.txt to isarcasm_test.txt.

The iSarcasmEval Dataset

The second dataset, which I consider an extension of the first, was introduced in the paper describing the Task 6 of The 16th International Workshop on Semantic Evaluation SemEval-2022. Luckily, this dataset already provided the text of a tweet and not just its ID. Apart from the column determining whether the tweet is sarcastic or not, it has other columns for the different types of sarcasm.

The information is the same as the previous dataset but in a different format. Like with the previous data, I was only interested in knowing if the tweet is sarcastic or not. The following steps helped me process this dataset and format it just like the previous one.

Step 1: Clone the repo

git clone https://github.com/iabufarha/iSarcasmEval.git

Step 2: Reformat the dataset

The following script reformats the dataset to match the format of the previous one (iSarcasm). I use “sarcastic” and “not_sarcastic” as my labels, feel free to change them to whatever you like.

Python script to reformat the iSarcasmEval dataset

import pandas as pd
import math

# Load the CSV file with Pandas
train_data = pd.read_csv("iSarcasmEval/train/train.En.csv")

# Get only the columns with the text and the sarcastic label
tweets_and_labels = train_data.loc[:, ["tweet", "sarcastic"]].values.tolist()

# Create a new file to save the dataset
file = open("isarcasmeval_train.txt", "w")

# For every tweet-label pair
for count, (tweet, label) in enumerate(tweets_and_labels):
	# If the label is 1, then save the tweet with "sarcastic" label, otherwise use "not_sarcastic"
    if label == 1:
        label = "sarcastic"
    else:
        label = "not_sarcastic"

    # Some rows had no tweet and Pandas interpreted the empty cell as a nan.
    # The code below skips those 
    if not isinstance(tweet, str) and math.isnan(tweet):
        #print(count, tweet)
        continue
    
    # This line removes new lines in the tweet    
    tweet = " ".join(tweet.split("\n"))

    file.write(tweet + "\t" + label + "\n")
file.close()

Finally, to process the test set change “iSarcasmEval/train/train.En.csv” to “iSarcasmEval/test/task_A_En_test.csv” and “isarcasmeval_train.txt” to “isarcasmeval_test.txt”.

Next Step: Do your Machine Learning!

After having retrieved and formatted your data, you are ready to do some cool sarcasm detection. Optionally, you may want to save the tweets in two different files, one for sarcastic and one for non-sarcastic tweets, avoiding having to include the labels inside the files. To do this you just need to make some simple changes to the scripts above. Have fun!

For the past two years, I have been working with robots. Earlier this year I stopped focusing on cameras only and decided to start working with LiDARs. So after much research, I settled for a 32 beams RoboSense device.

I had to spend some time setting it up, especially creating a suitable mount able to also carry a camera. After some playing around, the LiDAR was finally ready and I declare that I am in love with this kind of data.

The next step for my project was to start developing a system to detect and track objects in 3D using LiDAR point clouds. The applications are multiple but include detecting fixed objects (buildings, traffic signs, etc.) to create 3D maps, as well as detecting moving objects (pedestrians, cars, etc.) to avoid collisions.

Before any of the above-mentioned applications could be developed, I first needed to learn how to efficiently load point cloud data into TensorFlow, the tool that I use for Deep Learning. For now, my dataset consists of 12,200 point cloud-image pairs. The image is used as context to know what the LiDAR was looking at. I also pre-processed all point clouds to only show data approximately within the field of view of the camera, as opposed to the original 360° view.

Trying to load the data into TensorFlow was more challenging than I had expected. First, the point clouds were stored as PCD (Point Cloud Data) files, which is a file format for storing 3D point cloud data. TensorFlow cannot directly work with this type of file, so conversion was needed. Enter, the Open3D library, an easy-to-use tool to manipulate point clouds. Using this tool I could easily load a PCD file and extract the points as NumPy arrays of X, Y, and Z coordinates. Another tool, PyPotree, a point cloud viewer for large datasets was used to visualize and confirm that the points were extracted correctly on Google Colab.

So armed with the new tools I uploaded 12,200 PCDs and 12,200 JPGsto my Google Drive and connected it to a Google Colab. I then created some code to load the PCDs, extract the points and put them in a NumPy array, a structure that TensorFlow can easily process. I ran the code confidently and watched in horror how after waiting for several minutes, the Colab complained that it had run out of memory while converting the point clouds. Bad news, as I plan to collect and process a lot more data than I currently have.

Fortunately, this is a common problem when dealing with large datasets, and tools like TensorFlow have the functionality to deal with such situations. The needed solution is the Dataset API, which offers methods to create efficient input pipelines. Quoting the API’s documentation: Dataset usage follows a common pattern:

1. Create a source dataset from your input data.
2. Apply dataset transformations to preprocess the data.
3. Iterate over the dataset and process the elements.

Iteration happens in a streaming fashion, so the full dataset does not need to fit into memory.

So, in essence, the Dataset API will allow me to create a pipeline and the data will be loaded in parts as the training loop in TensorFlow requests it, avoiding running out of memory. So, I reviewed how to use the API, and created some code to make a data pipeline. Following step 1 of the abovementioned pattern, the code first loaded a list of URLs for all of the PCDs and the images, then in step 2, the PCDs were to be loaded and converted to points in NumPy, and the images loaded and normalized. But here is when I ran into trouble again.

To be efficient, everything in the Dataset API (and all TensorFlow APIs apparently) runs as Tensors in a graph. The Dataset API provides functions to load data from different formats, but there were none for PCDs. After studying different possible solutions, I decided that instead of having my data as multiple PCD and JPEG files and having TensorFlow load them and pre-process them, I would instead pre-process all of the data offline, and pack it in an HDF5 file.

The Hierarchical Data Format version 5 (HDF5), is an open-source file format that supports large, complex, heterogeneous data. I obviously verified that the Dataset API supports this type of file. The main advantage of using this format, apart from playing nicely with TensorFlow, is that I can pack all of my data into one nicely structured large file that I can easily move around. I created a simple Python script to load all of the PCDs, extract the points, and pack them together with their corresponding context file into a nice HDF5 file.

After loading the HDF5 file (approx 18 GB) into my Drive, I went back to Colab and added the corresponding Dataset API code. Essentially, step 1 of the pattern loaded the images and points from the HDF5 file and created the corresponding pairs, step 2 did some random selection of points from the point cloud (I will explain why in a later post), and normalized the images, and step 3 was ready to nicely serve the data upon request.

I tried the data pipeline with a very basic training code, and it worked beautifully. No more out-of-memory error. I am not sure if this is the most efficient way to serve my data but it did the trick, and especially, creating the pipeline was a first great exercise in point cloud data manipulation. Next up, training the first TensorFlow model using point clouds.