1. Download the dataset from the given link in the beginning of this document and put it in HDFS in a folder /data/<your-name>. List the content of the folder /data/<your-name> in HDFS using hdfs dfs command.

Proof of implementation

1. Show the content of the folder /data/<your-name> in HDFS.

2. Create a Python virtual environment using venv and add any packages you need.
3. Write a PySpark application which will read the data from HDFS, transform each line or the log of each request in the data file into an RDD of dictionaries where each item in the RDD is stored as a dictionary. We want to keep only the information (remote_addr, time_local, request, url, status, body_bytes_sent) per request. For example, the first line in the file should be transformed into:

{'remote_addr': '54.36.149.41', # str
 'time_local': { # dict
     'year': 2019, # int
     'month': 1, # int
     'day': 22, # int
     'hours': 3, # int
     'minutes': 56, # int
     'seconds': 14 # int
 },
 'request': 'GET', # str
 'url': '/filter/27|13%20%D9%85%DA%AF%D8%A7%D9%BE%DB%8C%DA%A9%D8%B3%D9%84,27|%DA%A9%D9%85%D8%AA%D8%B1%20%D8%A7%D8%B2%205%20%D9%85%DA%AF%D8%A7%D9%BE%DB%8C%DA%A9%D8%B3%D9%84,p53',  # str
 'status': 200, # int
 'body_bytes_sent': 30577 # int
}

Here I dropped some of the metadata from the request line. We assume that we need to keep only the fields mentioned in the example (remote_addr, time_local, request, url, status, body_bytes_sent).

Note: It is not allowed to change the format/schema or the data types of the fields, so they should be stored as the given schema and datatypes for the fields.

Proof of implementation

1. Show the number of items in the RDD.
2. Show first 10 items of the RDD.

4. Create an RDD that contains the number of GET requests per year.

Proof of implementation

1. Show the first 10 results from the RDD sorted by most recent years.

5. Create an RDD that contains the unique ip addresses that has top 10 most successful GET requests.

Proof of implementation

1. Show the first 10 results from the RDD sorted by top 10 most successful GET requests…

1. Create a Spark DataFrame with appropriate column names and data types based on the parsed data. For example, the first line in the file should be transformed into:

{'remote_addr': '54.36.149.41', # StringType
 'time_local': { # StructType
     'year': 2019, # IntegerType
     'month': 1, # IntegerType
     'day': 22, # IntegerType
     'hours': 3, # IntegerType
     'minutes': 56, # IntegerType
     'seconds': 14 # IntegerType
 },
 'request': 'GET', # StringType
 'url': '/filter/27|13%20%D9%85%DA%AF%D8%A7%D9%BE%DB%8C%DA%A9%D8%B3%D9%84,27|%DA%A9%D9%85%D8%AA%D8%B1%20%D8%A7%D8%B2%205%20%D9%85%DA%AF%D8%A7%D9%BE%DB%8C%DA%A9%D8%B3%D9%84,p53',  # StringType
 'status': 200, # IntegerType
 'body_bytes_sent': 30577 # IntegerType
}

Proof of implementation

1. Show the size of the dataframe (rows and columns).
2. Show the schema of the dataframes after reading.
3. Show first 10 samples of the dataframes.

2. Filter the DataFrame to show only the log entries with a status code of 200.

Proof of implementation

1. Show the number of records in the dataframe.
2. Show first 10 samples of the dataframes.

3. Find all requests that resulted in a 4xx error (status code between 400 and 499).

Proof of implementation

1. Show the number of records in the dataframe.
2. Show first 10 samples of the dataframes.

4. Categorize the requests based on the status code (e.g., “Success” for 2xx, “Client Error” for 4xx, “Server Error” for 5xx).

Proof of implementation

1. Show first 10 samples of the dataframe.
2. Show the number of records per category in the dataframe.

5. Calculate the total body_bytes_sent by each remote_addr.

Proof of implementation

1. Show first 10 samples of the dataframe sorted by maximum total of bytes of log body.

6. Analyse the linear correlation between total number of successful requests and the average body size for the logs per year and explain whether the correlation coefficient is significant or not. Use 5% significance level.

Proof of implementation

1. Show first 10 values for both variables (total number of successful requests, average body size) before calculating the correlation coefficient.
2. Show the number of the values for which the correlation will be calculated.
3. Show the final correlation coefficient.
4. Show the result of statistical significance of the correlation coefficient.

7. Define a UDF for extracting the first part from the request endpoint. For example, the request endpoint /filter/27|13%20%D9%85%DA%AF%D8%A7%D9%BE%DB%8C%DA%A9%D8%B3%D9%84,27|%DA%A9%D9%85%D8%AA%D8%B1%20%D8%A7%D8%B2%205%20%D9%85%DA%AF%D8%A7%D9%BE%DB%8C%DA%A9%D8%B3%D9%84,p53 has the first part as /filter. Your UDF should extract this part for all log lines.

Proof of implementation

1. Show the results for first 10 records after applying the UDF on the dataframe. In the output the column before and after applying the UDF should be displayed.

1. Register the DataFrame created in the DataFrame tasks as a temporary SQL view (e.g., nginx_logs).

Proof of implementation

1. Show all tables in the catalog.

2. Write a SQL query to select all columns from the nginx_logs view.

Proof of implementation

1. Show the first 10 records of this view.

3. Write a SQL query to find all GET requests for resources under the /product/ path.

Proof of implementation

1. Show the first 10 results of this query.

4. Write a SQL query to find the top 10 most frequent ip addresses by number of requests.

Proof of implementation

1. Show the top 10 results of this query.

5. Write a SQL query to find the top 10 most requested resources (request path) that resulted in a 404 error.

Proof of implementation

1. Show the top 10 results of this query.

1. Create a Pipeline of data transformers (pyspark.ml.Transformer) to encode all the following features in the dataframe.

remote address (categorical feature)
request method (categorical feature)
request uri (text feature)
status (categorical feature)
body_bytes_sent (numerical feature)

For categorical features you can use one-hot-encoding or any other appropriate encoding method. For text features, you can use word2vec or any other appropriate method but not one-hot-encoding. For numerical features, you need to scale it using an appropriate scaler. Add any other transformers needed here. Fit and transform the dataframe.
2. Create a new column label to indicate whether the request had 200 status code. Encode this column using an appropriate transformer.
3. Build two different classifier to classify the successfulness of requests. The classifiers should be based on different classification methods like Decision Tree and Support Vector Machine.
4. Split the dataset and use the training set for training and test set for reporting the performance of the models.
5. Train the models.
6. Evaluate the performance of both classifiers using accuracy and f1 metrics. Use the test data. Which model is better in terms of f1?
7. Using the best model, predict the success of a specific request from the test set.

Proof of implementation

1. Show the first 10 samples of nginx_logs dataframe before encoding.
2. Show the first 10 samples of nginx_logs dataframe after encoding.
3. Show the size of the train and test dataframes (only number of rows).
4. Show the value of f1 and accuracy for the first model.
5. Show the value of f1 and accuracy for the second model.
6. Answer the question.
7. Show the prediction result for the selected request. The output should have the label and the predicted label.