3. Scala on Lab.Koreanbear|한국곰연구소

Python #3 | Concurrency and Parallelism

Mon, 01 Aug 2022 09:00:00 +0000

Introduction

Overview
- Concurrency :
  - enables a computer to do many different things seemingly at the same time.
  - multiple jobs to take turns accessing the same shared resources, likes disk, network, or a single CPU core.
  - For I/O bounded tasks, Python offers two different mechanisms
    - threading
    - asyncio
- Parallelism :
  - involves actually doing many different things at the same time.
  - For CPU bounded tasks, Python offers multiprocessing
    - multiprocessing
    - subprocess

1. Threading

Threads : are units of work where you can take one or more functions and execute them independently of the rest of the program. You can then aggregate the results, typically by waiting for all threads to run to completition
- Global Interptreter Lock (GIL) : python threads can’t run in parallel on multiple CPU cores because of the GIL.
Advantages
- Python threads are still useful despite the GIL, because they provide an easy way to do multiple things seemingly at the same time.
- Use Python threads to make multiple system calls in parallel. This allows you to do blocking I/O at the same time as computation (things like reading and writing files, interacting with networks, communicating with devices like displays, and so on.)
Disadvatages
- Threads are cooperative. The Python runtime divides its attention between them, so that objects accessed by threads can be managed correctly.
- As a result, threads shouldn’t be used for CPU-intensive work. If you run a CPU-intensive operation in a thread, it will be paused when the runtime switches to another thread, so there will be no performance benefit over running that operation outside of a thread.

import requests
from concurrent.futures import as_completed, wait, ThreadPoolExecutor

def io_bounded(url):
    return requests.get(url).content

urls = ['http://www.foxnews.com/',
        'http://www.cnn.com/',
        'http://europe.wsj.com/',
        'http://www.bbc.co.uk/',
        'http://some-made-up-domain.com/']

with ThreadPoolExecutor(max_workers=5) as ex:
    url_to_futures = {url:ex.submit(io_bounded, url) for url in urls}
    futures = url_to_futures.values()
    # Use "wait" for ordered result or as_completed for not-ordered results
    results = wait(futures, timeout=0.3)

print(url_to_futures.get('http://www.foxnews.com/').result()[:1000])

2. Asyncio

Coroutines : are a different way to execute functions concurrently in Python

3. Multiprocessing

Multiprocessing : allows you to run many CPU-intensive tasks side by side by launching multiple, independent copies of the Python runtime.
Advantages : With threading and coroutines, the Python runtime forces all operations to run serially. Multiprocessing sidesteps this limitation by giving each operation a separate Python runtime and a full CPU core.
Disadvantages :
- Additional overhead is associated with creating the processes
- Each subprocess needs to have a copy of the data it works with sent to it from the main process. (related to pickle)

import time
import multiprocessing

# If the number of process is not given, cpu_count will be used as the default
# The number of processes running concurrently on your computer is not limited by the number of cores.
NUM_OF_PROC = multiprocessing.cpu_count()

def cpu_bounded(input_data):
    time.sleep(1)
    return "result"

with multiprocessing.Pool(NUM_OF_PROC) as p:
     outputs = p.map(cpu_bounded, inputs)

5. References

Python concurrency and parallelism explained

Mathematics for ML #1 | Introduction Part.I

Tue, 18 Jan 2022 09:00:00 +0000

1. Introduction and Motivation

Machine learning is about designing algorithms that automatically extract valuable information from data.
There are three concepts that are at the core of machine learning : data, a model, and learning.
- Data : Since machine learning is inherently data driven, data is at the core of machine learning.
- Model : would describe a function that maps inputs to real-valued outputs.
- Learning : can be understood as a way to automatically find patterns and structure in data by optimizing the parameters of the model

1.1 Finding Words for Intuitions

Data as vectors : there are (at least) three different ways to think about vectors:
- a vector as an array of numbers (computer science view),
- a vector as an arrow with a direction and magnitude (physics view),
- a vector as an object that obeys addition and scaling (a mathematical view)
Model : A good model can be used to predict what would happen in the real world without performing real-world experiments.
Learning : We learn from available data by using numerical optimization methods with the aim that the model performs well on unseen data.

1.2 Two Ways to Read This Book

Bottom-up : Building up the concepts from foundational to more ad-vanced.

Python #5 | 3rd-Party Modules

Tue, 12 Jan 2021 09:00:00 +0000

Introduction

Contents
- Matplotlib
- Pandas
- Pydantic

Matplotlib

Interface
- pyplot interface : Rely on pyplot to automatically create and manage the figures and axes, and use pyplot functions for plotting. (MATLAB-like interface)
- object-oriented interface : Explicitly create figures and axes, and call methods on them

import matplotlib.pyplot as plt
import numpy as np

# pyplot interface (MATLAB-like, rely on pyplot)
x = np.linspace(0,2,100)
plt.plot(x,x**2)
# Recommended) object-oriented interface (create figures and axes manually)
fig, ax = plt.subplots(figsize=(15,4))
ax.plot(x,x**2)

Plot

fig, ax = plt.subplots()
ax.plot(x,x**2)

Axes

# Axes 
fig, ax = plt.subplots()
# set axes
ax.set_title("Title")
ax.set_xlabel('xlabel')
# set limits
ax.set_ylim([0, 1e5])
# rotate tick
ax.tick_params(axis = 'x', labelrotation =90)

Pandas

Groupby

# select columns
df_channel = df[['COLUMN_A', "COLUMN_1", "COLUMN_2", "COLUMN_3", "COLUMN_4"]]
# groupby => will return aggregated list of each columns by COLUMN_A
df_channel_group = df_channel.groupby('COLUMN_A').agg(lambda x: list(x))

Pydantic

Overview
- Data validation and settings management using Python type annotations.
- pydantic enforces type hints at runtime, and provides user friendly errors when data is invalid.
- Define how data should be in pure, canonical Python; validate it with pydantic.
Custom Root Types : __root__
- Pydantic models can be defined with a custom root type by declaring the __root__field.

NLP #1 | Text Preprocessing

Tue, 05 Jan 2021 09:00:00 +0000

Summary

Tokenization
Lemmatization and Stemming
Stopword
Regular Expression
Text Preprocessing Tools for Korean Text

1. Tokenization

Tokenization : dividing a given corpus into tokens, which are units defined for a specific purpose.
- Sentence Tokenization : The unit of the token is a sentence.
  - A dot (.) usually serves as a boundary between sentences, but It is often used as abbreivation, such as ‘Ph.D.’
- Word Tokenization : The criterion of the token is a word.

Python #4 | asyncio

Tue, 01 Jan 2019 09:00:00 +0000

Introduction

asyncio
1. Coroutines
2. Awaitable
3. Creating Tasks

1. Coroutines

Declaration : Python functions defined with the def syntax are synchronous functions. However, to make a function asynchronous, you need to use the async keyword to the function definition. And we call this async funtion, Coroutine.
Run a coroutine : Simply calling a coroutine will not schedule it to be executed. asyncio provides the following mechanisms (multiple ways) to run a coroutine

MLCV #1 | Image Classification

Sat, 02 Jul 2016 09:00:13 +0000

Introduction

Tasks:
- Image Classification : The task of classifying an image according to its visual content.
- Image Representation : focus on the way to encode visual contents into vectors (embedding, encoding)

1. AlexNet (2012)

Introduction : CNNs have been prohibitively expensive to apply in large scale to high resolution images.
Method : Training on Multiple GPUs

def AlexNet(x):
  out = MP(relu(conv11x11(x)))
  out = MP(relu(conv5x5(out)))
  out = relu(conv3x3(out))
  out = relu(conv3x3(out))
  out = MP(relu(conv3x3(out)))
  out = FC(relu(FC(relu(FC(out)))))
  return out

2. VGG Net (2014)

Introduction : come up with significantly more accurate ConvNet
Method : deeper ConvNet

def VGG16(x):
  out = MP(conv3x3(conv3x3(x)))
  out = MP(conv3x3(conv3x3(out)))
  for i in range(3):
    out = MP(conv3x3(conv3x3(conv3x3(out))))
  out = softmax(FC(FC(FC(out))))
  return out

3. GoogleNet (2015)

Introduction : efficient deeper networks (with fewer params than AlexNet)
Method : inception module(NIN, Bottleneck)

def inception_block(x):
  branch_1x1 = conv1x1(x)
  branch_3x3 = conv3x3(conv1x1(x))
  branch_5x5 = conv5x5(conv1x1(x))
  branch_pool = conv1x1(MP3x3(x,same))
  out = concat([branch_1x1,branch_3x3,branch_5x5,branch_pool])
  return out

4. ResNet (Microsoft, 2015)

Introduction : to solve the degradation problem caused by deeper layer.
Method : Residual Block with shortcut(skip) connection defined as :

$$ \mathbf{x}_{l+1} = \mathbf{x}_l + F(\mathbf{x}_l,{W_i}) $$

Python #2 | Built In Modules

Tue, 02 Feb 2016 09:00:13 +0000

argparse

import argparse

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--name", type=str, required=True, help="help")
    args = parser.parse_args()
    print(args.name)

counter

from collections import Counter
Counter(['apple','red','apple','red','red','pear'])
>>> Counter({'red': 3, 'apple': 2, 'pear': 1})

datetime

from datetime import datetime
datetime.today().strftime("%Y%m%d%H%M%S")    # YYYYmmddHHMMSS 형태의 시간 출력

flask

from flask import jsonify, make_response

@application.route('/inference', methods=["GET"])
def infer():
  summary = {'class' : 'cat', 'score':'0.92'}   # make response data
  res = make_response(jsonify(summary), 200)   # make Response object
  res.headers.add("Access-Control-Allow-Origin", "*")   # CORS ERROR 대응
  return res

itertools

from itertools import chain, product, combinations, permutations, groupby, accumulate, islice, cycle

# chain: flatten iterables
list(chain([1,2], [3,4]))                 # [1, 2, 3, 4]

# product: cartesian product
list(product([1,2], ['a','b']))           # [(1,'a'), (1,'b'), (2,'a'), (2,'b')]

# combinations / permutations
list(combinations([1,2,3], 2))            # [(1,2), (1,3), (2,3)]
list(permutations([1,2,3], 2))            # [(1,2), (1,3), (2,1), (2,3), (3,1), (3,2)]

# groupby: consecutive runs (sort first if you want global groups)
data = sorted([('a',1), ('b',2), ('a',3)], key=lambda x: x[0])
{k: [v for _, v in g] for k, g in groupby(data, key=lambda x: x[0])}
# {'a': [1, 3], 'b': [2]}

# accumulate: running totals (or any binary op)
list(accumulate([1,2,3,4]))               # [1, 3, 6, 10]

# islice: slice a generator without materializing
list(islice((x*x for x in range(10)), 2, 6))   # [4, 9, 16, 25]

# cycle: infinite repetition (combine with islice or zip)
list(islice(cycle(['A','B']), 5))         # ['A', 'B', 'A', 'B', 'A']

os

os.chdir('/opt/vidClassifier/classifier/')
# sys.path.append('/opt/vidClassifier/classifier')

pandas

df = pd.read_pickle('PICKLED_PATH')

df.drop(i) # remove i-th row
df.sort_values(by, ascending=True) # sort 

# Filtering 
df.iloc[[0,1,2,3,4,5]] # get rows by indices # not iloc() => iloc[]
df_new = df.loc[df['Column'].str.contains("sub_str1|sub_str2", case=False)] # Filtering rows that contain either sub_str1 or sub_str2

# Groupby
agg_functions = {'col1':'first', 
                 'col2' : 'sum', 
                 'col3' : lambda col: ' && '.join(col), }
df_new = df.groupby(df['id']).aggregate(agg_functions)

pickle

import pickle
# load or save object using pickle
try:
    with open(path_pkl, 'rb') as f: obj_pkl = pickle.load(f)
except:
    obj_pkl = []
    with open(path_pkl, 'wb') as f: pickle.dump(obj_pkl, f)

requests

import requests

# GET
url = 'http://localhost/test'
params = {'arg1':'1', 'arg2':'2'}
response = requests.get(url=url, params=params).json()

# POST
response = requests.post(url=url, data=json.dumps(params))

typing

~~typing.Union~~

Python #1 | Basic

Fri, 01 Jan 2016 09:00:00 +0000

Introduction

Basic Python Syntax

1. Data Types

String
- Limiting floats to N demicmal points
  - round(1.23456, 4)
  - "{:.2f} / {:.3f}".format(1.2345, 1.2345)
  - f"{num:.2f}"
Set
- add: add value to set
- update: add multiples values to set
- remove: remove specific value from set

2. Class

Class : A set of related variables and methods as a blueprint for creating an object
- Methods (instance, static, class) are described in detail in the next chapter.
Object : declared as a class type,

NLP #2 | Text Representation

Wed, 13 Jan 2021 09:00:00 +0000

Summary

Text Representation (Embedding) : When working with text, the first thing you must do is come up with a strategy to convert strings to numbers (or to “vectorize” the text) before feeding it to the model.
Methods
- Sparse Representation : One-hot encoding, Document Term Matrix, etc.
- Dense Representation : Word2Vec, Glove, FastText, etc.
- Pretraind Word Embedding : ELMo, GPT, BERT

1. Sparse Representations

Introduction : Sparse Representation embeds word as a vector which have a relatively small number of nonzero elements. (most of elements in vectors are zero)

MLCV #2 | Object Detection

Fri, 22 Jul 2016 09:00:13 +0000

Introduction

Tasks
- Object Detection : a task of finding the different objects in an image and classifying them
- Salient Object Detection : a task based on a visual attention mechanism, in which algorithms aim to explore objects or regions more attentive than the surrounding areas on the scene or RGB images.
Metrics:
- AP or mAP is generally used as the primary metrics metric.click here for details
Others
- Non Maximum Suppression (NMS) :

MLCV #3 | Semantic Segmentation

Sun, 23 Jul 2017 09:00:13 +0000

Introduction

Tasks:
- Image Segmentation : The process of assigning a label to every pixel in the image.
- Semantic Segmentation : treats multiple objects of the same class as a single entity.
- Instance Segmentation : treats multiple objects of the same class as distinct individual objects.

1. FCN (2015)

Introduction : The first end-to-end pixel-wise prediction model based only on convolutional layers.
Method:
1. Feature Extraction : using convolution layers like conventional Image Classification Tasks (layer 1,2,3,4,5)
2. Convolutionalizing : Downsampling using 1x1 conv rather than FC layer(layer 6,7,8)
3. Pixel Wise Classification : Last conv1x1 layer performs pixel wise classification for 21 classes.
4. Upsampling : using deconvolution layer, also called transposed convolution
5. Fusing Output : x32 upsample from pool5 (FCN-32S) + x16 upsample from pool4 (FCN16S) + x8 upsample from pool3 (FCN8S)

MLCV #4 | Image Synthesis

Tue, 25 Jul 2017 09:00:13 +0000

0. Introduction

Tasks :
- Image Synthesis : The task of creating new images from some form of image description.

1. GAN (2014)

Introduction : A new framework for estimating generative models via an adversarial process
Method: simultaneously train two models : a generative model $G$ that captures the data distribution, and a discriminative model $D$ that estimates the probability that a sample came from the training data rather than $G$.

$$ \min_{G} \max_{D} V(D,G) = \mathbb{E} _ {x \sim p_{data}(x)} logD(x) + \mathbb{E} _ {z \sim p_z(z)}log(1-D(G(z))) $$

NLP #5 | Text Segmentation

Thu, 18 Nov 2021 09:00:00 +0000

Summary

Text Segmentation : is the process of dividing written text into meaningful units, such as words, sentences, or topics.
- To improve the results of Information Retrieval system and help users to find relevant passages faster
- keywords : Text Segmentation, Document Segmentation, Discourse Segmentation
Methods
- Lexical-Cohesion based methods : The first basic insight is that people talk about different topics in different ways: they use different words.
  - Discriminative approach : calculate similarity between neighboring sentences
    - Hearst. TextTiling, 1997 , Alemi and Ginsparg, 2015, and Glavaˇs et al., 2016
  - Clustering approach : generate topic clusters
    - Freddy. C99, 2000
- Feature based Topic Boundaries (Break) Prediction : The second basic insight is that boundaries between topics have their own characteristic features

MLCV #5 | Image Style Transfer

Thu, 26 Apr 2018 09:00:13 +0000

0. Introduction

Tasks :
- Image Style Transfer : The task of migrating a style from one image (Style Image) to another (Content Image).

1. Image Style Transfer using CNNs (2016)

Introduction : Introduce a algorithm that can separate and recombine the image content and style of natural images.
Method : Extract feature maps $F_l$ from each input image $I_{content} $ and $I_{style}$ using pretrained networks at $l_{th}$ layer. Then, optimize $I_{output}$ to have similar contents with $I_{content}$ and similar style with $I_{style}$.

Mathematics for ML #6 | Probabilirty and Distributions

Mon, 13 Jun 2022 09:00:00 +0000

6. Probability and Distributions

Probability, loosely speaking, concerns the study of uncertainty
- Probability can be thought of
  - as the fraction of times an event occurs,
  - as a degree of belief about an event.
We then would like to use this probability to measure the chance of something occurring in an experiment
- In ML, we often quantify
  - uncertainty in the data,
  - uncertainty in the machine learning model, and
  - uncertainty in the predictions produced by the model.
Random Variable : Quantifying uncertainty requires the idea of a random variable, which is a function that maps outcomes of random experiments to a set of properties that we are interested in.
Probability Distribution : Associated with the random variable is a function that measures the probability that a particular outcome (or set of outcomes) will occur

NLP #6 | Text Summarization

Mon, 28 Jun 2021 09:00:00 +0000

Summary

Text Summarization : is a technique to shorten long texts such that the summary has all the important points of the actual document.
- By Summarization Approache
  1. Extraction-based Summarization: The extractive approach involves picking up the most important phrases and lines from the documents. It then combines all the important lines to create the summary. So, in this case, every line and word of the summary actually belongs to the original document which is summarized.

MLCV #6 | Image Retrieval

Thu, 14 Jun 2018 09:00:13 +0000

Introduction

Tasks:
- Image Retrieval : aims to find similar images to a query image among an image dataset.
Tech Trend :
1. Conventional Methods : relying on local descriptor matching (scale invariant features - local image descriptors - reranking with spatial verifications)
2. using FC layers : after several conv layers as global descriptors [A Babenko et al, A Gordo et al.]
3. using global pooling methods : from the activations of conv layers.
4. boost the performance : by combining different global descriptors which are trained individually.

MLCV #7 | Action Classification

Sat, 03 Nov 2018 09:00:13 +0000

Introduction

Tasks:
- Action Classification : The task classfying an action in video sequences according to its spatio-temporal content.
Benchmark Set
- UCF-101 : is an action recognition data set of realistic action videos, collected from YouTube, having 101 action categories.
- HMDB-51
- Kinetics : has 400 human action classes with more than 400 examples for each class, each from a unique YouTube video.
Methods
1. CNN + RNNs
2. 3D Convolutional Networks
  - ResNeXt-101 : 6GFLOPs for 112x112x16
3. Two Stream Network (RGB + Optical Flow)

Mathematics for ML #8 | Introduction Part.II

Sat, 13 Aug 2022 09:00:00 +0000

8. When Models Meet Data

In the first part of the book, we introduced the mathematics that form the foundations of many machine learning methods
The second part of the book introduces four pillars of machine learning:
- Regression (Chapter 9)
- Dimensionality reduction (Chapter 10)
- Density estimation (Chapter 11)
- Classification (Chapter 12)

8.1 Data, Models, and Learning

Three major components of a machine learning system: data, models, and learning.
Good models : should perform well on unseen data.
There are two different senses in which we use the phrase “machine learning algorithm”: training and prediction.

8.1.1 Data as Vectors

MLCV #8 | Pose Estimation

Wed, 07 Aug 2019 09:00:13 +0000

Introduction

Tasks :
- Pose Estimation : The task aims to detect the locations of human anatomical keypoints (e.g., elbow, wrist, etc)

1. Deep Pose (2014)

Introduction : The first major paper that applied Deep Learning to Human pose estimation
Method :
1. DNN-based regression : Alexnet backend (7 layers) with an extra final layer that outputs 2k joint coordinates (where $k$ is the number of joints).
2. Cascade of pose regressors : refinement of the predictions using cascaded regressors.