DumpDocs

Robust MLOps Infrastructure

2025-02-25T19:16:40Z

Creating a reliable and flexible MLOps infrastructure is critical for organizations leveraging machine learning in production. In this post, I’ll explore practical challenges that emerge when deploying ML systems and propose pragmatic solutions using open-source tools and standard workflows.

The Challenge of Production ML Systems

After deploying a machine learning model in production, a new set of challenges emerges. The model pipeline typically involves data storage (like S3), orchestration tools (such as Airflow, Prefect, or Flyte), and back to storage for processed results. But this simple flow becomes complex when we consider the full ML lifecycle.

Let’s address the common incremental issues that arise and propose practical solutions:

Testing Models in Production

Challenge: How do you verify a newly deployed model performs well on critical test data? Pragmatic Solution: Create a dedicated testing workflow that:

Automatically runs inference on a curated set of critical test cases
Compares performance against predefined thresholds for precision, recall, etc.
Generates detailed reports highlighting any concerning areas

# Example test workflow with Prefect
from prefect import flow, task
import pandas as pd
from sklearn.metrics import precision_score, recall_score

@task
def load_critical_test_data(s3_path):
    return pd.read_parquet(s3_path)

@task
def run_model_inference(model, test_data):
    return model.predict(test_data.drop('target', axis=1))

@task
def calculate_metrics(y_true, y_pred):
    return {
        'precision': precision_score(y_true, y_pred),
        'recall': recall_score(y_true, y_pred)
    }

@task
def evaluate_results(metrics, thresholds):
    if metrics['precision'] < thresholds['precision']:
        raise Exception("Model precision below threshold")
    return "Model passed critical test evaluation"

@flow
def test_new_model():
    model = load_model_from_registry("latest")
    test_data = load_critical_test_data("s3://bucket/critical_test_data.parquet")
    predictions = run_model_inference(model, test_data)
    metrics = calculate_metrics(test_data['target'], predictions)
    result = evaluate_results(metrics, {'precision': 0.85, 'recall': 0.80})
    return result

Detecting Model Drift

Challenge: Models degrade over time as data distributions change. How do you proactively detect this? Pragmatic Solution: Implement an automated drift detection workflow that:

Periodically samples production data
Calculates key statistical measures like PSI (Population Stability Index) or KL divergence
Compares feature distributions between training data and current data
Triggers alerts when significant drift is detected

Model Rollback Strategy

Challenge: What if a deployed model starts performing poorly or causing issues? Pragmatic Solution: Implement a robust model registry integrated with your serving infrastructure:

Version and tag all models in the registry
Maintain metadata about each model version (training data, performance metrics)
Create fast-path rollback workflows that can quickly revert to prior stable versions
Automate the rollback process when critical metrics drop below thresholds

Flexible Model Serving

Challenge: How do you create a model serving infrastructure that handles updates, scaling, and canary deployments? Pragmatic Solution: Build a model serving layer that:

Integrates directly with your model registry
Supports A/B testing and canary deployments
Includes automatic scaling based on traffic
Enables request/response logging for performance monitoring
Implements circuit breakers to prevent cascading failures

Changing Model Architecture

Challenge: What if you need to fundamentally change your model architecture? Pragmatic Solution:

Create modular workflow definitions that can accommodate different model architectures
Use environment management tools like Docker and conda to handle different dependencies
Implement feature stores for consistent feature engineering across - model versions
Design input/output interfaces that remain stable even as architectures change

Model Comparison

Challenge: How do you effectively compare multiple model candidates? Pragmatic Solution:

Create a dedicated model comparison workflow that:
1. Runs several models on the same evaluation dataset
2. Computes performance metrics for each model
3. Generates visualizations to compare metrics
4. Automatically selects the best model based on predefined criteria
Keep the model registry at the center of this process to maintain versioning

Here are the few more best practices:

Start with a Minimal Viable Pipeline: Begin with the essential components and iterate
Adopt Standard Tools: Leverage battle-tested open-source tools:
- MLflow for experiment tracking and model registry
- Kubeflow or Flyte for workflow orchestration
- Seldon Core or KServe or custom gRPC / FastAPI for model serving
- Feast for feature stores
- Evidentialy for drift detection
- Prometheus and Grafana for monitoring
Centralize Configuration: Use a central configuration repository for all workflow parameters
Build for Failure: Assume components will fail and design graceful degradation
Standardize Metrics: Define organization-wide standards for model evaluation

DataBricks : ALternative Solution

Databricks provides a unified analytics platform that can significantly enhance and streamline any MLOps workflows. With the overhelming open source tools. Databricks is the safe approach for the MLOps ecosystem.

Databricks claims faster ML development cycles and learn more about it at its academy : https://customer-academy.databricks.com/

Top 50 SQL for Mastery

2024-12-25T15:52:11Z

Recyclable and Low Fat Products

def find_products(products: pd.DataFrame) -> pd.DataFrame:
    # return products[(products["low_fats"] == "Y") & (products["recyclable"] =="Y")].loc[:,["product_id"]]
    return products[(products["low_fats"] == "Y") & (products["recyclable"] =="Y")][["product_id"]]

select product_id from Products where low_fats = "Y" and recyclable = "Y"

Note: can use df.iloc[:, [“column_name”]] or df[[“column_name]] df[“column_name] returns series.

Find Customer Refree

def find_customer_referee(customer: pd.DataFrame) -> pd.DataFrame:
    return customer[(customer["referee_id"] != 2) | (customer["referee_id"].isnull())][["name"]]

def find_customer_referee(customer: pd.DataFrame) -> pd.DataFrame:
    customer.fillna(0, inplace=True)
    return customer.loc[customer["referee_id"] !=2 , ["name"]]

Note: can insert the logic in the loc

select name from Customer where referee_id != 2 or referee_id is Null

Big countries

def big_countries(world: pd.DataFrame) -> pd.DataFrame:
    return world[(world["area"] >= 3000000) | (world["population"] > 25000000)][["name", "population", "area"]]

select name, population, area from world where population >= 25000000 or area >= 3000000

Article View

def article_views(views: pd.DataFrame) -> pd.DataFrame:
    return views[views["viewer_id"] == views["author_id"]][["author_id"]].drop_duplicates()
                .sort_values("author_id").rename(columns={"author_id": "id"})

select distinct(author_id) as id from Views where author_id = viewer_id order by id

invalid tweets

select tweet_id from Tweets where length(content) >15

def invalid_tweets(tweets: pd.DataFrame) -> pd.DataFrame:22
    return tweets.query(f"content.str.len() >15")[["tweet_id"]]

    return tweets.loc[tweets.content.str.len() >15, ["tweet_id"]]

Replace Employee id

def replace_employee_id(employees: pd.DataFrame, employee_uni: pd.DataFrame) -> pd.DataFrame:
    return pd.merge(employees, employee_uni, how="left", on=["id", "id"])[["unique_id", "name"]]

select unique_id, name from Employees left join EmployeeUNI on Employees.id = EmployeeUNI.id

Product Sales Analysis

SELECT product_name, year, price from Sales left join Product on Sales.product_id = Product.product_id

customer-who-visited-but-did-not-make-any-transactions

SELECT 
 customer_id,
 COUNT(*) AS count_no_trans

FROM Visits LEFT JOIN Transactions ON Visits.visit_id = Transactions.visit_id WHERE transaction_id IS NULL GROUP BY customer_id; ``` 9. Rising Temperature

import pandas as pd

def rising_temperature(weather: pd.DataFrame) -> pd.DataFrame:
    if weather.size:
        weather.sort_values(by=['recordDate'], inplace=True)
        weather.reset_index(inplace=True)
        weather.recordDate= pd.to_datetime(weather.recordDate, format="-%Y-%m-%d")
        new_dates = pd.date_range(start=weather['recordDate'][0],
                                  end=weather['recordDate'][len(weather)-1],freq='D')

        weather = weather.set_index("recordDate").reindex(new_dates).reset_index()
        weather.rename(columns={"index": "recordDate"}, inplace=True)
        weather["change"] = weather.temperature - weather.temperature.shift(1, fill_value=weather.temperature[0])
        return weather[weather["change"] > 0.0][["id"]].rename(columns={"id": "Id"})
    else:
        return weather[["id"]].rename(columns={"id":"ID"})

WITH PreviousWeatherData AS
(
    SELECT 
        id,
        recordDate,
        temperature, 
        LAG(temperature, 1) OVER (ORDER BY recordDate) AS PreviousTemperature,
        LAG(recordDate, 1) OVER (ORDER BY recordDate) AS PreviousRecordDate
    FROM 
        Weather
)

SELECT 
    id 
FROM 
    PreviousWeatherData
WHERE 
    temperature > PreviousTemperature
AND 
    recordDate = DATE_ADD(PreviousRecordDate, INTERVAL 1 DAY);

This might be slower, as seen in leetcode.

SELECT 
    w1.id
FROM 
    Weather w1
WHERE 
    w1.temperature > (
        SELECT 
            w2.temperature
        FROM 
            Weather w2
        WHERE 
            w2.recordDate = DATE_SUB(w1.recordDate, INTERVAL 1 DAY)
    );

SELECT 
    w1.id
FROM 
    Weather w1
JOIN 
    Weather w2
ON 
    DATEDIFF(w1.recordDate, w2.recordDate) = 1
WHERE 
    w1.temperature > w2.temperature;

Bonus:

select EmpLastName, EmpFirstName, ProjDesc, BldName, DeptName
from Employee, Building, Department, Project
where DeptBuilding = BldID
    and empDeptID = DeptID
    and EmpID = ProjManager
order by EmpLastName, DeptName, BldName

Multiple Tables could be used to join without using the word. Make sure to add where clause otherwise, data are added with each new row.

from Department
inner join employee
inner join Building
on empDeptID = DeptID
and DeptBuilding = BldId
order by DeptName, EmpSalary desc

Average Time of Process per machine

select a1.machine_id, round(avg(a2.timestamp- a1.timestamp), 3) as processing_time
from Activity a1 
join Activity a2
on a1.machine_id = a2.machine_id and a1.process_id = a2.process_id and a1.timestamp < a2.timestamp 
group by a1.machine_id

Employee Bonus

select name, bonus
from Employee
left join Bonus
on Employee.empId = Bonus.empId
where Bonus.bonus < 1000 or Bonus.bonus is null

Students and Examinations

select S.student_id, S.student_name, SU.subject_name, COUNT(E.student_id) as attended_exams
from Students S
CROSS JOIN Subjects SU
left join Examinations E
on S.student_id = E.student_id 
AND SU.subject_name = E.subject_name
Group By S.student_id, SU.subject_name
ORDER BY s.student_id

Managers with at least 5 direct reports

select b.name
from Employee a
Join Employee b on a.ManagerId = b.id
group by a.ManagerId
having count(*) >= 5

Confirmation rate

select s.user_id, round(avg(if(c.action ="confirmed", 1, 0)), 2 ) as confirmation_rate
from SignUps as s
left join Confirmations as c
on s.user_id = c.user_id
group by user_id

Not Boring Movie

select * 
from Cinema
where description != "boring" and id%2 !=0
order by rating desc

Average Selling Price

select P.product_id, ifnull(round(sum(US.units * P.price)/ sum(US.units), 2), 0) as average_price
from Prices as P
left join UnitsSold as US
on US.product_id = P.product_id and US.purchase_date >= P.start_date
    and US.purchase_date <= P.end_date
group by P.product_id

Project employees

select project_id, ROUND(AVG(experience_years),2) as average_years
from Project P
left join Employee E
on P.employee_id = E.employee_id
group by P.project_id

Percentage of Users attended a contest

select contest_id, Round(Count(user_id)* 100/ (select count(*) from Users), 2) as percentage
from Register
group by contest_id
order by percentage desc, contest_id asc

Queries quality and Percentage

select query_name, ROUND(AVG(Q.rating/ Q.position), 2) AS quality,
    Round(Sum(Q.rating < 3) *100 / Count(Q.rating), 2) as poor_query_percentage
from Queries Q
group by query_name

Monthly Transactions I

SELECT DATE_FORMAT(trans_date, "%Y-%m") as month, 
    country, 
    Count(*) as trans_count, 
    SUM(state = 'approved') as approved_count,
    SUM(amount) as trans_total_amount,
    SUM(Case when state = 'approved' then amount else 0 end) as approved_total_amount
FROM Transactions
GROUP BY month, country

Immediate Food Delivery

Game Play Analysis

Machine Learning Box

2024-08-28T16:47:41Z

Imagine a box where you put all of your machine learning stuff, Here it is. [WIP] will update the structure

Bias vs Varience

Metrics

Precision

Recall

Accuracy

F1-score

Cross-Validation

How do you choose which cross validation technique will be used for your project. THink about how your model will be sued and interact with the data in a deployed setting. if the dataset is huge, use Hold-out, which is basically 80-20 method

K fold

if the data points are independent to each other.

if the dataset is unbalanced: stratifiedKfold, as it should be aware with the classes. if very little data, shufflesplit

Time split

if time is influencial in generation of the data, use Timesplit

GroupKfold

if the data is generated by the a patient , and we have lets say n patient generating the x data point, it would be great to use group kfold. Group in this case in the participants.

stratified Groupkfold

if the data points skewed, we can use the stratisfied GroupKfold as well.

Setting up Nvidia SDK Manager and Torch Library in Jetson Board

2024-08-19T12:37:53Z

The blog serves as a backup for setting up the Nvidia Jetson Orion AGX for Development/Production. Please note: review all the steps before proceeding.

I was updating from JetPack 5.x to 6.x. My Jetson Orion AGX had already been flashed.

Installation

The blog covers installing the Nvidia sdk manager. Make sure to install Ubuntu 22.04, not 24.04, for installing JetPack 6 (as of August 19, 2024). Similarly, upgrade the host machine, install all the necessary Nvidia drivers and CUDA on the host machine before installing the Nvidia SDK Manager. More information on installing Nvidia Drivers on ubuntu

Prerequistics

It is also advisable to review the dedicated developer kit user guide Go over Jetson Zoo and check for package versions if needed: ZOO

Prepare the Nvidia Jetson Device

Once you log into the Nvidia SDK Manager, you can't verify the target board. To do that, follow these steps:

Connect the USB Type-C port located above the power jack to the power source.
Connect the Type-C port located beside the GPIO pins to the host computer.
Connect the Micro-USB [UART] port located beside the HDMI pins to the host computer.

After this, make sure you boot the Nvidia Jetson in recovery mode. You can put the device in recovery mode in two ways:

Press the middle button (between power and restart) and the restart button together, then release them together. [This step didn't work for me.]
Turn on the Nvidia Jetson machine, open the terminal, and type:

sudo reboot --force forced-recovery

After this the nvidia board will be seen in the Nvidia SDK Manager. You can also check it via command in host machine.

lsusb

Nvidia SDK Manager

Once the target board is detected, select the drive location where you want to download the necessary files. Then, a prompt will appear to flash the board. Flash your board by selecting the eMMC drive. Once the flash process is completed, it will prompt you to proceed with the installation of the Nvidia SDK Manager. However, I encountered an issue where it wouldn't allow me to install everything.

Here’s a workaround: Open the terminal on the freshly flashed Nvidia board.

Connect to Wi-Fi, because the flashed Nvidia board functions like a new computer. Install the necessary software and drivers to proceed.

sudo apt update && sudo apt upgrade
sudo apt dist-upgrade
sudo apt install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev
sudo apt install python3-pip
sudo apt install nvidia-jetpack

And now, voila! You have the latest version of Nvidia Jetson.

Installing python packages.

Installing Python packages, including Poetry, can be tricky because TensorRT is installed in the system. To use it, you need to run the following command:

poetry config virtualenvs.options.system-site-packages true

Yes, TensorRT is included in JetPack. You can verify its installation using the following command:

sudo apt-get install tensorrt nvidia-tensorrt-dev python3-libnvinfer-dev

Next, install Torch, TorchVision, and ONNX Runtime.

wget https://nvidia.box.com/shared/static/mp164asf3sceb570wvjsrezk1p4ftj8t.whl -O torch-2.3.0-cp310-cp310m-linux_aarch64.whl
pip install torch-2.3.0-cp310-cp310m-linux_aarch64.whl

for the wheel file url check this.

forum posts , found both torch and torchvision, but might not update later.
jetson zoo I didn't torch here
jetson distribution only found torch not torchvision.

Yes, it’s a bit of a mess, but it works. We don’t have much flexibility with the Torch version—we have to use whatever Nvidia provides for the JetPack versions.

Benchmarking.

Now, use the script to benchmark the speed of Torch and TensorRT.

import torch
import onnx
import onnx_tensorrt.backend as backend
import tensorrt as trt
import time
import numpy as np

# Define a simple PyTorch model
class MyModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = torch.nn.Conv2d(3, 32, kernel_size=3, stride=1, padding=1)
        self.relu1 = torch.nn.ReLU()
        self.conv2 = torch.nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1)
        self.relu2 = torch.nn.ReLU()
        self.pool = torch.nn.MaxPool2d(kernel_size=2, stride=2)
        self.fc1 = torch.nn.Linear(64 * 16 * 16, 512)
        self.relu3 = torch.nn.ReLU()
        self.fc2 = torch.nn.Linear(512, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = self.relu1(x)
        x = self.conv2(x)
        x = self.relu2(x)
        x = self.pool(x)
        x = x.view(-1, 64 * 16 * 16)
        x = self.fc1(x)
        x = self.relu3(x)
        x = self.fc2(x)
        return x

# Export the PyTorch model to ONNX
model = MyModel()
input_shape = (1, 3, 32, 32)
input_names = ['input']
output_names = ['output']
dummy_input = torch.randn(input_shape)
torch.onnx.export(model, dummy_input, 'my_model.onnx', verbose=False, input_names=input_names, output_names=output_names)

# Load the ONNX model and create a TensorRT engine from it
model_onnx = onnx.load('my_model.onnx')
engine = backend.prepare(model_onnx, device='CUDA:0')

# Create a context for executing inference on the engine
context = engine.create_execution_context()

# Allocate device memory for input and output buffers
input_name = 'input'
output_name = 'output'
input_shape = (1, 3, 32, 32)
output_shape = (1, 10)
input_buf = trt.cuda.alloc_cuda_pinned_memory(trt.volume(input_shape) * trt.float32.itemsize)
output_buf = trt.cuda.alloc_cuda_pinned_memory(trt.volume(output_shape) * trt.float32.itemsize)

# Load the PyTorch model into memory and measure inference speed
model.load_state_dict(torch.load('my_model.pth'))
model.eval()
num_iterations = 1000
total_time = 0.0
with torch.no_grad():
    for i in range(num_iterations):
        start_time = time.time()
        input_data = torch.randn(input_shape)
        output_data = model(input_data)
        end_time = time.time()
        total_time += end_time - start_time
pytorch_fps = num_iterations / total_time
print(f"PyTorch FPS: {pytorch_fps:.2f}")

# Create a TensorRT engine from the ONNX model and measure inference speed
trt_engine = backend.prepare(model_onnx, device='CUDA:0')
num_iterations = 1000
total_time = 0.0
with torch.no_grad():
    for i in range(num_iterations):
        input_data = torch.randn(input_shape).cuda()
        start_time = time.time()
        output_data = trt_engine.run(input_data.cpu().numpy())[0]
        end_time = time.time()
        total_time += end_time - start_time
tensorrt_fps = num_iterations /total_time
tensorrt_fps = num_iterations / total_time
print(f"TensorRT FPS: {tensorrt_fps:.2f}")
print(f"Speedup: {tensorrt_fps/pytorch_fps:.2f}x")

Output:

(model-train-py3.10) jetson-dublin@ubuntu:~/Development/shekhar/model_train$ python benchmark.py 
PyTorch  Inference Time: 0.007773 seconds
TensorRT Inference Time: 0.002509 seconds
TensorRT is 3.10x faster than PyTorch

Install Cuda 12.2 in Ubuntu 24.04

2024-08-16T09:56:06Z

So, Ubuntu 24.04 LTS was released on 25 April 2024. Its been more 3 months from the release and I thought it would be safe enough to install it in my main working computer. Wrong!! I wanted a documentation for setting up the CUDA 12.2 in Ubuntu 24.04 but ended for reinstalling everything. Thus, here is the blog for future me to setup everything. Thanks to all the great help I found in the Internet [ referenced as links ].

Ubuntu Drivers:

At first glance, it was nice and clean. I really the ubuntu drivers, may be it was in the ubuntu 22.x or 20.04.

sudo ubuntu-drivers list

nvidia-driver-520, (kernel modules provided by nvidia-dkms-520)
nvidia-driver-470-server, (kernel modules provided by linux-modules-nvidia-470-server-generic-hwe-24.04)
nvidia-driver-555, (kernel modules provided by nvidia-dkms-555)
nvidia-driver-470, (kernel modules provided by linux-modules-nvidia-470-generic-hwe-24.04)
nvidia-driver-535-server, (kernel modules provided by linux-modules-nvidia-535-server-generic-hwe-24.04)
nvidia-driver-515, (kernel modules provided by nvidia-dkms-515)
nvidia-driver-550, (kernel modules provided by linux-modules-nvidia-550-generic-hwe-24.04)
nvidia-driver-525, (kernel modules provided by nvidia-dkms-525)
nvidia-driver-535, (kernel modules provided by linux-modules-nvidia-535-generic-hwe-24.04)
nvidia-driver-545, (kernel modules provided by nvidia-dkms-545)
nvidia-driver-560, (kernel modules provided by nvidia-dkms-560)

< The result might be different for the fresh out of the box Ubuntu >

and the best advertised part was :

sudo ubuntu-drivers autoinstall
# It will install the best latest version for your device.  haha

But, it is great to have all the nvidia-drivers installed right after the fresh OS update. cherry on top, NVIDIA-SMI works. Now, I only need to install libraries like cuda and cudnn for AI/ML pipeline.

Installing Nvidia CUDA

Before Installing the CUDA, check the compalibility table. Nvida CUDA Compability and Table

so, I chose (CUDA 12.2) as the nvidia driver 535.54.03+ was already installed on. Now to install the CUDA, Nvidia gave you three options based on the Architecture and Distribution selected: Nvidia CUDA installation

But wait, there is no ubuntu 24.04 version. So, You can’t run the network options or runfile. You have to rely on the 22.04 version deb local method.

so I followed the following instruction method. Note: I had added the -12-2, otherwise it was installing the latest CUDA, which was 12.6

sudo apt-get install linux-headers-$(uname -r)
sudo apt-key del 7fa2af80
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin
sudo mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/12.2.2/local_installers/cuda-repo-ubuntu2204-12-2-local_12.2.2-535.104.05-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2204-12-2-local_12.2.2-535.104.05-1_amd64.deb
sudo cp /var/cuda-repo-ubuntu2204-12-2-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-get -y install cuda-12-2

Here, the problem starts, it won’t install the cuda as it requires Nvidia-driver 560.x, and you have 535.x.

Installing latest or any Nvidia Driver version in Ubuntu [RabbitHole]

At this point, I thought lets install Nvidia-driver 560.x. I found a really well documented guide: If Not Trye Then False’s Guide for Install Nvidia Driver

This thing is great, but I have question for you. Do you know Nvidia Optimus ? what is Initramfs / prime-select / run level 3 / gdm3 / lightdm / DKMS / MODPROBE / Wayland / X org ?

if not, dont go in the rabbithole as you might end up having problem with it. if you want to learn about it : Nvidia Optimus: a comfy guide

Luckily the above guide have section : if you see Black Screen after Nvidia drivers install, then check following video

Manuall DPKG Nvidia Driver installation

Eventually, the better way to install the CUDA was to dont trust the Ubuntu for the Nvidia Drivers. Based on the great post

 sudo dpkg --force-all -P nvidia-firmware-535-535.54.03 nvidia-kernel-common-535 nvidia-compute-utils-535
 sudo apt --fix-broken install

Yes, you break the system and then fix the broken. Luckily, there was no black screen when I was doing this. And I am also

check Nvidia-smi, you should have Nvidia 535.x installed.

Now you could run the above code:

sudo dpkg -i cuda-repo-ubuntu2204-12-2-local_12.2.2-535.104.05-1_amd64.deb
sudo apt update
sudo apt install cuda

solving CUDA dependencies: libtinfo5

if you have problem with libtinfo5, Based on the site: libtinfo5

sudo apt install software-properties-common
curl -O http://launchpadlibrarian.net/648013231/libtinfo5_6.4-2_amd64.deb
curl -O http://launchpadlibrarian.net/648013227/libncurses5_6.4-2_amd64.deb
sudo dpkg -i libtinfo5_6.4-2_amd64.deb
sudo dpkg -i libncurses5_6.4-2_amd64.deb
sudo apt install libnss3-tools
curl -O http://launchpadlibrarian.net/646633572/libaio1_0.3.113-4_amd64.deb
sudo dpkg -i libaio1_0.3.113-4_amd64.deb
sudo apt install cuda-12-2

It should run and add this to ~/.profile

# set PATH for cuda 12.2 installation
if [ -d "/usr/local/cuda-12.2/bin/" ]; then
    export PATH=/usr/local/cuda-12.2/bin${PATH:+:${PATH}}
    export LD_LIBRARY_PATH=/usr/local/cuda-12.2/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
fi

That should give you nvcc version

. ~/.profile
nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Aug_15_22:02:13_PDT_2023
Cuda compilation tools, release 12.2, V12.2.140
Build cuda_12.2.r12.2/compiler.33191640_0

You can install cuDNN if you are into training Neural Network, check the compability version of cuDNN again: compability matrix

sudo apt install cudnn9

And thats all you need.

Bonus:

1. Installing Pyenv in the Fresh Ubuntu 24.04

Yes, in Ubuntu right out of the box, Pyenv installation won’t work. You need to add following Prerequists:

sudo apt install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

then

curl https://pyenv.run | bash
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc^Jecho '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc^Jecho 'eval "$(pyenv init -)"' >> ~/.zshrc
exec zsh
sudo apt install python3-pip
pyenv install 3.13
pyenv install 3.12
pyenv install 3.11

and so on ..

2. Installing Virtualenv in the Fresh Ubuntu 24.04

Pipenv

sudo apt install software-properties-common python-software-properties
sudo add-apt-repository ppa:pypa/ppa
sudo apt update
sudo apt install pipenv

Poetry:

curl -sSL https://install.python-poetry.org | python3 -

3. Checking Cuda with Pytorch Script.

Here is a small script to test the GPU and the packages. You need to install Torch, Torchvision , Numpy to run the script. You can use the latest packages for this.

import torchvision
import torch
import traceback

tcuda: torch.cuda = torch.cuda
print(f"How many gpu in the server ? -> {tcuda.device_count()}")
print(f"can torch access {tcuda.device_count()} GPUs? :> {tcuda.is_available()}")
print(f"Attempting to access GPU device name : {tcuda.current_device()} ....")


try:
    model = torchvision.models.detection.retinanet_resnet50_fpn(pretrained=True)
    model.cuda()
    model.eval()
    x = [torch.rand(3, 300, 400).cuda(), torch.rand(3, 500, 400).cuda()]
    predictions = model(x)
    print(predictions)
    print(f"is torch gpu is initialized ? :> {tcuda.is_initialized()}")
    print(f"torch cuda memory summary :> {tcuda.memory_summary()}")
    print(f"memory reserved {tcuda.memory_reserved()}")
    print(f"memory allocated {tcuda.memory_allocated()}")

except Exception as e:
    print(e)
    print("No CUDA available")
    traceback.print_exc()

4. setting up the Terminal

ZSH

Install Some cool Nerd Fonts

Install zsh and oh my zsh and Terminator

sudo apt install zsh
sudo apt install terminator
sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

Install Oh my zsh plugins

git clone https://github.com/zsh-users/zsh-autosuggestions.git $ZSH_CUSTOM/plugins/zsh-autosuggestions
git clone https://github.com/zsh-users/zsh-syntax-highlighting.git $ZSH_CUSTOM/plugins/zsh-syntax-highlighting
git clone https://github.com/zdharma-continuum/fast-syntax-highlighting.git ${ZSH_CUSTOM:-$HOME/.oh-my-zsh/custom}/plugins/fast-syntax-highlighting
git clone --depth 1 -- https://github.com/marlonrichert/zsh-autocomplete.git $ZSH_CUSTOM/plugins/zsh-autocomplete

Install Powerlevel10k

 git clone --depth=1 https://github.com/romkatv/powerlevel10k.git ${ZSH_CUSTOM:-$HOME/.oh-my-zsh/custom}/themes/powerlevel10k

Update this in the ~/.zshrc

ZSH_THEME="powerlevel10k/powerlevel10k"
plugins=(git zsh-autosuggestions zsh-syntax-highlighting fast-syntax-highlighting zsh-autocomplete python)

and update the zshrc and setup the theme

nano ~/.zshrc
exec zsh
p10k configure

Atuin

curl --proto '=https' --tlsv1.2 -LsSf https://setup.atuin.sh | sh
atuin login
source $HOME/.atuin/bin/env
atuin login
atuin sync

Handle SSH

I prefer to run multiple ssh key in a single machine and it is handled by a ssh config file.

cat config

#Default GitHub
Host github.com
  HostName github.com
  User git
  IdentityFile ~/.ssh/id_rsa

Host github-mac
  HostName github.com
  User git
  IdentityFile ~/.ssh/id_rsa_personal

Host gitlab
  HostName gilab.com 
  IdentityFile ~/.ssh/id_rsa_personal

id_rsa could be your one key and id_rsa_personal could be your personal / other github user.

cd ./ssh
sudo cp /home/shekhar/Downloads/ssh/. . 
ssh -T git@github.com
ssh-add id_rsa
ssh-add id_rsa_personal

Installing CUDA 12.x on Ubuntu 24.04

2024-08-16T09:56:06Z

MOTIVE

Ubuntu Drivers:

At first glance, it was nice and clean. I really the ubuntu drivers, may be it was in the ubuntu 22.x or 20.04.

sudo ubuntu-drivers list

nvidia-driver-520, (kernel modules provided by nvidia-dkms-520)
nvidia-driver-470-server, (kernel modules provided by linux-modules-nvidia-470-server-generic-hwe-24.04)
nvidia-driver-555, (kernel modules provided by nvidia-dkms-555)
nvidia-driver-470, (kernel modules provided by linux-modules-nvidia-470-generic-hwe-24.04)
nvidia-driver-535-server, (kernel modules provided by linux-modules-nvidia-535-server-generic-hwe-24.04)
nvidia-driver-515, (kernel modules provided by nvidia-dkms-515)
nvidia-driver-550, (kernel modules provided by linux-modules-nvidia-550-generic-hwe-24.04)
nvidia-driver-525, (kernel modules provided by nvidia-dkms-525)
nvidia-driver-535, (kernel modules provided by linux-modules-nvidia-535-generic-hwe-24.04)
nvidia-driver-545, (kernel modules provided by nvidia-dkms-545)
nvidia-driver-560, (kernel modules provided by nvidia-dkms-560)

< The result might be different for the fresh out of the box Ubuntu >

and the best advertised part was :

sudo ubuntu-drivers autoinstall
# It will install the best latest version for your device.  haha

Installing Nvidia CUDA

Before Installing the CUDA, check the compalibility table. Nvida CUDA Compability and Table

But wait, there is no ubuntu 24.04 version. So, You can't run the network options or runfile. You have to rely on the 22.04 version deb local method.

so I followed the following instruction method. Note: I had added the -12-2, otherwise it was installing the latest CUDA, which was 12.6

sudo apt-get install linux-headers-$(uname -r)
sudo apt-key del 7fa2af80
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin
sudo mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/12.2.2/local_installers/cuda-repo-ubuntu2204-12-2-local_12.2.2-535.104.05-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2204-12-2-local_12.2.2-535.104.05-1_amd64.deb
sudo cp /var/cuda-repo-ubuntu2204-12-2-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-get -y install cuda-12-2

Here, the problem starts, it won't install the cuda as it requires Nvidia-driver 560.x, and you have 535.x.

Installing latest or any Nvidia Driver version in Ubuntu [RabbitHole]

At this point, I thought lets install Nvidia-driver 560.x. I found a really well documented guide: If Not Trye Then False's Guide for Install Nvidia Driver

This thing is great, but I have question for you. Do you know Nvidia Optimus ? what is Initramfs / prime-select / run level 3 / gdm3 / lightdm / DKMS / MODPROBE / Wayland / X org ?

if not, dont go in the rabbithole as you might end up having problem with it. if you want to learn about it : Nvidia Optimus: a comfy guide {{< youtube Pn2iUgW3l6w >}} Luckily the above guide have section : if you see Black Screen after Nvidia drivers install, then check following video

Manuall DPKG Nvidia Driver installation

Eventually, the better way to install the CUDA was to dont trust the Ubuntu for the Nvidia Drivers. Based on the great post

 sudo dpkg --force-all -P nvidia-firmware-535-535.54.03 nvidia-kernel-common-535 nvidia-compute-utils-535
 sudo apt --fix-broken install

Yes, you break the system and then fix the broken. Luckily, there was no black screen when I was doing this. And I am also

check Nvidia-smi, you should have Nvidia 535.x installed.

Now you could run the above code:

sudo dpkg -i cuda-repo-ubuntu2204-12-2-local_12.2.2-535.104.05-1_amd64.deb
sudo apt update
sudo apt install cuda

solving CUDA dependencies: libtinfo5

if you have problem with libtinfo5, Based on the site: libtinfo5

sudo apt install software-properties-common
curl -O http://launchpadlibrarian.net/648013231/libtinfo5_6.4-2_amd64.deb
curl -O http://launchpadlibrarian.net/648013227/libncurses5_6.4-2_amd64.deb
sudo dpkg -i libtinfo5_6.4-2_amd64.deb
sudo dpkg -i libncurses5_6.4-2_amd64.deb
sudo apt install libnss3-tools
curl -O http://launchpadlibrarian.net/646633572/libaio1_0.3.113-4_amd64.deb
sudo dpkg -i libaio1_0.3.113-4_amd64.deb
sudo apt install cuda-12-2

It should run and add this to ~/.profile

# set PATH for cuda 12.2 installation
if [ -d "/usr/local/cuda-12.2/bin/" ]; then
    export PATH=/usr/local/cuda-12.2/bin${PATH:+:${PATH}}
    export LD_LIBRARY_PATH=/usr/local/cuda-12.2/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
fi

That should give you nvcc version

. ~/.profile
nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Aug_15_22:02:13_PDT_2023
Cuda compilation tools, release 12.2, V12.2.140
Build cuda_12.2.r12.2/compiler.33191640_0

You can install cuDNN if you are into training Neural Network, check the compability version of cuDNN again: compability matrix

sudo apt install cudnn9

And thats all you need.

Bonus:

1. Installing Pyenv in the Fresh Ubuntu 24.04

Yes, in Ubuntu right out of the box, Pyenv installation won't work. You need to add following Prerequists:

sudo apt install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

then

curl https://pyenv.run | bash
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc^Jecho '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc^Jecho 'eval "$(pyenv init -)"' >> ~/.zshrc
exec zsh
sudo apt install python3-pip
pyenv install 3.13
pyenv install 3.12
pyenv install 3.11

and so on ..

2. Installing Virtualenv in the Fresh Ubuntu 24.04

Pipenv

sudo apt install software-properties-common python-software-properties
sudo add-apt-repository ppa:pypa/ppa
sudo apt update
sudo apt install pipenv

Poetry:

curl -sSL https://install.python-poetry.org | python3 -

3. Checking Cuda with Pytorch Script.

Here is a small script to test the GPU and the packages. You need to install Torch, Torchvision , Numpy to run the script. You can use the latest packages for this.

import torchvision
import torch
import traceback

tcuda: torch.cuda = torch.cuda
print(f"How many gpu in the server ? -> {tcuda.device_count()}")
print(f"can torch access {tcuda.device_count()} GPUs? :> {tcuda.is_available()}")
print(f"Attempting to access GPU device name : {tcuda.current_device()} ....")


try:
    model = torchvision.models.detection.retinanet_resnet50_fpn(pretrained=True)
    model.cuda()
    model.eval()
    x = [torch.rand(3, 300, 400).cuda(), torch.rand(3, 500, 400).cuda()]
    predictions = model(x)
    print(predictions)
    print(f"is torch gpu is initialized ? :> {tcuda.is_initialized()}")
    print(f"torch cuda memory summary :> {tcuda.memory_summary()}")
    print(f"memory reserved {tcuda.memory_reserved()}")
    print(f"memory allocated {tcuda.memory_allocated()}")

except Exception as e:
    print(e)
    print("No CUDA available")
    traceback.print_exc()

4. setting up the Terminal

ZSH

Install Some cool Nerd Fonts

Install zsh and oh my zsh and Terminator

sudo apt install zsh
sudo apt install terminator
sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

Install Oh my zsh plugins

git clone https://github.com/zsh-users/zsh-autosuggestions.git $ZSH_CUSTOM/plugins/zsh-autosuggestions
git clone https://github.com/zsh-users/zsh-syntax-highlighting.git $ZSH_CUSTOM/plugins/zsh-syntax-highlighting
git clone https://github.com/zdharma-continuum/fast-syntax-highlighting.git ${ZSH_CUSTOM:-$HOME/.oh-my-zsh/custom}/plugins/fast-syntax-highlighting
git clone --depth 1 -- https://github.com/marlonrichert/zsh-autocomplete.git $ZSH_CUSTOM/plugins/zsh-autocomplete

Install Powerlevel10k

 git clone --depth=1 https://github.com/romkatv/powerlevel10k.git ${ZSH_CUSTOM:-$HOME/.oh-my-zsh/custom}/themes/powerlevel10k

Update this in the ~/.zshrc

ZSH_THEME="powerlevel10k/powerlevel10k"
plugins=(git zsh-autosuggestions zsh-syntax-highlighting fast-syntax-highlighting zsh-autocomplete python)

and update the zshrc and setup the theme

nano ~/.zshrc
exec zsh
p10k configure

Atuin

curl --proto '=https' --tlsv1.2 -LsSf https://setup.atuin.sh | sh
atuin login
source $HOME/.atuin/bin/env
atuin login
atuin sync

Handle SSH

I prefer to run multiple ssh key in a single machine and it is handled by a ssh config file.

cat config

#Default GitHub
Host github.com
  HostName github.com
  User git
  IdentityFile ~/.ssh/id_rsa

Host github-mac
  HostName github.com
  User git
  IdentityFile ~/.ssh/id_rsa_personal

Host gitlab
  HostName gilab.com 
  IdentityFile ~/.ssh/id_rsa_personal

id_rsa could be your one key and id_rsa_personal could be your personal / other github user.

cd ./ssh
sudo cp /home/shekhar/Downloads/ssh/. . 
ssh -T git@github.com
ssh-add id_rsa
ssh-add id_rsa_personal

Case study: Automated Image Clustering for E-commerce company

2024-05-04T13:49:57Z

Background

In this case study, Artificial Intelligence based system is used to take buisness decisions. The study is based on a e-commerce company which ingest lots of data like images, quarterly reports, invoices, customer feedbacks, customer reviews with images, etc. It would be great if this large amount of data is automatically categorized or grouped together. Previously, a machine learning team in the company had created a model to classify the input data. But the nature of the e-commerce data, new product launches, new customer feedbacks, etc. are not predictable. Thus, the model needs to be retrained frequently. The company is looking for a solution that can automatically perform and help the analyst to get the insight from the unstructured data.

Image Clustering

Lets dive in the Technical side of the case study. There are multiple paradigms of supervised machine learning, such as classification, object detection, and segmentation, among others. There are many robust supervised algorithms; however, to achieve high accuracy, we need labeled data, which is costly and time-consuming. Clustering is an unsupervised learning technique that can be applied to unlabeled datasets. It is one of the most interesting and challenging problems in the field of computer vision. The goal of image clustering is to group similar images together in the same cluster. There are also other forms of machine learning, such as semi-supervised learning, reinforcement learning, and transfer learning, which are not covered in this article. The main goal of the article is to explore the image clustering technique beyond its traditional use cases.

Image clustering: View from Professional settings

In real world scenario, a company could ingest images from multiple sources. The image sizes / format , interest point in the image are unknown. To fetch the insight from the unstructure settings is really hard and requires alot of regular hand picked tuning. Thus, Many companies moves towards supervised learning where they sample the data and label them and train a supervised model to get the insight.

However, the unsupervised learning is also a good choice when the data is unpredictable and the labels are not available. The image clustering is one of the unsupervised learning technique which can be used to cluster the images based on the similarity and thus using this in the automated workflow to get the insight from the unstructured data.

Methodology

Image clustering systems can be divided into two main categories:

Image feature Generation Image pixels are stored as matrics in the computer. The size of image matrics varies based on the image size and format. Thus Image pixel couldnot be used directly to custering algorithm due to its size and variability. Image features can be generated using various methods like neural networks, computer vision algorithms like SIFT, SURF, ORB, etc. Sometimes feature reduction techniques like PCA, LDA, etc are used to reduce the dimensionality of the features.
Feature clustering After a feature is generated, the clustering algorithm is applied to the features to cluster the images. The clustering algorithm can be K-means, DBSCAN, Hierarchical clustering, etc.

for the sake of this study, we are using a small dataset of brand logos. However in the study properitery e-commerce data [ logos from pdfs/ document ] are used.

Note: It really depends on the company data, sometimes the whole image are used to generate the features and sometimes the interest points are used to generate the features. Segment Anything model can be used to generate the interest points. Here, an invoice is parsed by the model when the text prompt is logo and the bounding box over the Tesla logo is generated.

Model Web Demo: Segment Anything

Image Feature Generation

The latest trend of Attention based models outperforms the traditional computer vision algorithms. The attention based models like ViT, ResNet, EfficientNet, etc are used to generate the features. Likewise, there are multi model models like CLIP, DALL-E, etc which are used to generate the features. Using the multi model models, the features can be used to compared with the possible text data and thus can be used to generate the insight from the images.

for the example of the image clustering, we will be using the openAI's CLIP model to generate the features.

from transformers import AutoProcessor, AutoTokenizer, CLIPModel

model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
processor = AutoProcessor.from_pretrained("openai/clip-vit-large-patch14")

a good alternative to the CLIP model is the VIT model.

from transformers import ViTImageProcessor, ViTForImageClassification

processor = ViTImageProcessor.from_pretrained('google/vit-base-patch16-224')
model = ViTForImageClassification.from_pretrained('google/vit-base-patch16-224')

Resource to browse the models for feature generation: Huggingface

Once the model is loaded, we can generate the features using the model.

with torch.no_grad():
      inputs = processor(images=batch_images, return_tensors="pt")
      image_feature_data = model.get_image_features(**inputs)

It is important that the processor should be feed in the batch image data. since the the underlying model is trained on the batch image data. Thus batch image helps to achieve better throughput for generating the features.

Image clustering Algorithm

Once the features are generated, the clustering algorithm is applied to the features. The clustering algorithm can be K-means, DBSCAN, Hierarchical clustering, etc. Here, we choose Faiss for the clustering as it has both CPU and GPU support.

import faiss

n_centroids = 5 # number of clusters
n_iter = 100 # number of iterations
verbose = True
feature_data_dim = feature_data.shape[1] # dimension of the feature data
kmeans = faiss.Kmeans(feature_data_dim, ncentroids, niter=niter, verbose=verbose)
kmeans.train(feature_data)

Lets see How it creates the cluster.

The image clustering result doesnot make sense. lets update the hyperparamter and create the clustering again with different number of clusters.

n_centroids = 35 # number of clusters

The clustering result is much better.

The cluster result is better than the previous as we increased the number of clusters. The number of clusters can be determined based on the domain knowledge of the dataset or using the metrics.

Image clustering Metrics

Finding the best cluster size is a challengine task. The perfect cluster size can be predicted if some domain knowledge of the dataset is available. In the absence of domain knowledge, we can use the metrics to find the best cluster size.

Image clustering can be compared based on two types of metrics.

Extrinsic metrics: Extrinsic metrics are based on the ground truth labels. Some of the popular extrinsic metrics are:
- Adjusted Rand Index (ARI)
- Normalized Mutual Information (NMI)
- Fowlkes-Mallows Index (FMI)
- Homogeneity, Completeness, and V-measure
- Precision, Recall, and F1-score
- Confusion matrix

However, we are into the unsupervised learning, we will not be using these metrics.

Intrinsic metrics: Intrinsic metrics are based on the clustering results. Some of the popular intrinsic metrics are:
- Silhouette score
- Davies-Bouldin Index
- Calinski-Harabasz Index
- Gap statistic
- Elbow method

Lets see how we can use these metrics to evaluate the clustering results.

elbow method is used to find the best cluster size.

The elbow method is used to find the best cluster size. The best cluster size is the point where the curve starts to flatten. In the above image, the best cluster size is 35/40.

Similarly, other metrics can be used to evaluate the clustering results.

The blog only highlights the specfic part of the image clustering system. All the necessary code snippets are provided to get started with the image clustering. Many software enhancement and glue code are skipped for the sake of the article.

Result

A image clustering system is created and implemented in the system and the data is forwarded to the analyst for the further analysis. The system is able to cluster the data based on the cluster and thus necessary action can be taken much faster. As the system is automated, many documents gets labelled eventually and thus labelling the cluster is much easier than labelling the whole dataset.

Data Sourcing for Machine Learning

2024-04-07T19:53:30Z

Terraform with AWS & Python

2023-11-20T21:43:18Z

This is based on a workshop on Pycon Ireland 2023 given by : Pavlo Tishkin

Setup AWS account
Setup AWS CLI
Setup Terraform
Setup Python
Setup AWS credentials
Setup AWS S3 bucket
Setup AWS Lambda function
Setup AWS CloudWatch event
Setup AWS IAM role
Setup AWS IAM policy
setup lamda function

<< updating soon>>

Installing Luminance HDR on Ubuntu

2023-09-13T10:43:23Z

How to install luminance HDR in ubuntu

install dependecies

sudo apt install -y qtcreator qtbase5-dev qt5-qmake cmake libexiv2-dev libtiff-dev libraw-dev libpng-dev libjpeg-dev libopenexr-dev libfftw3-dev libboost-all-dev libcfitsio-dev libgsl-dev

This command installs the following dependencies:

QT5 development tools
exiv2 library
libtiff library
libraw library
libpng library
libjpeg library
libopenexr library
fftw3 library
libboost library
cfitsio library
Gnu Gsl library

sudo apt install qttools5-dev sudo apt-get install qttools5-dev-tools sudo apt install libqt5webkit5-dev sudo apt-get install libqt5svg5-dev sudo apt-get install libeigen3-dev

install luminance hdr

git clone
cd luminance-hdr
mkdir build
cd build

cmake \                                                                                                                                                        5s 12:07:11 PM
-DCMAKE_BUILD_TYPE="Release"  \
-DCMAKE_INSTALL_PREFIX="$HOME/programs/lhdr" \
..

make
sudo make install