> For the complete documentation index, see [llms.txt](https://doc.eimtechnology.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://doc.eimtechnology.com/kt-series-project-based-learning/ai-with-hardware/lcd-control-by-hand-gesture.md).

# LCD control by Hand Gesture

<table data-view="cards"><thead><tr><th></th><th></th><th></th><th data-hidden data-card-cover data-type="files"></th></tr></thead><tbody><tr><td><strong>The Artificial Intelligence Learning (AI) + Machine Learning Kit</strong></td><td><a href="https://www.eimtechnology.com/collections/all-products/products/learn-convolutional-neural-networks-computer-vision-opencv-artificial-intelligence-learning-ai-and-machine-learning-kit?variant=50057188278591">Available Now</a></td><td></td><td><a href="https://2117864350-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FHy5TN04jkEM9s96yIKLT%2Fuploads%2FN9oSDFjrdMP1cSPaPx5H%2Fai-preivew.png?alt=media&amp;token=03e8de6f-04ec-43ed-a185-702092ad7bce">ai-preivew.png</a></td></tr><tr><td><strong>Instruction and resources</strong></td><td><a href="https://github.com/eimtechnology/AI-KickStarter">GitHub</a></td><td><a href="https://discord.com/invite/urh8hW2YFa">Discord</a></td><td><a href="https://2117864350-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FHy5TN04jkEM9s96yIKLT%2Fuploads%2F5rAUpcj5DiIvF80igdxb%2FAI_banner_1600_x_900_px.webp?alt=media&amp;token=d7011686-01b3-43d4-b7d3-629cda93bf43">AI_banner_1600_x_900_px.webp</a></td></tr></tbody></table>

## Project Overview

In this project, we build on the finger detection system to create a practical screen control interface. By enhancing how hand landmark data is processed and utilized, we’ve evolved from basic finger counting to a robust system capable of sending precise commands to external displays. The original finger detection project focused on identifying the number of extended fingers using MediaPipe hand landmarks, providing a solid framework for accurate gesture recognition and real-time analysis. These advancements take that foundation further, enabling seamless and intuitive screen control.

## Hardware

The following components will be prepared in the kit for this project:

* A breadboard
* A STEPico microcontroller
* Multiple dupont wires
* A ST7789 LCD display
* A USB cable

<figure><img src="https://2117864350-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FHy5TN04jkEM9s96yIKLT%2Fuploads%2FocYWMjTp4gDUf17CA20g%2FFigure%205.0.2.jpg?alt=media&amp;token=f37137c0-9199-4ffc-9ae8-3518cf9049e9" alt=""><figcaption><p>Breadboard, STEPico, ST7789 display, jumper wires, and USB cable</p></figcaption></figure>

## Tutorial

The key advancement in this project lies in data processing and output formatting. Instead of simply counting extended fingers, we now track each finger's state individually and convert this information into binary signals. This modification enables our system to interface with external devices through serial communication, making it suitable for screen control applications.

<figure><img src="https://2117864350-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FHy5TN04jkEM9s96yIKLT%2Fuploads%2FPkzy8xE40b6WknevFuAL%2F02.jpg?alt=media&amp;token=764bd5c5-61cc-4faf-82ba-66239ded1b3d" alt=""><figcaption></figcaption></figure>

## **The Learning Part**

Output Processing

* Individual finger state tracking
* Binary signal conversion&#x20;
* Structured data formatting for serial communication

System Integration

* Serial communication protocol implementation
* Hardware interface configuration

<figure><img src="https://2117864350-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FHy5TN04jkEM9s96yIKLT%2Fuploads%2FxUSNhICIrZJ4y48Evqrj%2FFigure%205.0.1.png?alt=media&amp;token=74c07267-7b70-4455-9f38-c753176a0a4a" alt=""><figcaption><p>Project data flow</p></figcaption></figure>

### Hardware

The display for this and following projects is ST7789. This display is a type of thin-film-transistor liquid-crystal display (TFT-LCD) controller driver developed by Sitronix.

<figure><img src="https://2117864350-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FHy5TN04jkEM9s96yIKLT%2Fuploads%2FjjMgrDe4qwDMBCYh9FHF%2FFigure%205.2.1.jpg?alt=media&amp;token=eabff27a-f30a-48e7-b106-e9ca3c9d60b3" alt="" width="563"><figcaption><p>A ST7789 display</p></figcaption></figure>

To wire your STEPico to your ST7789, follow the steps to connect them on breadboard:&#x20;

1. Power Connections:

* Connect the VCC pin of the display to the 3.3V power pin on STEPico.
* Connect the GND pin of the display to any GND pin on the STEPico.

2. SPI Interface:

* Connect the SCL pin on the display to GP18.
* Connect the SDA pin on the display to GP19.

3. Control Pins:

* Connect the RES pin on the display to GP16.
* Connect the DC pin to GP21.
* Connect the CS pin to GP17.

4. Backlight:

* Connect the BL pin to GP20.

<figure><img src="https://2117864350-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FHy5TN04jkEM9s96yIKLT%2Fuploads%2F9Z1WGkWK9JBsoI7d3XhP%2FFigure%205.2.3.png?alt=media&amp;token=1ac09a60-197f-4e63-ac2a-985a4775276c" alt=""><figcaption></figcaption></figure>

**System Dependencies and Library Integration**

To implement hardware control and display management, various libraries are needed:

```python
from machine import Pin, SPI
import framebuf
import st7789
from sys import stdin
import uselect
```

Here's a brief overview of the required libraries:

* **machine**: Provides `Pin` and `SPI` for hardware control.
* **framebuf**: Manages display buffers for rendering.
* **st7789**: Offers display-specific commands.
* **sys (stdin)** and **uselect**: Facilitate data reception.

Note: The ST7789 driver isn't built into MicroPython by default. Use compatible external libraries like Russ Hughes’ fork of the st7789py\_mpy module for enhanced performance and flexibility. Access it here:&#x20;

<https://github.com/russhughes/st7789py_mpy/blob/master/lib/st7789py.py>

After you complete the project, it looks like following:

<figure><img src="https://2117864350-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FHy5TN04jkEM9s96yIKLT%2Fuploads%2FQewOQaTq5vBXcpE2Bupo%2FFigure%205.2.7.jpg?alt=media&amp;token=4aa7c7ad-7965-4ccd-85d8-8248a9442697" alt=""><figcaption></figcaption></figure>

### Video Walkthrough

{% embed url="<https://youtu.be/7ZuSH5Mq3RU>" %}