Gathering Diablo 2 Resurrected Gem Data for my Yolo Network (part 1).

My next Diablo project will require some specialized data. I need images of Gems. But I am way too lazy to move all of these gems around myself. So I will be building a python script that moves gems and takes screenshots.

In this game each type of gem has different qualities. My goal will be to train a Yolo network to identify gems. Then once I have their coordinates I want to build a reinforcement learning neural network to combine gems for me. The neural network will receive rewards when gems are transformed to the next rank. Note I will not be telling it about the ranking system directly it will have to train itself.

I am using Photoshop guides to calculate pixel locations of the various boxes on the screen.

I now have the ability to using Python run my cursor over all of the squares in my stash. Here I used OpenCV to color in the locations that my cursor can move to. I will use this data to mix up the gems in my stash. The goal is to build data for my Yolo network without me having to label images. Essentially I am building box files for Yolo without having to edit them.

Using Tesseract to find text data from screenshots. (Partial screenshot taken using Pyautogui)

Initial results were a bit rough. The results were usable but would require a bit of work to clean up.

As you can see below it did find a chipped diamond in the text. But alot of garbage with it. I decided to train the Tesseract OCR to understand diablo 2 font.

i

4 Re AuSL E SHARED SHARED |

7 T

Vey oe) fee/aefasj mlm) a] |

CHIPPED DIAM@ND

CAN BE INSERTED INT® S@CKETED [TEMS

i

Hy

Hil

WEAPONS: *28% DAMAGE T® UNDEAD

ARMOR: +20 Te ATTACK RATING

el Se HeLms: *2Q@ Te ATTACK RATING

= ter

(yy: SHIELDS: ALL RESISTANCES #6

| Os CTRL * LEFT CLIcK Te Meve

‘|

Py % ay ros

STG (Si 25176 Lis

—_

Training Tesseract 5.0 OCR to understand Diablo 2 font.

Exocet is the official typeface for the Diablo franchise. It is the font for the title and most of the text throughout the entirety of Diablo and Diablo II. The font was originally designed by British graphic designer, typographer and lecturer Jonathan Barnbrook, and named after the type of rocket with the same name.

I now am training tesseract to use this font type hopefully it will generate better results.

To achieve this I used the windows snippet tool to quickly load images to my clipboard then using python I programmatically saved the image data to disk. This was achieved by taking the image data and turning it into NumPy arrays so that they can be quickly compared. If anything changed I would save the new image to disk allowing me to quickly build some one line training data.

I wrote a script that renamed my png files to eng.exocet.exp0.png: eng for english, my font was exocet, exp + number was the naming convention.

I used JTessBoxEditor to fix the box files. The OCR had quite a bit of trouble with the letter O since it was very unique. It also struggled in general with + signs.

I only did 49 images for this part because I wanted to be sure that I could get through an entire training process. If everything works out I will add more to my training set.

I have now successfully trained a model for tesseract. I used text2image to build the training set using text that was scraped off of a diablo website.

My first attempt was not as successful because I allowed lowercase letters.

(My initial training data)

My final training data used non bold words with capitalized letters.

(Much higher success rate)

Here is an example of the untrained original tesseract results.

Which was far from optimal.

Here is an example of the cleaned data using my trained OCR. As you can see we are getting far better results.

Note the garbage text is explainable. I decided to clean the image using NumPy. Its important to note that I used the where command to find pixels that had a value less than white as well as pixels that were fully white. Looping through these arrays would have taken too much time. Anything less than white I colored white and white I colored black. I wanted the test set to look similar to my training set. My training set has a white background with black text.

Here is an example:

After swapping pixel values I had this.

The small dots from the other gems are the reason I am getting strange text hits between the end of the gem description and the life amount.

Eventually I will detect the item flyout and only send it to the tesseract OCR. For now this feature will have to wait till a later version of my program.

Right now I am getting good enough results to train my Yolo Network. This marks a major milestone for my project. I will now use the OCR as well as the item locations to train my yolo network.

To be continued.

If you know of a business that needs a data scientist to help them in the fields of computer vision or A.I. please send me a message on LinkedIn. I would love to hear about your projects.

My LinkedIn profile: https://www.linkedin.com/in/jason-ismail-46268463/