Gathering Diablo 2 Resurrected Gem Data Yolo v5 Automated Box File Generation and Factories (part 3)

I have reached the next major milestone for my Diablo Gem sorter project. The image below shows my latest results. Try and look past the overlapping text. There is currently 38 possible classes in my project all classes were built using my trained tesseract OCR. The box file that was built for this image was created with no human interaction.

You can see the process leading up to these results in the following posts part 1 and part 2.

Last time I had built logic to determine the size of objects in inventory. Notice that the Yolo training data is properly accounting for items that take up more than one square in inventory including the two books and Horadric Cube below.

My current class list that was generated passing cleaned images into my trained Tesseract OCR is as follows. Using some looping logic I defined the order of the gems. I also had to convert all Jewel names to have them fall into a single class for training.

0_CHIPPED_AMETHYST
1_FLAWED_AMETHYST
2_AMETHYST
3_FLAWLESS_AMETHYST
4_PERFECT_AMETHYST
5_CHIPPED_DIAMOND
6_FLAWED_DIAMOND
7_DIAMOND
8_FLAWLESS_DIAMOND
9_PERFECT_DIAMOND
10_CHIPPED_EMERALD
11_FLAWED_EMERALD
12_EMERALD
13_FLAWLESS_EMERALD
14_PERFECT_EMERALD
15_CHIPPED_RUBY
16_FLAWED_RUBY
17_RUBY
18_FLAWLESS_RUBY
19_PERFECT_RUBY
20_CHIPPED_SAPPHIRE
21_FLAWED_SAPPHIRE
22_SAPPHIRE
23_FLAWLESS_SAPPHIRE
24_PERFECT_SAPPHIRE
25_CHIPPED_SKULL
26_FLAWED_SKULL
27_SKULL
28_FLAWLESS_SKULL
29_PERFECT_SKULL
30_CHIPPED_TOPAZ
31_FLAWED_TOPAZ
32_TOPAZ
33_FLAWLESS_TOPAZ
34_PERFECT_TOPAZ
35_JEWEL
36_HORADRIC_CUBE
37_TOME_OF_IDENTIFY
38_TOME_OF_TOWN_PORTAL

I have also included other color Jewels for this part of my project that the yolo network will be trained on. This is important since the jewels look similar to perfect gems.

The object oriented structure of my program is really paying off. Making the leap to building the Yolo text files was incredibly fast since I had already defined the Rectangle class and only needed to normalize the data based on screen size.

The program automatically generates box file data based on the objects that are currently visible on the screen. The data is represented by the class name index then normalized values for the x and y values for the center of the box as well as width and height.

Here is an example box file that was created. There are 95 entries in this box file. At any given time there could be up to 140 items on the screen.

The next step for the process is to build a new set of virtual containers using the container factory I built. That factory will build containers in memory without items attached.

I have access to this new set of containers in three lines of code which could be combined into a single function if needed.

Notice these new containers have all of the groundwork already laid out for them. But do not have items attached.

I can then randomize placing items into the new set of containers. The old container which I call source containers will be the current "real world" reality and the new destination container will be the goal. I will then move the items in the container in the real world to match the virtual container. Once the items are in their place we can build a new set of Yolo training data which will include the screenshot as well as the box file. This way I can create as much training data as needed without having to do any labeling for my project. Imagine labeling an image that could have 140 items on the screen while keeping track of 39 classes.

The ultimate goal of course once the Yolo network is trained will be to use my trained network to take one image of the screen and quickly locate all gems. This will eventually lead me to sorting gems as well as using reinforcement learning for combining gems in the Horadric Cube.