How the neural network helps to search for the best advertising spots on the streets: LUN case

How did the FLAIR team appear?

LUN began as an advertising search engine: it collected and processed information from other sites. “We use various kinds of information processing algorithms starting from the first day of the project’s existence, that is, for more than 10 years. Back in 2008, we implemented the first ad analysis system that used simple “if that” rules to find duplicate ads,” says Volodymyr Kubitskyi, the head of the team that works on AI-solutions. Fully neural networks were started to be introduced in LUN only in 2014, since then its number has exceeded two dozen. To support existing neural networks and work on new AI/ML-algorithms in the company, they created the Flatfy & Lun Artificial Intelligence Researchers team, FLAIR.

Outdoor advertising AI

They remembered about the developments of the team only this summer. In June, the company conducted an advertising campaign called “Do they build or only advertise?”. It promoted a new feature on the site – checking the reliability of the developer when buying an apartment in a new building. Within the framework of the advertising campaign, advertising boards were also planned, and it was necessary to find locations for them. According to Andriy Mima, co-founder of LUN, usually in such cases advertising agencies work according to one pattern: they send a list of several hundred selected advertising spots from which you need to choose, for example, 50. “I call it an “advantage”, like you can choose to your own taste.  And to make a choice, you only have the address of the board, its old low-quality photo and a strange OTS figure. And if you ask to make a choice for you, then they choose the most beautiful ones “by photo”— says Mima.

The company has decided to act differently and asked the agency to shoot a video with the boards, so that it was possible to determine whether the advertisement was clearly visible, whether tree branches or other boards were not obscuring. Then the company created a table with parameters for each board, which would make it easier to decide on each location. The main parameters for each board were identified as:

• board size;
• real picture (that shows whether the trees obscure the sign);
• time of non-visibility;
• competition with other boards in sight.

This label was created by viewing the video prepared by the agency, and manually writing out all the numbers. As a result, they managed to select 50 boards from the proposed 500 ones. However, the team had to spend several weeks on watching the video and filling in the table. Therefore, the next time the process was decided to automate:

• in order to analyze any number of ad slots,
• to increase the speed up such analysis.

At this stage, work with neural networks began. Just at the same time, the FLAIR team worked on a system for determining watermarks in photos of apartments, in order to select pictures without such signs for the title photo in search results. As a result, a system of localization of the object in the photo was created: neural networks showed several examples of what they would like to find in the photo, and then it learned to search basing on the new data itself.

“Just at that moment, the marketing guys were busy selecting sites for boards. We saw each other in the corridor, someone spoke a word about how they learned to detect watermarks and can generally find any object on any photos. So, the idea was to try to analyze the video from the registrar to find the boards,” says Volodymyr Kubitskyi.

According to him, the solution to search for advertising space was created from scratch. “For the implementation of the neural network, the tensorflow framework from Google was used to simplify the solution of the localization problem. In the transition from localization to tracking (localization is single-frame, and the board is contained in many video frames and should not be lost) they wrote their own post-processor,” he says.

How a neural network works

As a result, they managed to create a neural network that can accept video of almost any quality, in any light and at any shooting angle, and determine all ad slots on it, calculate the specified parameters for them: size, visibility time and competing boards that are located beside. This data is entered into a table, and then the marketing team can select any data evaluation formula and filter the results by it. For example, in the summer campaign, the formula looked like this:

Effect = Board area x Time in visibility zone / Number of other boards on the screen.

According to Kubitskyi, the neural network works this way:

• The input video is streamed into a set of frames (usually 24 frames per second) using OpenCV.
• Then the neural network begins to work with the Faster R-CNN architecture, trained to find the boards on the frames: the network gives the answer for each frame in the form of the coordinates of the found boards, if any.
• After this, the post-processing stage begins, where you need to reassemble the video from the frames and understand that the same board is tracked from frame to frame, and a new board is not tracked every time. Here, computer vision algorithms are connected to compare images and a Kalman filter for working with the found coordinates from frame to frame. Since the neural network has learned to work with multiple boards simultaneously, it calculates whether competing boards are in sight.
• All of this data is summarized in the report

According to Andriy Mima, the capabilities of such system can be broader than the current prototype. For example, you can show specific boards to the system and it will automatically find them on video, select them and make an assessment on the parameters. This can be useful for monitoring ongoing ad campaigns.

“I read somewhere that there are 1,600 km of roads in Kyiv. With an average speed of 16 km/h, you can drive around all of them with 3 cars in 4-5 working days, and if you remove the small roads, then, most likely, this task will still be a couple of times easier and cheaper. However, what to do with 100 hours of video then? It is 360,000 seconds, where you need to stop every 5-10 seconds, note the time, count the area and write it down in the table? It is necessary to measure about 50,000 structures, and it seems that this can take several months. It’s a huge amount of work that we have just automated,” says the co-founder of the company.

If it turns out that there is a demand for such technology among advertising agencies or sales houses, the company is ready to sell or license it.