Keywords

1 Introduction

1.1 Use Case

We focus on large-scale dynamic light installations. Consider the light installation at 131 South Dearborn, Chicago, the USA [22] as an example of an ideal use case. The installation consists of 925 glass bubbles, and each bubble has its light source. Install 925 individually programmable light sources and ensure that the wiring is right pose a considerable challenge and is prone to mistakes during installation. We aim to automate the light position mapping process with the proposed system.

Another use case we tested was mapping multiple programmable led rings and strips. Instead of manually marking each fixture’s position, we have automated the process. As a result, the setup can be changed frequently to help designers find the ideal configuration and immediately test it. Please watch the experiment video [15] documenting the process.

1.2 Programming Light Show

Without knowing the light positions, we can create only simple light shows, such as changing the light parameters uniformly or creating animations based on noise. To create a more complex light show, we need to distinguish individual lights and know their position.

We can then create a simulation with light sources represented as points in space. See Figure  1. We add virtual objects and animate their position. We can achieve various light effects by detecting the collision of light sources’ positions with a volume of moving virtual objects. We turn on the light source when the light is inside the virtual object and turn it off when it is outside. We can also use multiple virtual objects to create more elaborate animations or map them interactively based on sensor inputs. For example, one can map people’s movement to light intensity in different sectors.

Fig. 1.
figure 1

Schema of light scene animation: Light sources represented as points in space collide with the animated virtual 3D object. Lights inside the object will be turned on.

Full size image

1.3 Light Network Control

Light fixtures are often controlled using DMX512 [10] protocol. Traditionally DMX address (a unique id of the light) is selected using a hardware DIP switch directly on the light fixture. Another option is to use Remote Device Management (RDM [9]) commands from the control software [6]. However, RDM is only available with some DMX light fixtures, so we often rely on manual selection, making changing DMX addresses difficult. According to the technical plan, each light with the appropriate DMX address must be installed in the correct position.

To control the lights, most often, we use the Artnet [2, 22] or sACN [1, 11] protocol - an extension of DMX standard that enables us to control more lights and use network topology and devices such as Ethernet switches to send DMX packets over the network.

In essence, we can control individual lights from a single networked computer. We need an ethernet adapter and Artnet/sACN node to convert the signal to DMX. DMX signal is then sent to DMX driver that maps the DMX values to voltage and current to dim individual lights. See Figure  2.

Fig. 2.
figure 2

Artnet light network diagram (https://www.ledlightinghut.com/artnet-dmx512-converter.html).

Full size image

2 State of the Art

Various systems using a camera to control lights exist. Most of them deal with finding a user or controlling the light to track the user. In most cases, available solutions rely on knowing the light fixtures’ position in space. We are trying to solve a different problem, localizing light sources relative to the camera. Still, some principles can be used for both problems.

Luxapose [14] system used a mobile phone as a camera and modified light fixtures to track a mobile phone’s position. Light sources are modulated to produce pulses of light with encoded position and ID information that can be captured in a single frame exploiting the rolling shutter effect. While we are trying to determine the position of the light fixture, Luxapose is tracking a user. Theoretically, we could repurpose the system and use the known camera position to determine the light position. Unfortunately, the Luxapose system requires light source modification which is not practical in our use case. Moreover, the camera position relative to the light source would have to be known. While feasible, it would require precise measurement on site.

Similar to the Luxapose, a paper by Hossan and Tanvir [13] uses multiple known light fixtures to localize the camera position. We can not use triangulation methods to determine the camera position when using a single camera in our system. In practice finding the precise camera position relative to the light fixture would have to be repeated for all lights as we can not guarantee their position relative to each other. Such manual measurements would completely negate any benefits of automatization. The authors use found blob pixel count to measure the distance from the camera to the light fixture. A similar approach could be adopted while moving the camera back and forth to determine the distance from the camera to the light source. Other approaches [24] use a photo-diode as a sensor installed in the light fixture to track objects based on changes in light propagation. Multiple light fixtures have to be taken into account, or a quadrant photo-diode is used [7]. The camera is no longer needed in these cases, but the light fixtures have to be modified to include the appropriate sensor and communication unit. For example, BlackTrax [8], a commercial solution for motorized lights, uses multiple cameras and an infrared beacon to track the moving target. Craig Hiller developed a system to detect light’s positions and even classify them as a fluorescent bulb, tube, or LED light [12]. The project relies on a Tango tablet [17] aligned with a DSLR camera and IMU in one package. The user walks under the lights to map their position and create a spatial map. Tracking is only possible with sensors being perpendicular to the light fixture, which is fine when detecting ceiling-mounted lights but would fail with vertical or organically shaped installations. The reported error is also not suitable for the light show scenario, with some lights being false positives and others missed during detection.

3 Software

3.1 Programming Environment

The main program is written in Java Processing framework [4]. To acquire the camera stream, we used Gstreamer [23] based library [11]. To perform computer vision tasks, we have used the Java wrapper of OpenCV library [5].

3.2 Light Fixture Detection

To find the light position, we create a frame difference between a frame with all lights turned off and a frame with a single light turned on. Then we threshold the resulting absolute difference to a binary image. We find the contours of the light blobs. Finally, we sort found blobs based on their area. We select the blob with the biggest area and find its centroid. The resulting x and y coordinates are saved and associated with the light DMX address. When we have processed all the lights, we normalize their coordinates and save all the information into a JSON file. Later this file can be loaded into light control software such as Touchdesigner [21], Madrix [3], OpenFrameWorks [20], Processing [4], VVVV [18], and similar software used to create light shows.

3.3 GUI and User Input

We have also developed a GUI for easier usage. We provide multiple options, such as selecting a target IP, setting custom binary thresholds, selecting from multiple cameras, and setting minimal and maximal blob size. Users can also select whether to cycle through all lights automatically or one by one by clicking the button. The user can manually adjust every detected light position with a mouse if needed. Users can also create custom masks to select the area in the camera image that should be ignored during detection. Furthermore, we have also enabled perspective corner pin transformation to be applied to the camera image. Acquiring undistorted camera images is essential to measure uniform distances between light fixtures correctly.

4 Experiment

Fig. 3.
figure 3

Experiment setup - LEDs mounted on the wooden plate

Full size image

4.1 Lights Setup

We have tested our system on a setup with 30 individually programmable LED light sources controlled via the ArtNet network. Lights were installed on 250cm by 125cm wooden plate, all facing the same direction. See Figure  3. Each light is 3.5cm in diameter with 6 LEDs controlled as a single symmetrical light source.

4.2 Camera

We have used the Logitech C922 camera, a widely available and affordable standard web camera with a USB interface. The camera was positioned perpendicular to the wooden plate 272cm away. The camera’s Diagonal Field of View (FOV) is 78\(^\circ \), horizontal FOV is 70.42\(^\circ \), vertical FOV is 43.3\(^\circ \), and Focal Length is 3.67mm. We tested the setup, once using 640*480px camera resolution and once 1920*1080px, so we could determine if the used resolution correlates with accuracy.

5 Results

Table 1. Mean error in the position detection and standard deviation
Full size table

Experiment data is available at Zenodo 6814223 [16]. All installed lights were detected correctly. See the results in Table 1. In the case of 640*480px camera resolution, 1 pixel corresponded to 4.53mm in the wooden plate plane, and the average position error on the horizontal axis was 2.912mm and 2.426mm on the vertical axis. The maximum error on the horizontal axis was 2 pixels, and 1 pixel on the vertical axis with standard deviation 0.678px and 0.507px respectively.

In the case of 1920*1080px camera resolution, 1 pixel corresponded to 2.02mm in the wooden plate plane, and the average position error on the horizontal axis was 2.885mm and 1.947mm on the vertical axis. The maximum error on the horizontal axis was 4 pixels, and 6 pixels on the vertical axis with standard deviation 1.069px and 1.318px respectively.

6 Discussion

We assumed that the higher resolution of the camera and shorter distance to the lights would produce more accurate results as 1cm would be represented by more pixels in the camera image. Resolution does not play a crucial role, as we have not observed a significant accuracy increase when using 1920* 1080px over 640* 480px.

We need to know if particular light is left or right relative to the other light to propagate the animated wave through them correctly. We have successfully obtained correct relative relationships between lights irrespective of the used camera resolution. High absolute accuracy can be beneficial but unnecessary for the light-show creation.

The light source shines directly at the camera and produces a lens flare. Lens flare does not worsen the detection as long as it is uniform in all directions, so we can assume that the light source is in the center.

7 Conclusion

We offer a robust way to automatically detect installed light fixtures positions with a single monocular camera. Furthermore, we save the information with the respective light DMX address in both machine and human-readable formats to be used in various light control software. Our proposed automatic light detection method can be thought of as a proof of concept with clear time and cost-saving benefits. More importantly, it opens the way for new stage light design possibilities. More testing in real-world scenarios is needed to further verify the presented system’s viability.

8 Limitations

An unobstructed view of all light sources is needed to detect their position correctly. Position mapping is possible only if we find a view in which the light fixtures do not overlap. For example, it would be challenging to automatically get the positions of light fixtures organized in a 3D helix shape.

The best use case for our system is the light sources positioned in a single layer. For example, lights are hanging from the ceiling.

The problem occurs when lights are organized in multiple layers. Such as multiple lights on a single string or rod beside each other. It might not be practical to use our method in such a case.

Reflections cause another limitation. When reflective surfaces surround the light sources, it can create several false-positive hot spots in-camera images that might be hard to distinguish from the actual light source. For example, a chrome-plated ceiling has high reflectivity. In this case, reflections have to be eliminated before detection.

9 Future Research

We can improve the tool to enable 3D position mapping. We can use the binocular stereoscopic camera to calculate the distance from the camera to the light. Alternatively, we can reposition a single monocular camera to acquire two points of view to achieve the same result. We could further improve the usability by merging multiple cameras to cover a larger area to map large-scale installations. Another approach would be to enable sequential mapping. After mapping one place, the user would physically move the camera to cover another neighboring area. The relative position change of camera origin could be calculated by the standard SLAM method [19].

10 Declarations