Let us consider a smooth region G made of the upper half of an ellipse and completed by a curve which connects the foci p and q by a straight segment and form two lobes as in Figure 1. We will refer to this mushroom like shape as the Penrose's mushroom.
Let us consider, without loss of generality, a ray r leaving the left lobe from point a (see figure 2). Ray r will hit the boundary, say at point b, above line pq. It will then be reflected to hit the boundary again at point c. Let angle ABC be the angle made by point a, b and c. Note that angle ABC is the sum of the incident angle and the reflected angle made by ray r at point b.
Figure 2 - Evolution of ray r in Penrose's mushroom
Imagine now a ray u leaving focus p and hitting the boundary at point b. By (1), we know that ray u will be reflected back to the second focus q. Let the angle PBQ be the angle made by p, b and q.
The angle of incidence of ray r at point b is always bigger than the angle of incidence of ray u at point b. To be smaller, ray r would have to emerge to the right of focus p which is physically impossible since ray r starts in the right lobe. By (2) we conclude that angle ABC is always greater than angle PBQ.
Therefore, no matter in what direction ray r emerges out of the left
lobe of G it will never be reflected onto line pq. By symmetry, this also
holds for the right lobe.
We can now build a region H that will not be illuminable from any point located in either lobe. To do this, we add a little dimple below the line pq of region G (figure 3 ). Because no ray emerging out of the lobe can be reflected on line pq, it will never enter the dimple put on this line. Therefore, region H is a not an illuminable region. This provides a negative answer to question 1 of the illumination problem for smooth surfaces.
Connecting two Penrose mushrooms as shown in figure 4, we get a region that is not illuminable from any point. Any ray starting in a region below pq will not illuminate regions labelled with an S. Similarly, any ray starting in a region above p'q' will not illuminate regions labelled with S'. No matter where you put a light source in a room of this shape, there will always remain some dark areas. In other words, this region is not a directly illuminable region.
The illumination problem as stated traditionally considers one light source illuminating a bounded region. Now that we have found some regions that are neither illuminable nor directry illuminable by one light source, we may wonder how many light sources are needed to illuminate such regions. It takes three light sources to illuminate Region H since it is the union of three starlike set. Rauch shows that it is possible to use a finite number of points to illuminate any bounded and smooth region. In order to prove this, he uses the following claim "Any bouded region G lying on one side of its C1 boundary is the finite union of starlike sets". He also provide an example where a bounded region G, smooth in every point except one, gives raise to a region that is not illuminated from any finite set of points (figure 5).
5 - Region Not Illuminable
from any finite set of points
The illumination problem may be changed if the region is allowed to have holes or if constraints are added such as limiting the position of the light source to the boundary.