An IR system can be seen as an open system, which interacts with its environment. The system itself is the object being observed and measured during an experiment. Again, from a system theoretic point of view, the environment corresponds to the rest of the universe. An interaction happens, when "something" passes the boundary between the system and the environment. In classical theory this "something" are typically objects or energy. In the case of an IR system it can be seen as messages.
Hence, Robertson suggested as objectives for IR evaluation to identify the boundaries of the system and to analyse the messages crossing that boundary. In this way, also the environment can be incorporated in the evaluation process, namely via the messages it sends into the system. As one important -- if not the most important -- feature living in the environment of an IR system, he mentioned relevance. After all, it is the user, who eventually decides about relevance.
To illustrate the boundaries of an IR system, Robertson referred to the "IR-Onion", as shown in the figure to the right. In addition to the naive boundary of the IR system, i.e. the one between user and system, it shows several other layers. Just like the layers of an onion, the core of the IR system is wrapped up in the interface/interaction component. Typically the user does not see (nor needs to understand) the internal algorithms and mechanisms of the system, but deals only with the user interface. This is also the afore mentioned naive boundary from a system theoretic point. However, also the user side, i.e. the environment, can be decomposed into several layers. The closest layer is the user's model of the IR system. A user has certain ideas or expectations on how the system operates and reacts. This influences his interaction and behaviour. Next comes the user's model of the information seeking task. Depending on what the user knows or thinks about his information need, he might behave differently. Finally the information need is embedded in a larger problem or task, of which the user might not even be completely aware (Robertson referred to Nick Belkin's ASKs: the anomalous states of knowledge).
The "IR-Onion" reflects, that search is a rather difficult task. Traditionally, that is why there used to be an intermediary between the user and the system (e.g. a librarian). The intermediary helped the users to organize and maybe better understand their search, even if the user eventually had direct access to the search system.
For, evaluation purposes the intermediary between user and system was an important resource. She or he could formulate reference interviews with the user and thereby provide a deeper insight on the users' side, i.e. the environment. Nowadays, the challenge in IR evaluation -- particular for context based systems -- is to get a better picture of what is going on in the user's mind. Interaction events are a good basis, but research still has to make sense of those events.
