Learning from Hamming Why the Big Problems Matter Peter J Bentley Google: “you and your research hamming”

Richard Hamming Hamming distance Hamming code Hamming matrix Hamming window Hamming bound

Richard Hamming: you and your research I saw Feynman up close. I saw Fermi and Teller. I saw Oppenheimer. I saw Hans Bethe: he was my boss. I saw quite a few very capable people. I shared an office for a while with Shannon. I became very interested in the difference between those who do and those who might have done. One of the characteristics of successful scientists is having courage. Once you get your courage up and believe that you can do important problems, then you can. If you think you can't, almost surely you are not going to.

Richard Hamming: you and your research I went over and said, ``Do you mind if I join you?'' They can't say no, so I started eating with them for a while. I started asking, ``What are the important problems of your field?’’ And after a week or so, ``What important problems are you working on?’’ And after some more time I came in one day and said, ``If what you are doing is not important, and if you don't think it is going to lead to something important, why are you working on it?’’ I wasn't welcomed after that; I had to find somebody else to eat with!

Search Based Software Engineering ``What are the important problems of your field?’’

Search Based Software Engineering “SBSE could become a coherent field of activity that combines the expertise and skills of the Metaheuristic Search community with those of the Software Engineering community.” “Open problems: Stopping Criteria, Landscape Visualisation, Characterizing the Software Engineering Search Spaces, Human Competitive Results” “Future developments: Multi Objective Optimization, Interactive Optimization, Sensitivity Analysis, Hybrid Optimization Algorithms, On Line Optimization” “Benefits: Scalability, Robustness, Feedback and Insight”

Richard Hamming: you and your research ``What important problems are you working on?’’

Search Based Software Engineering “Testing, Multiobjective optimisation, RE, program slicing, constraint satisfaction” BUT for multiobjective optimisation: “Once the real world scenario has more than about 10 stakeholders, the visualisations we propose and the animations we introduce will become hard to read. Furthermore, the search-based optimisation techniques we use do not, presently, scale well beyond four or five objectives.

Example big(ish) problem: Scalable Multi-objective Optimisation The cause? An unhealthy obsession with the “Pareto Optimal Front”:

Example big(ish) problem: Scalable Multi-objective Optimisation Why not address this problem? Let’s do it now.

Pareto Vilfredo Pareto Italian engineer, sociologist, economist, and philosopher. Pareto's biographer: "In the first years of his rule Mussolini literally executed the policy prescribed by Pareto, destroying political liberalism, but at the same time largely replacing state management of private enterprise, diminishing taxes on property, favoring industrial development, imposing a religious education in dogmas”. Karl Popper called him the "Theoretician of Totalitarianism"

Pareto’s words Order is "the disposition of a number of things with reference to some specified end and proportioned to a desired effect.” "Propriety {convenance) approximates order itself. It is a relationship of conformity [What is this conformity?] among several things, each of which is in itself promotive of the preservation and perfection of the other [And what this perfection?], and does its share in maintaining it in a good and advantageous estate.” Good for whom? Advantageous for whom ? A poison that leaves no trace "is promotive of the preservation and perfection" of the man who wants to murder a neighbour, and maintains him in an estate that is "good and advantageous" for him; but it cannot be said to be "promotive of the preservation and perfection" of the victim, or that it maintains the victim in a "good and advantageous estate." Pareto, Vilfredo (1935). The Mind and Society [Trattato Di Sociologia Generale]. Harcourt, Brace.

Pareto optimality is from economics “Pareto realized that it was not necessary to know how much a person valued this or that, only that he preferred X of this to Y of that. Pareto said ‘good’ cannot be measured.” “He replaced it with the notion of Pareto- optimality, the idea that a system is enjoying maximum economic satisfaction when no one can be made better off without making someone else worse off.” Vijay K. Mathur, "How Well Do We Know Pareto Optimality?" Journal of Economic Education 22:2 (1991) pp

Pareto optimality and genetic algorithms “At the end you have a set of solutions - a set of bridge designs, or factory production schedules, or whatever. So, why do multiobjective at all? Well the difference is potentially twofold. first, via the multiobjective search you may have found a better result, in terms of the single-objective score… Second you are presented with a diverse set of solutions that provides information about the trade offs available to you.” “True what we are suggesting here is that a decision needs to be made and in that sense the multiobjective search seems not to have automatically solved your problem for you.” “Multiobjective search is therefore viewed as a way of providing the opportunity for a decision maker to make informed decisions about the solution based on the information about the solutions that inhabit the Pareto front.” Multiobjective problem solving from nature: from concepts to applications by Joshua Knowles, David Corne, Kalyanmoy Deb

Pareto optimality and genetic algorithms But it doesn’t scale! For many objectives, forced diversity of populations and a weak selection pressure onto a messy and disparate Pareto front results in no convergence at all, and failure.

Pareto optimality and genetic algorithms Two objectives is not multi-objective. Twenty is multi-objective! In real world problems, those objectives may be unknown black boxes that continuously change over time, are noisy, discontinuous, multimodal and constrained. There are no perfect “knee points” for real world problems. Clustering objectives just throws away information. You can’t visualise Pareto fronts in these cases. The whole idea doesn’t work! And do clients actually want to look at Pareto fronts anyway? Would they not prefer to just get the optimum solution?

Pareto optimality and genetic algorithms So if Pareto optimality doesn’t work, why not just look for real optimality? In the general case, you cannot say which answer is most optimal because in the general case you do not know the preferences of the user. But when we optimise, we do not seek an answer in general, we seek one specific optimal that meets the preferences of the user. We can learn these preferences (perhaps through interaction with the client), use them, and thus optimise. We can judge which is best.

Pareto optimality and genetic algorithms Bentley, PJ and Wakefield, JP (1998) Finding Acceptable Solutions in the Pareto-Optimal Range using Multiobjective Genetic Algorithms. In: Chawdhry, PK and Roy, R and Pant, RK, (eds.) Soft Computing in Engineering Design and Manufacturing. (pp ). Springer- Verlag: Berlin/Heidelberg, Germany. (Work done in 1995)

Pareto optimality and genetic algorithms “Simple variants of the often-overlooked ‘Average Ranking’ strategy usually outperform other methods tested, covering problems with 5—20 objectives and differing amounts of inter-objective correlation.” “Our main finding is that ARF, based on the somewhat overlooked AR method [3] seems a very strong candidate for many-objective search. It seems to outperform two recent methods that have been proposed specifically for many-objective problems. Usefully, it is also computationally simple and efficient.” “Perhaps because it was not published in mainstream MOEA literature, and also perhaps because the comparisons were not based on quality measures of the Pareto set, and also perhaps because it tends to be associated with objective-priority approaches, Bentley & Wakefield’s WAR has been little used, omitted for example in the several large or moderate scale MOEA comparison papers.” “From our results, we recommend, of course, that ARF be further tested and used in many- objective scenarios; but we also suggest both ARF and AR be revisited as candidates for ‘standard’ (2—5 objective) problems; preliminary studies indicate that it performs comparatively well in such cases too.” Techniques for Highly Multiobjective Optimisation: Some Nondominated Points are Better than Others by David Corne, Joshua Knowles. GECCO 2007.

Richard Hamming: you and your research And after some more time I came in one day and said, ``If what you are doing is not important, and if you don't think it is going to lead to something important, why are you working on it?’’ I wasn't welcomed after that; I had to find somebody else to eat with!

SBSE is young. Don’t let it be like this: Simply combining two successful areas (optimisation and SE) does not equal success. You need to optimise the ideas themselves and form a new, cohesive whole.

Some Big Dangers in SBSE Don’t use or reinvent the same flawed wheel when it looks like this: (multiobjective optimisation, constraint handling, scalable search)

Some Big Dangers in SBSE Beware! When you’re holding a hammer, everything looks like a nail. For example, if you can calculate the answer directly or other methods solve the problem better (e.g. best prioritised list of requirements), search is the wrong tool for the job.

Some Big Problems in SBSE Learn which search methods should be used with which SE problems. Develop new search methods more suitable for SE problems. Develop new SE methods compatible with search. Surely SBSE should exemplify cutting edge work in search and SE, not gluing old or inappropriate ideas together to make flying cars.

Search & Software Engineering ``What are the important problems of your field?’’

Big problems in Search Scalabilty Complexity Evolvability Adaptive systems Coevolving systems Good representation design And many others…

Big problems in SE “The major challenge faced by software projects is increasing complexity as we create systems by integrating systems and services from different providers.” “The assumption that underlies much of software engineering is that the developer of the software is in control of the system and can therefore make rational decisions about how to develop and change the system.” “When this assumption is no longer true, methods such as software testing methods have to evolve to take this into account.”

Big problems in SE “Engineering scalability: ‘Internet-scale’ services. Handling large and rapid variations in the demand for resources” “Resource estimation. We are unable to reliably predict the cost/effort required to build a system.” Development of emerging classes of ‘adaptive’ system The apotheosis of ‘apps’. Requirements engineering for mass-markets? Software project failures. Mismatches at the boundaries between business and software engineering? Addressing complex inter-product and inter-supplier dependencies. Software ecosystems?

Richard Hamming: you and your research When you get beyond their modesty, most people will say, ``Yes, doing really first-class work, and knowing it, is as good as wine, women and song put together,'' or if it's a woman she says, ``It is as good as wine, men and song put together.'' I think it is very definitely worth the struggle to try and do first-class work because the truth is, the value is in the struggle more than it is in the result. The struggle to make something of yourself seems to be worthwhile in itself. The success and fame are sort of dividends, in my opinion. Go forth, then, and do great work!

Questions?