CS321 - Course Introduction/Review of Algorithm Analysis - Last Revised 10/15/99
OBJECTIVES:
1. To introduce the course
2. To review the notion of algorithm analysis in terms of time and
space
3. To review the O() measure of complexity and how the O() measure can
be obtained by inspecting an algorithm.
4. To review the significance of the O() complexity of an algorithm.
5. To introduce the algorithm classes P and NP
MATERIALS:
1. Syllabus
I. Course Introduction
- ------ ------------
A. Call roll
B. Pass out, go over syllabus - stress:
1. Centrality of ability to select the proper data structure; requires
mastery of pool from which to draw.
2. Responsibility to review material learned previously as necessary.
3. Importance of homework; no lates
II. Review of Algorithm Analysis
-- ------ -- --------- ---------
A. As you already know, one mark of maturity as a Computer Scientist is the
ability to choose intelligently from among alternative ways of solving a
problem. This implies some ability to measure various options to assess
their "cost". The two most common measures are:
1. Time
a. CPU cycles (typically < 10 nanoseconds on modern machines)
b. Disk accesses (typically 10 milliseconds)
(Note: Disk accesses is only relevant to certain kinds of algorithms.
However, in such cases disk access time generally so dominates CPU
time that it is the only measure considered.)
2. Space
a. Main storage - Bytes/KB/MB
b. Secondary storage - blocks
3. Often, there is a trade-off between space and time; one can gain speed
at the expense of more space and vice versa. (But some bad algorithms
are hogs at both)
B. Therefore, one must often analyze algorithms for performing various tasks
in order to discover the best method. Such analyses are generally done
with a view to measuring time or space as a function of n -some parameter
measuring the size of a particular instance of the problem.
1. We often find statements like "such and such algorithm has time (or
space) complexity O(N)" or O(N^2) or O(2^N).
2. By this, we mean that the time (or space as the case may be) required
by the algorithm grows linearly with the problem size (O(N)) or
as the square of the problem size (O(N^2)) or exponentially with the
problem size O(2^N). [The latter case - exponential growth - implies
that the algorithm is impractical for all but the smallest problems.]
C. Formal Definition of "Big O"
ASK CLASS
Formally, we say that a function f(n) is O(g(n)) if there exist positive
constants c and n0 such that:
f(n) <= c*g(n) whenever n >= n0.
We then say that f(n) = O(g(n)) - note that the less precise O function
appears on the right hand side of the equality.
D. To compute the order of a time or space complexity function, we use the
following rules:
a. If some function f(n) is a constant independent of n (f(n) = c), then
f(n) = O(1).
b. We say that O(f1(n)) is greater than O(f2(n)) if for any c >= 1
we can find an n0 such that |f1(n)|/|f2(n)| > c for all
n > n0. In particular, we observe the following relationship among
functions frequently occurring in analysis of algorithms:
2 3 n
O(1) < O(loglogn) < O(logn) < O(n) < O(nlogn) < O(n ) < O(n ) < O(2 )
c. Rule of sums: If a program consists of two sequential steps with
time complexity f(n) and g(n), then the overall complexity is
O(max(f(n),g(n))). That is, O(f(n)) + O(g(n)) = O(max(f(n),g(n))).
Note that if f(n) >= g(n) for all n >= n0 then this reduces to
O(f(n)).
Corollary: O(f(n)+f(n)) = O(f(n)) - NOT O(2f(n))
d. Rule of products: If a program consists of a step with complexity g(n)
that is performed f(n) times [i.e. it is embedded in a loop], then
the overall complexity is O( f(n)*g(n) ), which is equivalent to
O(f(n)) * O(g(n))
Corollary: O(c*f(n)) = O(f(n)) since O(c) = 1
E. It is often useful to calculate two separate time or space complexity
measures for a given algorithm - one for the average case and one for
the worst case. For example, some sorting methods are O(nlogn) in the
average case but O(n^2) for certain pathological input data.
F. The O() measure of a function's complexity gives us an upper bound on
its rate of growth.
1. As such, it is not necessarily a tight bound. For example, if we
know that a particular function happens to be O(n), we are allowed
by the definition of Big-O to say it is O(n^2) or (n^3) or even
O(2^n) [though it would be silly to do so]. That is, big O only
tells us that the behavior can be no worse than some bound.
2. For this reason, we sometimes use one or more additional metrics to
characterize the behavior of an algorithm:
a. Big Omega measures the LOWER bound:
We say that f(n) is Omega(g(n)) if there exist positive constants
c and n0 such that
f(n) >= c*g(n) whenever n >= n0.
(Some writers say "for infinitely many values of n" instead).
Alternately, we can say that f(n) is Omega(g(n)) iff g(n) is O(f(n))
Note that - as with big O, a big Omega bound is not necessarily
tight - e.g. we can say that ANY non-zero function is Omega(1).
b. Theta provides a TIGHT bound.
We say that f(n) is Theta(g(n)) if there exist positive constants
c1 and c2 and n0 such that
c1 * g(n) <= f(n) <= c2 * g(n) whenever n >= n0
Alternately, we can say that f(n) is theta(g(n)) if is true both
that f(n) = O(g(n)) and g(n) = O(f(n))
c. Little oh provides a STRICTLY GREATER UPPER BOUND.
We say that f(n) is o(g(n)) if f(n) is O(g(n)) but not theta(g(n)).
3. Another way of getting at the relative size of two functions is what
happens to their RATIO as n approaches infinity.
a. If lim f(n)/g(n) = 0, then f(n) is little o(g(n))
n -> infinity
b. If lim f(n)/g(n) = some nonzero constant, then
n -> infinity f(n) is theta(g(n))
c. If lim f(n)/g(n) is unbounded, then f(n) is omega(g(n))
n -> infinity
G. While these measures of an algorithm describe the way that its time or
space utilization grows with problem size, it is not necessarily the
case that if f1(n) < f2(n) then an algorithm that is O(f1(n)) is better
than one that is O(f2(n)). If it is known ahead of time that the
problem is of limited size (e.g. searching a list that will never
contain more than ten items), then the algorithm with worse behavior
for large size may actually be better because it is simpler and thus
has a smaller constant of proportionality.
H. Also, in many practical situations one must also consider programmer
time and effort to create and maintain an algorithm. In some cases,
a "poorer" algorithm may be preferred - especially for code to be
run only once or infrequently.
IV. The Complexity Classes P and NP.
-- --- ---------- ------- - --- --
A. Thus far, we have been dealing with the complexity of ALGORITHMS. It
is also possible to consider the complexity of a PROBLEM.
1. The complexity of a problem is simply the complexity of the best
possible algorithm for solving it.
2. For example, we saw in CS122 that the problem of sorting an array of n
items by comparing pairs of items has inherent complexity n log n -
i.e. any algorithm to solve this problem is omega(n log n). (And we
saw some algorithms that did in fact meet this lower bound - such
as merge sort - while others were O(n^2).)
B. Clearly, determining the complexity of a problem can be harder than the
problem of determining the complexity of an algorithm.
1. On the one hand, we may have an algorithm for solving the problem
with a complexity O(f(n)). We know, then, that the complexity of
the problem is no greater than O(f(n)) - but it could be less.
Thus, knowing the complexity of an algorithm for solving a problem
does not mean we know the complexity of the problem itself.
2. On the other hand, we may be able to show that there is a certain
minimum complexity for any algorithm solving a given problem.
However, unless we can exhibit an algorithm of that complexity that
does solve the problem, we do not know whether the complexity of
the problem is equal to that minimum or some greater value.
Example: any problem has complexity at least O(1). But that does
not mean all problems have complexity O(1). To show that a given
problem is O(1), we would have to exhibit an O(1) algorithm for
solving it! Obviously, in most cases we cannot do this.
C. An interesting empirical result is that most problems apparently fall
into one of two categories as regards complexity.
1. There are many problems whose complexity is a polynomial of small
degree, or is bounded from above by such a polynomial.
a. Example: O(1), O(n), O(n^2), O(n^3) are all polynomial complexities.
b. Example: O(log n) and O(n log n) are not polynomial complexities,
but they are bounded from above by polynomials - e.g.
O(log n) < O(n) and
O(n log n) < O(n^2)
c. We call the class of all problems whose complexity is a
polynomial the class P (sometimes written with a script P). We
say, then, that a polynomial-complexity problem is in P or is a
member of P.
Example: most of the algorithms we have studied in previous courses
are members of P.
2. There are other problems whose APPARENT complexity is exponential -
i.e. of the form f(n) * O(r^n) - where f(n) is a polynomial in n
(often 1) and r is a constant > 1 (often 2).
a. Observe that any exponential problem is not in P - i.e. there does
not exist a polynomial in n p(n) such that r^n <= p(n) for all
n > some n0.
b. Notice that I am speaking of problems whose APPARENT complexity is
exponential. In many cases, problems in this class have no known
non-exponential algorithmic solution, but it is not possible to
prove that the exponential solution is the best one possible.
c. Among apparently exponential-complexity problems, there are many
which have the following characteristic:
i. FINDING a solution apparently requires exponential time.
ii. But TESTING a proposed solution to see if it is valid can be
done in polynomial time.
iii. Such problems are said to belong to the class NP. The name
comes from the concept of a NON-DETERMINISTIC machine - an
NP problem can be solved in polynomial time on a \
non-deterministic Turing machine (one that somehow guesses the
right solution and then proceeds to test it). Obviously, such
machines cannot be built, but they are of theoretical interest
for proving theorems.
iv. The class NP is not limited, however, to apprently exponential
complexity problems. It includes any problem for which the
validity of a proposed solution can be tested in polynomial
time, regardless of the complexity of the task of finding a
correct solution. Thus, in particular, the class P is a subset
of NP (P <= NP).
3. One of the most important open questions in theoretical computer
science is the question as to whether the class P is a proper subset
of NP (P < NP), or whether, in fact, P = NP. That is, at the present
time:
a. There is no algorithm in NP which has been PROVEN to have no
polynomial-time solution (though many problems are strongly
conjectured to be such.) Thus, there is no proof that P < NP.
b. No one has found a general way to translate a polynomial complexity
algorithm for TESTING a solution into a polynomial complexity
algorithm for FINDING a solution. Thus, there is no proof that
P = NP.
4. One thing that has been proven, though, is that there is a large
subset of NP problems (called the NP-complete problems) which have
the property that if a polynomial complexity algorithm for any of
them is ever found, then a polynomial complexity algorithm for all
NP problems can be found.
That is, finding a polynomial-complexity algorithm for any
NP-complete problem would constitute a proof that P = NP.
5. The recognition of the existence of the class of NP problems is
important for two practical reasons.
a. Many well-known and important problems fall into this category,
including most optimization problems. (E.g. the travelling salesman
problem.)
b. If P <> NP, then these important problems are of unavoidable
exponential complexity. But exponential algorithms are not practical
for problems of any significant size (whereas polynomial algorithms
generally are).
c. Thus, algorithmic solution of NP problems of bigger than "toy" size
is not possible, and cannot be made possible even with any
conceivable reasonable growth in the speed of computers. Such
problems must, instead, be tackled either by techniques that give
approximate answers (answers that are good but maybe not best) or
by heuristic techniques that work in some cases but not all.
Copyright ©1999 - Russell C. Bjork