What are some of the unique topics that will be well-suited for this Q&A site and what things should belong to other SE sites and betas? How much is this site related to general computer science, algorithms, implementations, programming, mathematics and so on?
This is hard to do in a way that covers all the possibilities, but I think one good approach is to identify the other SE sites that would be likely to overlap with us, and then specify how we want to draw our border with that site. So in that context, here I go:
Math is generally more abstract than what we do here, and tends to be oriented towards proofs and analytical calculation. Therefore, I would suggest that a question on the mathematical properties of an algorithm and/or the rigorous proof of such properties would be the domain of math.SE. For example, asking about how to make your prime-finding algorithm more efficient would be fine on scicomp, but asking how to prove that it is the optimal algorithm for primes larger than N (or whatever) is math.SE
Theoretical Comp. Sci.
Pretty much the same distinction as Math.SE, except that this is where we migrate questions about whether your problem is NP-hard. Note: We probably want to keep questions about the computational complexity of a problem, like "Is there an algorithm to solve that is better than O(n^2)?"
Easy call here. Asking about an efficient means of modeling the diffraction of a beam of light via Fourier transforms is scicomp, asking why light behaves that way is physics.
Questions about how to best compute a solution are scicomp, but questions about how to quantify the quality of the result is stats. I'm think here about goodness-of-fit tests, for example.
Questions about how to tweak your algorithm based on the statistical qualities of your input data is scicomp, but asking how you are supposed to determine what might be a good assumption for those properties is stats. For example, "My data has a standard deviation of foo, what is the optimal value of ?" is scicomp, but "I got this data from . What shape should I expect its distribution to take?" is stats.
I'll add more if I think of them. If you like the way I'm answering this question, feel free do do the same in another answer, or give me a suggestion in the comments and I'll edit it in. Maybe this should be CW?
Following @Colin K's format
Questions about how to use a particular piece of software, either at a user interface (graphical or not) or at a programming interface, are scicomp. Questions about how to write code in a specific language to implement a particular algorithm is SO.
More probably needs to be said about this, if for no other reason SO is so large that any question flux might be larger.
Note: this is not a site I frequent, so I'm speculating somewhat.
Determining what is needed to specify the characteristics of an HPC cluster would be scicomp. Questions about day-to-day administration of an HPC cluster would be Server Fault.
The mathematics that is dealt with on math.stackexchange.com - not to speak of www.mathoverflow.com - is way more abstract then what anybody would research in the environment of numerical mathematics, scientific computing and its applications. Similarly, there are intersections of scientific computing and theoretical computer science.
For example, questions on the convergence of finite element methods, related error estimates, or the convergence properties of algorithms in numerical linear algebra are relevant for research in scientific computing. Explanations of these algorithms, of course, cannot be seperated from their proofs. This is de facto not in the domain of math.SE.
Similarly, the analytical properties of some partial differential equation are often relevant (or inspiring) for their numerical approximation. So questions like: "Does the solution of this PDE blow-up?" or "Do we have this and that conservation property?" can be very useful for the understanding of discretizations. I think it is within the resposibility of the user to decide whether it is better located in pure analysis (e.g, like some Besov-norm-estimates) or to scientific computing (e.g., when the stability of a numerical method depends on the well-definedness of trace operators).
This is even more visible for numerical linear algebra. Any text book on that subject will include exercises on linear algebra which is canonical content of a mathematics program as well, and questions on these topics clearly belong to math stackexchange. Contrary, the convergence behaviour of the conjugate gradient algorithm - or even more algorithms like minimum residual method or Uzawa-type algorithms - depends on spectral properties of the underlying matrix. While spectral theory belongs to pure mathematics, its link to numerical linear algebra is to applied to belong to math.SE. The same can be said for the choice and understanding of preconditioners.
As for the intersection with theoretical computer science, a good example might be anything that involves randomness. The development of a good pseudo-random number generator is relevant. It is hard to draw a boundary here. For example, in simulations of protein folding or Ising models, the particular properties of a model might be relevant to design a pseudo-random number generator that is (slightly) better than standard constructs. (I am not very experienced with these applications).
Other example topics, which are clearly draw from various sources of mathematics (and theoretical computer science) are questions on the geometric stability of mesh-refinement algorithms or (very recent) combinatorial preconditioners.
For the relationship with Stackoverflow, I think questions like "How do I use STL" or "What is a fibonacci heap" are much better located there. Contrarly, expert topics for scientific computing, like,e.g., program models for grid management, are too specific for SO.
The scope should be consistent with the intended scope of the proposal, i.e. "for scientists doing science by heavy computations". See also example questions on the definition phase of the proposal. The scope should not be change to what it was not intended to be, it should not be generalized to significantly overlap with other existing sites. Remember that the intention of this proposal was creating a site for computational science, not something more general like "computer science", the science of computing is different from the use of (heavy) computation in science.
Note also that there is also a proposal for computer science.