12
$\begingroup$

This question is in the same tradition as How appropriate are software package-specific questions?, but I'd like to put more specific guidance in the FAQ to help stem the tide of "debugging questions" we've been getting every so often -- more now that we've started getting FEniCS posts. It's also prompted by Thoughts, comments, and reactions about the FEniCS experiment, in that I want to come up with a concrete set of guidelines.

What are characteristics of software package-specific questions that we should encourage?

What are characteristics of software package-specific questions that we should discourage?

I've already put a preliminary list in the FAQ. Please suggest edits and provide feedback so that we can make it easier for newcomers to get an idea of what types of software package-specific questions are on-topic here.

$\endgroup$
6
  • $\begingroup$ The FAQ mentions that StackOverflow may not be appropriate for programming questions about languages used in computational science, like MATLAB. Is this true? The matlab tag there has nearly 20,000 hits. $\endgroup$
    – Ben
    Commented Jun 3, 2013 at 17:48
  • 1
    $\begingroup$ If I understand correctly, the FAQ merely states "if not MATLAB, more appropriate at SO". This does not preclude that MATLAB questions are appropriate at SO (and in fact, may be more appropriate). My rule of thumb here would be: If it's about MATLAB as a (general-purpose) programming language, ask on SO; if it's about the numerical capabilities of MATLAB, ask on SciComp. Bonus points for scientific context. $\endgroup$
    – Christian Clason
    Commented Jun 3, 2013 at 18:22
  • 3
    $\begingroup$ I think we want to distinguish between "general MATLAB programming" questions and "using MATLAB to solve a specific computational science" questions. I think Stack Overflow is generally really good at answering programming questions, if you can get past the users who will flag your question or tell you it's terrible. (Can I say how glad I am that almost all of you are really nice to the new users?) Where I find Stack Overflow lacking is in the numerical analysis questions. At best, if you take the right parts of all the answers, you get good advice. At worst...it's just bad. $\endgroup$
    – Geoff Oxberry Mod
    Commented Jun 4, 2013 at 5:27
  • 1
    $\begingroup$ @GeoffOxberry A bit off-topic for this thread, but there seem to be similar issues when asking questions about, e.g., fluid dynamics or optimization on Math.SE or Physics.SE. Should we take into account the expected reception of a question on other SE sites when determining whether to move/close? $\endgroup$
    – Ben
    Commented Jun 5, 2013 at 18:00
  • 1
    $\begingroup$ @Ben: Yes. The preferred procedure is to for our sitr mods ask the mods of another site if their site would be a good fit. This process doesn't always happen, though. $\endgroup$
    – Geoff Oxberry Mod
    Commented Jun 5, 2013 at 18:03
  • 2
    $\begingroup$ On a related note, there will be a network-wide change in the closing mechanism; see this question on meta.SO. Particularly relevant is item 5: "off-topic" now allows for site-specific options as well as a free-form field. $\endgroup$
    – Christian Clason
    Commented Jun 14, 2013 at 17:37

5 Answers 5

Reset to default
13
$\begingroup$

Here are two categories of questions we should discourage:

  1. Debugging questions. If a user posts a section of code, says "it doesn't work", and doesn't include discussion of algorithms or underlying mathematics, then the question should be closed. This is admittedly tricky; if the failure is due to not understanding the API correctly, the question is truly not relevant to SciComp.SE, but if the failure is due to a misunderstanding about the underlying problem, the question is much more relevant and interesting. Here is an example of a post that was, in my mind, rightly closed under this metric.

  2. Documentation questions. A reasonable metric for whether a question "shows research effort", to quote the upvote hovertext, is whether it could be fully answered by reading the documentation of the software package in question. This question, for example, is not appropriate because it boils down to correctly understanding the MATLAB/Octave API and solicits little further insight. (Bill Barth gave a great answer, but I would still suggest the question is not appropriate in the first place.)

$\endgroup$
16
  • $\begingroup$ I like this answer, and I want to pose some further questions for thought, because they're situations we may face in the future (or that we've already faced): 1. Suppose a user posts a question that is answered in the documentation, but it's buried and kind of obscure. Or it's towards the end of the documentation, but they've only read the tutorial. On-topic, or off-topic? 2. We're likely to get the occasional new user who is starting out in computational science, and knows next to nothing about searching documentation. What is the best way to help those people? $\endgroup$
    – Geoff Oxberry Mod
    Commented Jun 4, 2013 at 5:38
  • 1
    $\begingroup$ Incidentally, your first example post is one that (one of) the FEniCS developers thinks was wrongly closed (under the metric of FEniCS support -- I believe it was rightly closed under your second category); this is a good touchstone for @GeoffOxberry's comment I think. I would say buried documentation questions are on-topic (simply because I'd find it very useful to have code snippets for more obscure use-cases), but need to clear a strict "shows research effort" bar, as Ben points out ("I've read that [relevant] bit, but here's where I have problems.") $\endgroup$
    – Christian Clason
    Commented Jun 4, 2013 at 8:00
  • $\begingroup$ Regarding @Geoff's second point, I would point them to the tutorials (if they exist) and suggest they work through all of them (even if they're not immediately relevant to their problem), try to modify one for their problem, and come back if they hit a concrete snag. If there are no tutorials (shame on the devs), I would see value in having a worked example on this site -- which the next new user could then be pointed to. $\endgroup$
    – Christian Clason
    Commented Jun 4, 2013 at 8:05
  • $\begingroup$ @ChristianClason: I think there's probably a way to salvage the FEniCS question Ben linked to. The chief problem we've had with questions like that is the localized, time-intensive nature of debugging. Someone has to volunteer to do the debugging, and with a question like that, it's not immediately obvious if the answer will help people besides the OP (although I hope it would). If a FEniCS dev has an issue with closing that question, they should post a thread on Meta and make their case, and hopefully, we as a community can work something out. $\endgroup$
    – Geoff Oxberry Mod
    Commented Jun 4, 2013 at 8:57
  • $\begingroup$ @GeoffOxberry - I agree, the main problem with these questions is not so much the content as the form (not focussed enough, no context etc.). The question is who is willing to invest the time to get these questions into suitable shape (especially if the poster does not have the (scientific or language) skills to do that). If (e.g.) the FEniCS community takes responsibility for that (and not just the answer) I can see things working quite well. $\endgroup$
    – Christian Clason
    Commented Jun 4, 2013 at 9:15
  • $\begingroup$ @GeoffOxberry - I should point out that the FEniCS dev did not have an issue with closing that SciComp question, but thought that it was a relevant FEniCS question which should be answered somewhere, with SciComp apparently not being the right place. I apologize if I have misrepresented that. $\endgroup$
    – Christian Clason
    Commented Jun 4, 2013 at 9:17
  • $\begingroup$ @GeoffOxberry: A reasonable metric I've used in the past is whether I can answer the [documentation] question in under 2 minutes with a few choice Google searches. If that is the case, I'd say leaving a comment with a link and closing the question is an appropriate course of action. (On the other hand, this may have the effect of encouraging more low-quality questions, and I don't have an answer for what to do about that.) $\endgroup$
    – Ben
    Commented Jun 5, 2013 at 18:03
  • $\begingroup$ @ChristianClason: You make an excellent point about the first example, and in doing so illustrate the challenge debugging questions present. I agree with you and Geoff that the structure of debugging questions is the major issue; it takes an enormous amount of effort on the part of the reader to understand the underlying question, and only then can a determination be made about the underlying scientific merit. $\endgroup$
    – Ben
    Commented Jun 5, 2013 at 18:12
  • $\begingroup$ FYI, we have realized that scicomp.stackexchange is a bad fit for FEniCS questions: in addition to being an extra burden on the scicomp moderators (and an annoyance to some), it's too hard for our users to decide what should go on scicomp and what should go to our mailing list, and we as FEniCS developers need to constantly worry that we make a bad impression on scicomp as a result of a bad question by a random user... :-) So we are looking for alternative Q&A engines/forums (suggestions welcome). Anyway, thanks for letting us do this experiment! It didn't work out but we learnt from it. $\endgroup$
    – Anders Logg
    Commented Jun 5, 2013 at 18:22
  • 1
    $\begingroup$ @AndersLogg - I'd say it's too early to call the experiment a failure (although I do understand the argument about multiple places for questions). I for one would be sad to lose the FEniCS presence on this site, not least because I think it can be a great teaching tool: "I have this numerical PDE question" - "Here's the answer, and here's a quick illustration using Dolfin". $\endgroup$
    – Christian Clason
    Commented Jun 5, 2013 at 18:37
  • $\begingroup$ @AndersLogg: I like the FEniCS presence on the site, and support it. I think it can be a good fit; it doesn't annoy me (or Paul, I think). I think we as a site need to do a better job of describing what sorts of software package-specific questions are best answered here, and what questions are best answered elsewhere. In that respect, I think FEniCS users have done us a service by forcing us to think about that issue. $\endgroup$
    – Geoff Oxberry Mod
    Commented Jun 5, 2013 at 21:18
  • 1
    $\begingroup$ @AndersLogg: My hope is that the FEniCS experiment here will continue, because I think it is making the site better. I hope it helps FEniCS and its user base too; if it does, I urge you to reconsider. $\endgroup$
    – Geoff Oxberry Mod
    Commented Jun 5, 2013 at 21:19
  • 1
    $\begingroup$ @Ben: I think your suggestion highlights an interesting point about expert knowledge. I believe this site (and you) can make a powerful contribution because you can do a 2 minute Google search and answer a question easily that a non-expert may not be able to do in 2 minutes (or two hours, or days), because they have not acquired that expert knowledge. For that reason, I think your metric should be altered to something like "has this person made a good-faith effort to locate the information they're asking about?" $\endgroup$
    – Geoff Oxberry Mod
    Commented Jun 5, 2013 at 21:26
  • $\begingroup$ I really like scicomp.stackexchange, but we (FEniCS users and developers) need a forum where users can ask whatever questions they like without fear of reprimand. We looked around for alternatives and Question2Answer looks like it does what we need. We've set up an experimental forum here: fenicsproject.org/qa $\endgroup$
    – Anders Logg
    Commented Jun 6, 2013 at 20:28
  • $\begingroup$ @GeoffOxberry You make a compelling point about expert knowledge, and I must admit I had not considered that angle. The only problem I see with the "good-faith" metric is that it might encourage users to post excessively long and detailed questions to demonstrate their past attempts to find an answer. This wouldn't necessarily be bad, but might make it harder for other users to find useful information. $\endgroup$
    – Ben
    Commented Jun 8, 2013 at 0:39
7
$\begingroup$

I think the FAQ already succinctly captures the main points; the problems likely are due to those questions where askers either did not read the FAQ or failed to apply them to their questions. For those people (and others watching), a much more visible signal is how questions are treated which fail to conform to these guidelines, and why.

First, some philosophical thoughts to motivate what follows. Clearly, SciComp should not and cannot be a replacement for a dedicated support forum, but it can be a very valuable complement. In particular, the main advantages of the former over the latter are

  1. Long-term discoverability: Solved problems are much easier to find, either with Google or a site-specific search. As a corollary, it's very convenient for pointing users with already solved questions to. (This is the reason developers want to have questions on a StackExchange site.)
  2. Scientific expertise: Ideally, users will not only get a solution to their problem, but also an explanation why the problem is solved this way (i.e., the why together with the how). (This should be the reason developers want to have questions on this StackExchange site. I want to make clear that I'm not saying the developers don't have the necessary expertise; just that there is more of it (and broader) here due to the larger audience, and there is already a culture of tackling questions in this context.)

If the question does not make use of (at least!) one of these advantages, it is either better posed somewhere else or posed incorrectly. (More on that below.)

Regarding how to treat specific questions, I would hope to be as inclusive as possible -- for the sake of the askers and of the site -- in the sense that no topic is a priori off-topic as long as sufficient effort was made to leverage the above advantages (i.e., without sacrificing quality). For example, this means the question should

  1. Make it about a problem, not your problem. Provide code, but make it minimal so that it contains what and only what is necessary for this question (as brief as possible, as specific as necessary).
  2. Provide (scientific) context. This helps the answerer select the most appropriate from multiple possible solutions, and also catch cases where something is not working because the whole approach is wrong. In particular, it should be made clear that a valid answer to the question "How do I do X in package Z?" would be "Don't do that in Z, use Y; here's how." (If this answer is not acceptable to the asker, it's a sign he's using the wrong forum.)

Clearly, the more software-related the question, the higher the bar on these points. Just as clearly, the burden is on the asker to convince the community in this respect. (I think that is already the practice in the community; there have been questions that were as software-specific as some closed questions but received up-votes and excellent answers because they were well-posed. This could be made explicit in the FAQ.)

There is also the issue of how to treat questions that failed to convince the community. The difficulty here is that closing a question can mean two different things: The question is not welcome here (as in "reject") or that it is not posed appropriately (as in "major revision") but will be reopened if suitably modified. In principle, the difference shows in the reason for closing, but that is not always so obvious, especially to a new user coming from outside the SE network (especially, support forums). We should be much more explicit about this difference, both in the FAQ and the comments. (And my personal feeling -- at least from the latter -- is now that we do too much of the former, and not enough of the latter, which lead to some frustration. We could also use a "minor revision" option: Down-vote and leave a comment asking for improvement, but (publicly) give a grace period (e.g., 24 hours) before voting to close.)

On a related note, if the asker is not willing (or, more importantly, does not possess the necessary scientific, programming or language skills) to make his question pass those bars but someone (e.g., developers) thinks the question should be answered for the sake of the (software-specific) community, that should be acceptable as long as the question is also edited accordingly by them. (Of course, that might send wrong signals; probably deserves a separate question on meta.)

To end with a concrete suggestion: In addition to the general guidelines, the FAQ could also link to example questions that are considered good and bad, respectively, together with comments on what makes them so:

  • "The following question was up-voted and received several helpful answers because..."
  • "The following question was closed as too localized because it only asked about a technical problem with the software..."
  • "The following question was closed as off-topic because it asked about a specific implementation question without context" or "did not show that it wasn't already answered in the documentation", possibly adding "(but would have been on-topic if ...)"

(This would have the added benefit of making visible the results the asker can expect for his question -- carrot and stick, if you will.)

$\endgroup$
1
  • $\begingroup$ I really like this answer, and I'd say that "make it about a problem, not your problem" cannot be stressed enough, as it ties into long-term discoverability. Not only are debugging questions hard to parse for those of us attempting to answer the question, but they may also involve interesting [underlying] issues for which they won't appear in search results on account of poor formulation and keyword use. $\endgroup$
    – Ben
    Commented Jun 5, 2013 at 18:16
4
$\begingroup$

This is more a suggestion on how to give guidance: In addition to the general guidelines in the FAQ, the tag wikis for the different software packages could give more concrete hints on what questions are considered on-topic, as well as links to the software-specific mailing lists or webpages. They are probably not as visible as the FAQ, but pointing users to the wiki instead of writing comments could save some time.

As a rough example (feel free to correct), we could have something like

  • on-topic: Questions about the mathematical capabilities of Matlab (solving linear systems, how to use eigs with anonymous functions, which functions/toolboxes to use for a certain problem)

    off-topic: Questions about the programming language of Matlab (vectorizing loops, array manipulation; these should go on Stackoverflow)

  • on-topic: Questions about using FEniCS for solving specific problems (Formulating specific PDEs, how to use FEniCS in optimization routines)

    off-topic: Questions about problems when using FEniCS (error messages when compiling/linking/running C++/Python code, UFL syntax, clarifications on interfaces; these should go on http://fenicsproject.org/qa)

  • on-topic: Question about using PETSc in application code (which function to use for solving nonsymmetric indefinite system with preconditioner, how to set tolerances)

    off-topic: Questions about PETSc problems, bug reports; these should go on petsc-maint list

$\endgroup$
2
  • 1
    $\begingroup$ This is a fantastic idea for the sake of clarity, but visibility would be a major issue; I'm not sure most new users are even aware of tag wikis. Perhaps the on-/off-topic summaries for the most popular tags could go in the FAQ? $\endgroup$
    – Ben
    Commented Jun 8, 2013 at 0:22
  • 1
    $\begingroup$ @Ben - Sure; for the rest I'd say a) replace the text by the corresponding [tag:] links, and b) prominently write "Before asking a question about one of these packages, make sure to read the corresponding tag wiki first." (On the topic of visibility, I notice that the site-specific FAQ has gone into hiding somewhere in the depths of the new help center. Is there anything we can do to get it front and center again?) $\endgroup$
    – Christian Clason
    Commented Jun 8, 2013 at 8:41
3
$\begingroup$

Some concrete guidelines I think should exist, but have been absent in some questions (but later rectified after comment/edit iterations): If the user of a software package is noticing behavior that deviates from expectations, they should not only post the relevant code (which is only understandable in many cases to people who use the same package), but also post all relevant physical auxiliary conditions and numerical methods that are going on under the hood. Incorrect usage of a package isn't the only kind of bug that can occur, sometimes a numerical method isn't appropriate or sometimes auxiliary conditions aren't imposed properly. This should obviously be accompanied by a detailed explanation of results.

More importantly if the questioner provides all relevant information as if they wanted the reader to be able to reproduce the same result, then people without package-specific knowledge can chime in and provide valuable feedback if it's relevant. These kinds of questions may not be research grade, but they can still be very valuable if well explained.

One practice that is taken seriously in the LaTeX stack exchange is to provide a MWE (minimum (non)working example). The minimum code which produces the spurious behavior. I think the criteria should be such that everybody could participate in an answer as much as possible, maximizing the possibility that simple questions get simple responses quickly, and difficult questions can therefore receive more attention.

$\endgroup$
1
  • 1
    $\begingroup$ The minimum working example part is an excellent point. It is covered in Eric Raymond's "How to ask questions the smart way", but is well worth repeating. So is the part about including enough information in order for other people to reproduce their results. With code, my preference tends towards snippets that make up a MWE. If you look at examples in PETSc, these tend to be multiple pages, and people may not have the patience to read through that, in which case, providing a link to the code and excerpting key segments may be better. $\endgroup$
    – Geoff Oxberry Mod
    Commented Jun 9, 2013 at 1:16
1
$\begingroup$

From my point of view ALL software package-specific questions should be encouraged if the asked software package is intended to use in computational science. These questions can seem not interesting for those who don't use these packages, but this is even more true for any other SE community: nobody is interested in computational science software outside computations science community at all. I.e. these questions turn orphan while for newcomers the questions about software is 99%. Prohibiting these questions make community too elite and too closed.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .