Methodologist’s Toolchest

Methodologit’s Toolchest V. 2.0 (1998) by the Idea Works, Inc. - "The researcher still makes the final decision, but now he or she can be confident that it is an informed one."

I obtained a free demonstration version but it unfortunately has a predefined tutorial. I was not able to work with my own data. And it’s not really a tutorial, it’s a tour.

The Methodologist’s Toolchest is for anybody planning a research project. It is designed "to help you eliminate costly errors in funding proposals" by walking you through the construction of a research proposal. It states that researchers writing funding proposals, students designing theses and dissertations, or staff scientists designing in-house research projects will all benefit from it use.

The program has the following subprograms:

• Peer Review Emulator – Simulates peer review using an expert system with iterative feedback, by comparing sections that roughly correspond to major federal research agencies’ (e.g., NSF, NIH) proposal requirements. It asks you for information and then evaluates that information to identify contradictions, suggests concerns to address, compares against standards, and tracks the proposal’s status. It is possible to edit the proposal from within MT, or link it to a word processor.

• hypertext
• progress indicators and summaries
• easy to track many proposals and to reuse boiler plate text

• PRE produces a "first draft" automatically from text you input as you go through the checklist.
• Editable
• Includes recommendations and critiques
• Ex-Sample – Helps you determine appropriate sample size by suggesting test statistics and calculating power values. Suggests modifications to the study to reduce sample size, and formats for reporting results. Sample Size Control Panel lays it all out for you. Provides extensive definitions and descriptions of 60 different statistical analyses. Calculation worksheet allows for simple sensitivity tests on assumptions.
• Designer Research – Evaluates specific designs and design strategies for 35 threats to validity as an expert system built from three literatures: social science (emphasis on threats to validity, basic methodological issues), psychology and behavior science (internal validity from lab studies), and physical sciences and industry (efficient research design, complex designs).
• huge list of threats to validity due to contamination, internal validity, statistical conclusion validity, external validity, and construct validity.
• helps you strategize to reduce threats to validity using worksheets
• Data Collection Selection – Another expert system, it asks researcher a set of questions (e.g., kinds of data, precision required, epistemological issues (?), importance of data completeness, analysis claims, pragmatic concerns, ethical issues), and then "using artificial intelligence strategies" rates the appropriateness of various data collection methods. Includes hypertext definitions of methods and terms. Rating system allows you to do simple sensitivity tests of your answers.
• Measurement and Scaling Strategist – Another expert system that asks a series of questions to help you construct questionnaires based on identification of the most appropriate forms of measurement and scaling. Questions can be answered on scale from 0 to 10, instead of yes/no. A text editor allows you to rewrite questions on the fly. Database includes a large list of possible scaling methods (e.g., scales for organizational concepts). Also has a checklist of common problems against which to assess each question.
• Statistical Navigator – Compared to having a statistical consultant at your fingertips, assists in the appropriate choice of over 200 different types of statistical analyses. Through a series of questions about research objectives and assumptions you are willing to make, it supposedly combines "expert systems" with "decision support systems" to make suggestions based on "pattern matching". Large database statistical techniques’ descriptions, references and glossary of terms. Process: Questions, Consult Mode, Summary Mode, Detailed Report. There is also a Browse Mode, and an ability to do simple sensitivity analyses.
• WhichGraph – Another expert system asks user a set of questions about objectives and what assumptions can be made about the data. Then it "provides users with sound advice for designing the best graphical display of data." Working from a database of over 100 graphs, 400 definitions of graphical terms, and 200 types of statistical analyses, it recommends graphs, tells you what each graphs does, and how it does or does not meet the researcher’s objectives.
• Hyper-Stat – Hypertext dictionary of statistical, graphical and methodological terms. Interacts with other subprograms or can be used as a stand alone tool. The dictionary includes over 1,900 terms, including 1,400 definitions and 500 synonyms, as well as over 4,000 hypertext links.
• ETHX – Using general guidelines culled from Institutional Review Boards, it helps researcher determine if human subjects are involved, whether the research can be exempt from full human subjects reviews, and, if full review is necessary, if the standards for review are met. Helps you design human subjects proposal (consent forms too?) Example, provides a list of strategies to assure confidentiality. Follows same format as Peer Review Emulator, using a set of questions, checklists, and status screens.

Overall observations:

• pretty cheap, but appropriately simple, layout of screens
• use of scoring system seems silly, but confidence factors (instead of yes/no) is nicely flexible
• lots of text to wade through for definitions and questions
• ultimately, you have to make all of the decisions and do the work – good as a checklist?
• working through MT is a lot of work (answering questions, reading hypertext definitions) that I probably would not make or have time to do. But in the unlikely event that I did dive into MT, I think it would be worth it.

Difficult to say, each subprogram has something to offer. For example, I don’t expect you’d use the draft editor, but the peer review checklist may be very useful. Some researchers may appreciate having a tool that tracks your progress, or that organizes boiler plate text. I probably wouldn’t use it for that.

The Ex-Sample subprogram may be a good disciplined way to work out sampling needs, but some may find it to be too constrained. The language seems very "plain english" which would help me. The sensitivity modeling would be quite helpful, I think.

MT is a comprehensive research design tool, but it won’t do you research for you. And you really need to do the designing. It just provides a good checklist of questions and design tests, as well as one-stop shopping for definitions and descriptions.

Difficult to say, because I couldn’t really test it. In fact, the demonstration isn’t really a way of demonstrating how the program works. Instead, the demo is just a walk through program features. Unfortunately, to truly test it, you’d need to use it. I suspect that it doesn’t really deliver all that the demo says, that the "interaction" isn’t quite so fluid, and that the expert system isn’t really such a smart "expert". If it were, you wouldn’t have to buy the program to find out, you’d be allowed to work with it as a true demonstration version.